Build a local Yumrepo Server

part 3 of 4

Posted by paul on 2015.03.18

Build local a local Yumrepo Server - part 3 of 4

2015.03.18 Wed

  1. Part 1: Intro
  2. Part 2: Yumrepo 6.6basic - RPMs from DVDs only
  3. Part 3: Yumrepo 6.6 - exact mirror of a full Yumserver
  4. Part 4: How to use yumupdate.sh

Overview

In this 3rd part of the series on setting up a local yumserver, I will explain how to set up a local yumserver that your CentOS 6.6 servers in your home/office can use for yum install or yum update. This local yumserver will act exactly as a normal yumserver out in the internet, except your yumserver is located in your LAN, allowing much faster download and better control. Having a local yumserver will help save a lot of time, especially when you have more than a few (or a lot more) CentOS servers to manage. Having a local yumserver will also allow you to use Kickstart to completely automate installing CentOS servers. I will post a blog on using Kickstart in a few weeks.

You will be downloading almost 10GB of data from a CentOS mirror server with a script, yum-rsync-minimal-centos6.6.sh, which will take a few days. Please be patient.

Prep a CentOS server

If you have not completed part 2 and you don't have a CentOS server ready to be set up as a yumserver, prepare a CentOS 6.6 server with about 100GB - 150GB of data space. Creating yumrepo 6.6 will require about 25GB of space (some of it only temporarily), although it will only need about 15GB to operate.

Downloadable scripts

You can copy/paste script from this web page. Or download yum-scripts.zip and unzip the content (2 scripts) to /root/bin/. You should see the 2 scripts in /root/bin/.

[[email protected] ~]# ls /root/bin/ | sort
yum-create-server-centos6.6.sh
yum-rsync-minimal-centos6.6.sh

Prep script, yum-create-server-centos6.6.sh

Create /root/bin/yum-create-server-centos6.6.sh and make it executable. Or you can download it from yum-scripts.zip.

#!/bin/bash
# Run this script to create yumrepo "6.6" from scratch on a new CentOS server.
# It goes from copying files from .ISO files to updating httpd.conf.
# You need to have 2 ISO files in /root/isos/
# 1. CentOS-6.6-x86_64-bin-Dvd1.iso
# 2. CentOS-6.6-x86_64-bin-Dvd2.iso

# Whole process will take about 30 min to 1 hour, depending on the hardware.


#####
# variables
#####

# Use 1 of the 2 variables (never both) below to specificy hostname of the yumserver.
YumServer="yumserver.home.loc"
#YumServer="`hostname -f`"

Version1="6.6"  #Actual version of CentOS: 6.5 or 6.6.
Version2="${Version1}"  #Version of CentOS plus type. ex 6.6basic.
VersionMajor=${Version1:0:1} # This will be 6 if $Version1 6.6. Or 7 if $Version1 is 7.1.

Isos="/root/isos"
Dvd1="CentOS-6.6-x86_64-bin-DVD1.iso"
Dvd2="CentOS-6.6-x86_64-bin-DVD2.iso"

DvdContent="/root/dvdcontent"

Target1="/data/www"
Target2="yumrpms/centos$Version2"
Target3="$Version1/os/x86_64"
Target="$Target1/$Target2/$Target3"

BinFolder="/root/bin"  # where my scripts are kept.

ApacheDir="/data/www/"

TotalSteps="4"

#####
# variables used in generating $Create
#####
Create="yum-createrepo-$Version2.sh"  # yum-createrepo script to be created.
RepoFile="66.repo"  # .repo file to be put into Yum Clients.
File1="centos$Version1-x86_64-comps.xml" # .xml file to be used for 'createrepo -g' command.
Rpms1="/data/www/yumrpms/centos$Version2" # where source rpms are.
Folder1="/data/yumcomps"  # folder to hold 6.xxx_x86_64-comps.xml.




#####
# script
#####

echo -e "\nStep 1 of $TotalSteps: Copying files from ISO files to $DvdContent/$Version1."
# Check if 2 .iso files are present
if [ -f $Isos/$Dvd1 ];
then
echo -e "\n$Dvd1 found."
else
echo -e "\n$Dvd1 not found. Please copy in $Dvd1 to $Isos and restart this scipt."
exit 1
fi

if [ -f $Isos/$Dvd2 ];
then
echo -e "\n$Dvd2 found."
echo -e "\nScript will proceed"
else
echo -e "\n$Dvd2 not found. Please copy in $Dvd2 to $Isos and restart this scipt."
exit 1
fi



# Copy files from DVD to /root/dvdcontent
mkdir /mnt/cdrom 2>/dev/null
umount /mnt/cdrom 2>/dev/null

mkdir -p $DvdContent/$Version1 2>/dev/null

mount -o loop $Isos/$Dvd1 /mnt/cdrom
mkdir $DvdContent/$Version1/1 2>/dev/null
rsync -a --delete /mnt/cdrom/* $DvdContent/$Version1/1

umount /mnt/cdrom

mount -o loop $Isos/$Dvd2 /mnt/cdrom
mkdir $DvdContent/$Version1/2 2>/dev/null
rsync -a --delete /mnt/cdrom/* $DvdContent/$Version1/2

umount /mnt/cdrom


# Copy DVD content to $Target
mkdir -p $Target 2>/dev/null
rsync -a $DvdContent/$Version1/1/* /$Target/
rsync -a $DvdContent/$Version1/2/* /$Target/


# link 6.6 to 6
cd $Target1/$Target2
if [ ! -h $Target1/$Target2/$VersionMajor ]; then
cd $Target1/$Target2
ln -s $Version1 $VersionMajor
fi



echo -e "\nStep 2 of $TotalSteps: Copying centos6.6-x86_64-comps.xml."
# Copy comps.xml
mkdir $Folder1 2>/dev/null
cp $DvdContent/$Version1/1/repodata/*x86_64-comps.xml $Folder1/centos6.6-x86_64-comps.xml



echo -e "\nStep 3 of $TotalSteps: Configuring Apache."
# Install httpd and createrepo
yum -y install httpd createrepo

# Prep Apache
chkconfig httpd on

# Back up /etc/httpd/conf/httpd.conf
cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.orig

# Change DocumentRoot to $ApacheDir
sed 's/var\/www\/html/data\/www/'  -i /etc/httpd/conf/httpd.conf


chown -R apache:apache $Target1
service httpd restart




echo -e "\nStep 4 of $TotalSteps: Generate and execute $BinFolder/$Create."
# Create $Create
cat <<EOM > $BinFolder/$Create
#!/bin/bash
# This script runs createrepo command in all directories with .rpm filddes.
EOM

find $Rpms1 -name "x86_64" >> $BinFolder/$Create

sed -i "s/\/data/createrepo \/data/g" $BinFolder/$Create

echo "createrepo -g $Folder1/$File1 $Target" >> $BinFolder/$Create
echo "chown -R apache:apache $Target1/$Target2" >> $BinFolder/$Create




# Prep $RepoFile file.
echo -e "\nTo configure Yum Client to use this newly created yumrepo:"
echo -e "\nTo configure Yum Client to use this newly created yumrepo, $Version2"

echo "###############################"
echo -e "1. copy yumserver01:$BinFolder/$RepoFile to yumclient:/etc/yum.repos.d/$RepoFile "
echo -e "\n2. On yumclient: rename /etc/yum.repos.d/CentOS-Base.repo to /etc/yum.repos.d/CentOS-Base.repo.orig"
echo -e "\n3. On yumclient: run 'yum clean all'. Now you can run 'yum install httpd' or 'yum update'."
echo "###############################"

cat <<EOM > $BinFolder/$RepoFile
# $RepoFile
# Change hostname to IP if you do not have DNS for local machines.
[base]
name=CentOS-\$releasever - Base
baseurl=http://$YumServer/yumrpms/centos$Version2/\$releasever/os/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6


#released updates
[updates]
name=CentOS-\$releasever - Updates
baseurl=http://$YumServer/yumrpms/centos$Version2/\$releasever/updates/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6


#additional packages that may be useful
[extras]
name=CentOS-\$releasever - Extras
baseurl=http://$YumServer/yumrpms/centos$Version2/\$releasever/extras/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6


#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-\$releasever - Plus
baseurl=http://$YumServer/yumrpms/centos$Version2/\$releasever/centosplus/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#END of file
EOM


# Execute $Create
echo -e "\ncreaterepo command in progress. Will be completed in about 10 - 20 mins.\n\n"
chmod 750 $BinFolder/$Create
/bin/bash $BinFolder/$Create

Run yum-create-server-centos6.6.sh

  1. When running /root/bin/yum-create-server-centos6.6.sh, use following command. It will send the output (including error) to both screen and to /root/log-create-yumrepo-66.log.
  2. [[email protected] ~]# log2=/root/log-yum-create-server-centos6.6.log;yum-create-server-centos6.6.sh 2>&1 | tee -a $log2; echo -e "\n# Done" >> $log2
  3. It will take about 30 mins to complete.
  4. Once /root/bin/yum-create-server-centos6.6.sh has completed running, yumrepo 6.6 at this stage has same RPMs as yumrepo 6.6basic. However if you run yum command on a yumclient with 66.repo, you will run into 404 error.
  5. Yumrepo 6.6 is not yet fully functional as a true yumrepo with latest updates but will become fully functional after you create and execute /root/bin/yum-rsync-minimal-centos6.6.sh, as explained below.
  6. Look for /root/bin/66.repo that was generated by yum-create-server-centos6.6.sh. This 66.repo will be used by yumclients.
  7. Before creating /root/bin/yum-rsync-minimal-centos6.6.sh though, you need to find a public CentOS mirror server to download (using rsync) from.

Find Rsync source

You need to find a CentOS Mirror server near your location that you want to rsync from.

  1. Go to http://mirror-status.centos.org/
  2. Pick a region close to you by clicking on one of the links under 'regions'.
  3. I will pick mirror.rackspace.com for this example. Please, please, pick another server that's closer to your location.
  4. Now ssh into your Linux server and run rsync command against the server. Note the URL starts with rsync, not http or ftp. You should see centos as one of the result.
  5. [[email protected] ~]# rsync rsync://mirror.rackspace.com
    centos         	CentOS
    centos-vault   	CentOS vault
    cpan           	CPAN
    ...
    
  6. The rsync command found 'centos' folder. Next, run following 2 commands. We are looking to download CentOS 6.6, so look for 6.6
  7. [[email protected] ~]# rsync rsync://mirror.rackspace.com/centos/
    ...
    drwxrwxr-x        4096 2014/10/19 14:36:15 6.6
    ...
    [[email protected] ~]# rsync rsync://mirror.rackspace.com/centos/6.6/
    drwxrwxr-x        4096 2014/10/19 14:36:15 .
    drwxr-xr-x          27 2014/10/19 14:36:15 SCL
    drwxrwxr-x          42 2014/10/20 02:36:56 centosplus
    drwxr-xr-x          42 2014/10/19 14:36:15 contrib
    drwxr-xr-x          42 2014/10/19 14:36:15 cr
    drwxr-xr-x          42 2014/10/19 14:36:15 extras
    drwxr-xr-x          42 2014/10/19 14:36:15 fasttrack
    drwxrwxr-x          42 2014/10/21 10:08:01 isos
    drwxrwxr-x          42 2014/10/21 09:56:44 os
    drwxrwxr-x          42 2015/01/27 15:13:28 updates
    drwxr-xr-x          27 2014/10/19 14:36:15 xen4
    [[email protected] ~]#
    
  8. So the Rsync source will be rsync://mirror.rackspace.com/centos/6.6/ (in almost all cases it will use ../centos/6.6. This will be added in /root/bin/yum-rsync-minimal-centos6.6.sh, with a bit of change.

Prep yum-rsync-minimal-centos6.6.sh

  1. Create /root/bin/yum-rsync-minimal-centos6.6.sh, used to rsync down rpms from a public yumserver to your local yumserver's yumrepo 6.6. Or you can download it from yum-scripts.zip. Make it executable. Do not run yum-rsync-minimal-centos6.6.sh just yet.
  2. #!/bin/bash
    # This script rsyncs rpms down to the local Yumrepo server.
    # It then creates a separate shell script (yum-createrepo-$Version2.sh)
    # which runs createrepo in necessary folders.
    
    
    #####
    # variables that should be updated
    #####
    Version1="6.6"  #Actual version of CentOS: 6.5 or 6.6.
    Version2="${Version1}"  #Version of CentOS plus type. ex 6.6basic.
    RsyncSource="rsync://rsync.gtlib.gatech.edu/centos"
    
    #####
    # variables
    #####
    Source1="$RsyncSource/$Version1"
    
    Target1="/data/www"
    Target2="yumrpms/centos$Version2"
    Target3="$Version1/os/x86_64"
    Target="$Target1/$Target2/$Target3"
    #Target="/data/www/yumrpms/centos$Version2" # where rpms are kept on local server
    
    Create="yum-createrepo-post-rsync-$Version2.sh"  # script (to run createrepo) to be generated.
    BinFolder="/root/bin"  # where my scripts are kept.
    File1="centos$Version1-x86_64-comps.xml" # .xml file to be used for 'createrepo -g' command.
    Folder1="/data/yumcomps"  # folder to hold 6.xxx_x86_64-comps.xml.
    
    LogOutput="/root/log-yum-rsync-minimal-centos6.6.log"
    
    ###
    # script
    ###
    
    echo -e "\n`date`\nRsync and createrepo commands started" 2>&1 | tee -a $LogOutput
    ### Rsync rpms down
    # Rsync down $SOURCE1 only during initial setup of Yumrepo server or when you are moving from version 6.5 -> 6.6.
    # Normally, rsync down of $SOURCE1 should be done only when you are ready to move from 6.5 -> 6.6. If you are ok
    # with upgrading whenever CentOS does, then leave below uncommented.
    
    
    # Rsync down the RPMs.
    /usr/bin/rsync -av --bwlimit=524 --exclude=repodata --exclude=i386 --exclude=debug --exclude=isos --delete $Source1 $Target1/$Target2/.
    echo "`date +%F.%H-%M`: Rsync was executed successfully." 2>&1 | tee -a $LogOutput
    
    
    # Create $Create
    cat <<EOM > $BinFolder/$Create
    #!/bin/bash
    # This script runs createrepo command in all directories with .rpm filddes.
    EOM
    
    find $Target1/$Target2 -name "x86_64" >> $BinFolder/$Create
    
    sed -i "s/\/data/createrepo \/data/g" $BinFolder/$Create
    
    echo "createrepo -g $Folder1/$File1 $Target1/$Target2" >> $BinFolder/$Create
    echo "chown -R apache:apache $Target1/$Target2" >> $BinFolder/$Create
    
    
    # Execute $Create.
    chmod 750 $BinFolder/$Create
    /bin/bash $BinFolder/$Create
    echo "`date +%F.%H-%M`: Createrepo was executed successfully." 2>&1 | tee -a $LogOutput
    
    
    chown -R apache:apache $Target1/$Target2
    
    
  3. Note the value of variable RsyncSource, which is "rsync://mirror.rackspace.com/centos" in this tutorial. We are rsyncing from variable Source1 which is "$RsyncSource/$Version1", which in this script is "rsync://mirror.rackspace.com/centos/6.6".
  4. The value of variable "RsyncSource" should be changed to a server that's close to your location.
  5. This script will generate log file, /root/log-yum-rsync-minimal-centos6.6.log. No need to specify log file location while running the script as done earlier.
  6. Note the actual rsync command has some options.
    1. "--bwlimit=524" caps the download speed to 524bytes. I would highly recommend setting the cap no higher than 1000 to not negatively impact internet usage experience of others (even yourself) in your office/home.
    2. Following folders are excluded: i386, debug, isos. Hence the script filename has 'minimal'.
    3. "--delete" option deletes files/folders from target (/data/www/yumrpms/centos6.6/6.6/) if it's not in source (rsync://mirror.rackspace.com/centos/6.6/). This option should be used to sure the target stays synced exactly as the source. Be careful that you are using correct source/target. If not careful, you can have accidental loss of data.
    4. Using ls and "du -hs", check downloaded RPMs are saved in correct folder (
  7. Look for $BinFolder/$Create (aka /root/bin/yum-createrepo-post-rsync-6.6.sh) that is generated and executed at the end, which runs 'createrepo' command against folders with rpms.
  8. Do not run /root/bin/yum-rsync-minimal-centos6.6.sh yet.

Execute yum-rsync-minimal-centos6.6.sh

Do a test run of /root/bin/yum-rsync-minimal-centos6.6.sh and let it run for a minute. Then stop the script with ctrl-c command. Next check and you should see following folders. These folders (except for os) were downloaded by yum-rsync-minimal-centos6.6.sh. Make sure you see the folders exactly as shown below. With an extra character or variable accidentally left in wrong place in yum-rsync-minimal-centos6.6.sh, you can end up with RPMs in wrong folder, breaking yumrepo 6.6.

[[email protected] ~]# ls /data/www/yumrpms/centos6.6/6/ | sort
centosplus
contrib
cr
extras
fasttrack
os
SCL
updates
xen4

[[email protected] ~]# find /data/www/yumrpms/centos6.6/6/ -maxdepth 2 | sort
/data/www/yumrpms/centos6.6/6/
/data/www/yumrpms/centos6.6/6/centosplus
/data/www/yumrpms/centos6.6/6/centosplus/x86_64
/data/www/yumrpms/centos6.6/6/contrib
/data/www/yumrpms/centos6.6/6/contrib/x86_64
/data/www/yumrpms/centos6.6/6/cr
/data/www/yumrpms/centos6.6/6/cr/x86_64
/data/www/yumrpms/centos6.6/6/extras
/data/www/yumrpms/centos6.6/6/extras/x86_64
/data/www/yumrpms/centos6.6/6/fasttrack
/data/www/yumrpms/centos6.6/6/fasttrack/x86_64
/data/www/yumrpms/centos6.6/6/os
/data/www/yumrpms/centos6.6/6/os/x86_64
/data/www/yumrpms/centos6.6/6/SCL
/data/www/yumrpms/centos6.6/6/SCL/x86_64
/data/www/yumrpms/centos6.6/6/updates
/data/www/yumrpms/centos6.6/6/xen4


After verifying the folders are in right place, run /root/bin/yum-rsync-minimal-centos6.6.sh in background with & option. This initial, complete run of the script will take days because the rsync part in the script will need days to download about 14GB of files. You are most likely sshed into yumserver01 from your PC which may not stay on for all those days. Hence by using & option, the /root/bin/yum-rsync-minimal-centos6.6.sh will continue to run in the background even when you log out from yumserver01. If /root/bin/yum-rsync-minimal-centos6.6.sh seems to be hanging, reboot yumserver01 and restart the script. Or kill the process and restart the script.

[[email protected] ~]# /root/bin/yum-rsync-centos6.6-minimal.sh &

Tail /root/log-yum-rsync-minimal-centos6.6.log occasionally to see the status.


It will take days for /root/bin/yum-rsync-centos6.6-minimal.sh to complete because of the large amount of data (about 14G) to rsync dowm from a public yumrepo server. So be patient.

[[email protected] ~]# du -hs /data/www/yumrpms/centos6.6/6.6/
14G	/data/www/yumrpms/centos6.6/6.6/
[[email protected] ~]# du -hs /data/www/yumrpms/centos6.6/6.6/*
1.3G	/data/www/yumrpms/centos6.6/6.6/centosplus
44K	/data/www/yumrpms/centos6.6/6.6/contrib
52K	/data/www/yumrpms/centos6.6/6.6/cr
21M	/data/www/yumrpms/centos6.6/6.6/extras
108M	/data/www/yumrpms/centos6.6/6.6/fasttrack
5.8G	/data/www/yumrpms/centos6.6/6.6/os
408M	/data/www/yumrpms/centos6.6/6.6/SCL
4.7G	/data/www/yumrpms/centos6.6/6.6/updates
976M	/data/www/yumrpms/centos6.6/6.6/xen4

Test with 66-test

Install or find a bare CentOS 6.6 server to test the new yumrepo 6.6. Calling this server 66-test. Make sure no 'yum update' has been done on the server. On 66-test, do following to set it to use only yumrepo 6.6 for all yum commands.

  1. Copy in from yumserver01:/root/bin/66.repo >> 66-test:/etc/yum.repos.d/66.repo.
  2. 66-test: Rename /etc/yum.repos.d/CentOS-Base.repo to /etc/yum.repos.d/CentOS-Base.repo.orig
  3. 66-test: Run 'yum clean all'.
  4. That's it. Now when you run any yum command on 66-test, 66-test will actually get rpms from yumrepo 6.6, not any of the yumrepo servers out in the internet.

Schedule /root/bin/yum-rsync-centos6.6-minimal.sh to run regularly

Once you are done setting up yumrepo 6.6, you may want to schedule a cron job to run yum-rsync-minimal-centos6.6.sh periodically (daily or every 3 days, etc). That way your local yumrepo 6.6 will always have the latest update rpms. Remember, if /root/bin/yum-rsync-minimal-centos6.6.sh is not executed regularly on your local yumrepo server, yumclients on your local LAN will not have the latest update rpms released for CentOS.

Next, how to use yumupdate.sh on yumclients.