Build local a local Yumrepo Server

part 2 of 4

Posted by paul on 2015.03.16

Build local a local Yumrepo Server - part 2 of 4

2015.03.16 Mon

  1. Part 1: Intro
  2. Part 2: Yumrepo 6.6basic - RPMs from DVDs only
  3. Part 3: Yumrepo 6.6 - exact mirror of a full Yumserver
  4. Part 4: How to use yumupdate.sh

Overview

In this part 2, I will show how to create a Yumrepo with rpms from CentOS 6.6 install media only, hence the yumrepo is called 6.6basic. You can name the yumrepo 6.6dvd or anything you like. The yumrepo won't include any updates released after the initial release of CentOS 6.6. The point of this to show how the script works. Remember you should not use un-updated CentOS 6.6 in production use. However if you ever need a CentOS 6.6 server that should not be allowed to install later updates because you are using it for dev/qa/doc purpose, you can use yumrepo 6.6basic. Just make sure the un-updated CentOS server is not accessible from the internet.

Prep a CentOS server

  1. Set up a new CentOS Server (calling it yumserver01.local.loc) with 100GB - 150GB of HD space for /. For yumrepo 6.6basic, we will need only about 20GB but having that much space will allow adding more yumrepos later.
  2. Disable SELinux on yumserver01. Some insist on enabling it. Some say it is not required. For this tutorial, I will disable SElinux for the sake of simplicity.
  3. [[email protected] ~]# vi /etc/selinux/config
    
    # before
    SELINUX=enabled
    
    # after
    SELINUX=disabled
    
    # save /etc/selinux/config and REBOOT yumserver01.
    
  4. Yumserver01: Make sure rsync and openssh-clients are installed. I installed yumserver01 with Minimal option so I needed to install these. If you picked Basic Server option, they are already installed.
  5. [[email protected] ~]# yum install rsync openssh-clients
    
  6. Yumserver01: Create 2 directories on Yumserver01. These are for holding ISO files and scripts.
  7. [[email protected] ~]# mkdir /root/isos
    [[email protected] ~]# mkdir /root/bin
    
  8. Yumserver01: Download CentOS 6.6 install DVD ISO files to /root/isos/. Make sure root owns the ISO files.
  9. [[email protected] ~]# chown -R root:root /root/isos
    [[email protected] ~]# ls -lh /root/isos/
    total 5.6G
    -rw-r--r-- 1 root root 4.4G Feb 26 13:24 CentOS-6.6-x86_64-bin-DVD1.iso
    -rw-r--r-- 1 root root 1.3G Oct 24 07:19 CentOS-6.6-x86_64-bin-DVD2.iso
    
  10. Make sure rsync and openssh-clients are installed. I installed yumserver01 with Minimal install option which means I had to install the 2 rpms after initial install.

Prep script, yum-create-server-centos6.6basic.sh

  1. Yumserver01: Create /root/bin/yum-create-server-centos6.6basic.sh and make it executable. In the code shown below, I have removed most of comments for clarity. I will post latest script with full comments on github (although it's not ready just yet). If possible, copy code from github instead of copying/pasting from this webpage.
  2. #!/bin/bash
    # Run this script to create yumrepo "6.6basic" from scratch on a new CentOS server.
    # It goes from copying files from .ISO files to updating httpd.conf.
    # You need to have 2 ISO files in /root/isos/
    # 1. CentOS-6.6-x86_64-bin-Dvd1.iso
    # 2. CentOS-6.6-x86_64-bin-Dvd2.iso
    
    # Whole process will take about 30 min to 1 hour, depending on the hardware.
    
    
    #####
    # variables
    #####
    
    # Use 1 of the 2 variables (never both) below to specificy hostname of the yumserver.
    # Or use IP for YumServer.
    YumServer="yumserver01.base.loc"
    #YumServer="`hostname -f`"
    
    Version1="6.6"  #Actual version of CentOS: 6.5 or 6.6.
    Version2="${Version1}basic"  #Version of CentOS plus type. ex 6.6basic.
    VersionMajor=${Version1:0:1} # This will be 6 if $Version1 6.6. Or 7 if $Version1 is 7.1.
    
    Isos="/root/isos"
    Dvd1="CentOS-6.6-x86_64-bin-DVD1.iso"
    Dvd2="CentOS-6.6-x86_64-bin-DVD2.iso"
    
    DvdContent="/root/dvdcontent"
    
    Target1="/data/www"
    Target2="yumrpms/centos$Version2"
    Target3="$Version1/os/x86_64"
    Target="$Target1/$Target2/$Target3"
    
    BinFolder="/root/bin"  # where my scripts are kept.
    
    ApacheDir="/data/www/"
    
    TotalSteps="4"
    
    #####
    # variables used in generating $Create
    #####
    Create="yum-createrepo-$Version2.sh"  # yum-createrepo script to be created.
    RepoFile="66basic.repo"  # .repo file to be put into Yum Clients.
    File1="centos$Version1-x86_64-comps.xml" # .xml file to be used for 'createrepo -g' command.
    Rpms1="/data/www/yumrpms/centos$Version2" # where source rpms are.
    Folder1="/data/yumcomps"  # folder to hold 6.xxx_x86_64-comps.xml.
    
    
    #####
    # script
    #####
    
    echo -e "\nStep 1 of $TotalSteps: Copying files from ISO files to $DvdContent/$Version1."
    # Check if 2 .iso files are present
    if [ -f $Isos/$Dvd1 ];
    then
    echo -e "\n$Dvd1 found."
    else
    echo -e "\n$Dvd1 not found. Please copy in $Dvd1 to $Isos and restart this scipt."
    exit 1
    fi
    
    if [ -f $Isos/$Dvd2 ];
    then
    echo -e "\n$Dvd2 found."
    echo -e "\nScript will proceed"
    else
    echo -e "\n$Dvd2 not found. Please copy in $Dvd2 to $Isos and restart this scipt."
    exit 1
    fi
    
    
    # Copy files from DVD to /root/dvdcontent
    mkdir /mnt/cdrom 2>/dev/null
    umount /mnt/cdrom 2>/dev/null
    
    mkdir -p $DvdContent/$Version1 2>/dev/null
    
    mount -o loop $Isos/$Dvd1 /mnt/cdrom
    mkdir $DvdContent/$Version1/1 2>/dev/null
    rsync -a --delete /mnt/cdrom/* $DvdContent/$Version1/1
    
    umount /mnt/cdrom
    
    mount -o loop $Isos/$Dvd2 /mnt/cdrom
    mkdir $DvdContent/$Version1/2 2>/dev/null
    rsync -a --delete /mnt/cdrom/* $DvdContent/$Version1/2
    
    umount /mnt/cdrom
    
    
    # Copy DVD content to $Target
    mkdir -p $Target 2>/dev/null
    rsync -a $DvdContent/$Version1/1/* /$Target/
    rsync -a $DvdContent/$Version1/2/* /$Target/
    
    # link 6.6 to 6
    cd $Target1/$Target2
    if [ ! -h $Target1/$Target2/$VersionMajor ]; then
    cd $Target1/$Target2
    ln -s $Version1 $VersionMajor
    fi
    
    
    echo -e "\nStep 2 of $TotalSteps: Copying centos6.6-x86_64-comps.xml."
    # Copy comps.xml
    mkdir $Folder1 2> /dev/null
    cp $DvdContent/$Version1/1/repodata/*x86_64-comps.xml $Folder1/centos6.6-x86_64-comps.xml
    
    
    
    echo -e "\nStep 3 of $TotalSteps: Configuring Apache."
    # Install httpd and createrepo
    yum -y install httpd createrepo
    
    # Prep Apache
    chkconfig httpd on
    
    # Back up /etc/httpd/conf/httpd.conf
    cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.orig
    
    # Change DocumentRoot to $ApacheDir
    sed 's/var\/www\/html/data\/www/'  -i /etc/httpd/conf/httpd.conf
    
    
    chown -R apache:apache $Target1
    service httpd restart
    
    
    
    
    echo -e "\nStep 4 of $TotalSteps: Generate and execute $Create."
    # Create $Create
    cat <<EOM > $BinFolder/$Create
    #!/bin/bash
    # This script runs createrepo command in all directories with .rpm filddes.
    EOM
    
    find $Rpms1 -name "x86_64" >> $BinFolder/$Create
    
    sed -i "s/\/data/createrepo \/data/g" $BinFolder/$Create
    
    echo "createrepo -g $Folder1/$File1 $Target" >> $BinFolder/$Create
    echo "chown -R apache:apache $Target1/$Target2" >> $BinFolder/$Create
    
    
    
    
    # Prep $RepoFile file.
    echo -e "\nTo configure Yum Client to use this newly created yumrepo:"
    
    echo "###############################"
    echo -e "1. copy yumserver01:$BinFolder/$RepoFile to yumclient:/etc/yum.repos.d/$RepoFile "
    echo -e "\n2. On yumclient: rename /etc/yum.repos.d/CentOS-Base.repo to /etc/yum.repos.d/CentOS-Base.repo.orig"
    echo -e "\n3. On yumclient: run 'yum clean all'. Now you can run 'yum install httpd' or 'yum update'."
    echo "###############################"
    
    
    
    
    cat <<EOM > $BinFolder/$RepoFile
    # $RepoFile
    # Change hostname to IP if you do not have DNS for local machines.
    
    [base]
    name=CentOS-\$releasever - Base
    baseurl=http://$YumServer/yumrpms/centos$Version2/\$releasever/os/\$basearch/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    #END of file
    EOM
    
    
    # Execute $Create
    echo -e "\ncreaterepo command in progress. Will be completed in about 10 - 20 mins.\n\n"
    chmod 750 $BinFolder/$Create
    /bin/bash $BinFolder/$Create
    
  3. The script does following: check if the 2 ISO files are present > mounts them and copies the content of ISO files to /root/dvdcontent/6.6 > copy /root/dvdcontent/6.6/ to /data/www/yumrpms/centos6.6basic/6.6/os/x86_64/ > copy in centos6.6-x86_64-comps.xml > install httpd and createrepo > prep httpd.conf > generate yum-createrepo-v6.6basic.sh > generate 66basic.repo > execute yum-createrepo-v6.6basic.sh.
  4. Again, note 66basic.repo that's created in /root/bin/. The file, 66basic.repo, can be copied to yumclients to configure them to use yumrepo 6.6basic. More on that below.
  5. Save /root/bin/yum-create-server-centos6.6basic.sh and make it executable. Do not run it just yet.

Run yum-create-server-centos6.6basic.sh

  1. Note because the script is in /root/bin/ directory, there's no need to run it as ./yum-create-server-centos6.6basic.sh (with ./). I can just run yum-create-server-centos6.6basic.sh from any directory and the script will run.
  2. Instead of simply running yum-create-server-centos6.6basic.sh, run following command. It will take about 30min or longer to complete, depending on the hardware. The command is all in 1 line but I broke them up with \ for better legibility.
  3. [[email protected] ~]# log1=/root/log-create-yumrepo-66basic.log;\
    yum-create-server-centos6.6basic.sh 2>&1 |\
    tee -a $log1; \
    echo -e "\n# Done" >> $log1
    
  4. ( log1=/root/log-create-yumrepo-66basic.log;\) is for assigning a log file name to a variable, log1.
  5. By running ( yum-create-server-centos6.6basic.sh 2>&1 |\ tee -a $log1;\ ), you will see both standard output and error output on the screen. And the same text will be saved in /root/log-create-yumrepo-66basic.log. By saving all output in a text file, you can review /root/log-create-yumrepo-66basic.log later for troubleshooting. No need to sit and watch the text scroll down.
  6. The last part ( echo -e "\n# Done" >> $log1 ) is to add a note in /root/log-create-yumrepo-66basic.log when the entire command has finished executing. You can simply tail /root/log-create-yumrepo-66basic.log to find out when the script has finished running.
  7. If you see following by tailing /root/log-create-yumrepo-66basic.log, the script has completed running.
  8. ...
    Saving Primary metadata
    Saving file lists metadata
    Saving other metadata
    Generating sqlite DBs
    Sqlite DBs complete
    
    # Done
    [[email protected] ~]#
    
  9. When the script is finished running, try browsing http://yumserver01.local.loc/yumrpms/centos6.6basic/6/os/x86_64/. Replace yumserver01.local.loc with your hostname or IP address in use in your setup. You should see a list of files/folders.
  10. Your new yumrepo 6.6basic is now ready. To set a CentOS server (aka yumclient) to use the yumrepo, you would 1) rename CentOS-Base.repo to CentOS-Base.repo and 2) copy in 66basic.repo. You should not configure a production CentOS server to use the new yumrepo 6.6basic. With production CentOS servers, you need to set up yumrepo 6.6, to be explained in part 3.
  11. To learn how to set a CentOS server to use the yumrepo 6.6basic, follow the testing procedure below.

Optional Test: installing CentOS using yumrepo 6.6basic

This test is optional. Basically on a KVM server, you will install a new instance of CentOS guest using the yumrepo 6.6basic as the source of RPMs. Note in step 2, I use IP, not hostname.

  1. Start installing CentOS 6.6 on your KVM server. Name the install as 66basic-test
  2. Step 1 of 5: Choose Network Install. Click on Forward.
  3. Step 2 of 5: URL: http://192.168.11.115/yumrpms/centos6.6basic/6/os/x86_64/
  4. Step 3 of 5: Click on Forward.
  5. Step 4 of 5: Click on Forward.
  6. Step 5 of 5: Check "Customize configuration before the install". Click on Finish.
  7. In next window, click on "Display VNC". Change Keymap to en-us.
  8. Click on Apply. Click on "Begin installation".
  9. In next window, click on View > Scale Display > Always.
  10. Answer rest of prompts for installing CentOS.
  11. CentOS 6.6 should be installed.

Test a CentOS 6.6 server with yumrepo 6.6basic

Find or set up a system (using server named 66basic-test here) with bare CentOS 6.6 installation, without any updates installed. Make sure 'yum update' has not been done on the server. If you still have not set up a KVM host, this should serve as another reason for setting up a KVM host.

  1. To set a yumclient (that is a CentOS 6.6 server) to use the new yumrepo 6.6basic, you need to 1) rename CentOS-Base.repo to something else and 2) add 66basic.repo, as explained below.
  2. 66basic-test: rename CentOS-Base.repo to CentOS-Base.repo.orig.
  3. [[email protected] ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.orig
    
  4. 66basic-test: try running yum install httpd and it will not work as no valid yumrepo is found.
  5. 66basic-test: create /etc/yum.repos.d/66basic.repo with following. You can also copy it from yumserver01:/root/bin/66basic.repo > 66basic-test:/etc/yum.repos.d/66basic.repo. In 66basic.repo, make sure to change "yumserver01.local.loc" with IP or actual hostname of your Yumrepo server.
  6. # 66basic.repo
    # Change hostname to IP if you do not have DNS for local machines.
    [base]
    name=CentOS-$releasever - Base
    baseurl=http://yumserver01.local.loc/yumrpms/centos6.6basic/$releasever/os/$basearch/
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    #END of file
    
  7. 66basic-test: try to install httpd now and it will succeed.
  8. yumserver01: tail httpd access log and you will see some files that were accessed.
  9. 192.168.......:17 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/repodata/repomd.xml HTTP/1.1" 200 4062 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:18 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/repodata/036dd72d77ba4abf25f5583b7973558f5f7a6bc90ee18be1f27a9d93072313f8-primary.sqlite.bz2 HTTP/1.1" 200 4702749 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/httpd-2.2.15-39.el6.centos.x86_64.rpm HTTP/1.1" 200 844936 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/httpd-tools-2.2.15-39.el6.centos.x86_64.rpm HTTP/1.1" 200 76520 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/apr-util-ldap-1.3.9-3.el6_0.1.x86_64.rpm HTTP/1.1" 200 15824 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/apr-1.3.9-5.el6_2.x86_64.rpm HTTP/1.1" 200 126392 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/mailcap-2.1.31-2.el6.noarch.rpm HTTP/1.1" 200 27280 "-" "urlgrabber/3.9.1 yum/3.2.29"
    192.168.......:36 -0700] "GET /yumrpms/centos6.6basic/6/os/x86_64/Packages/apr-util-1.3.9-3.el6_0.1.x86_64.rpm HTTP/1.1" 200 89340 "-" "urlgrabber/3.9.1 yum/3.2.29"
    
  10. 66basic-test: 'yum update' will return "No Packages marked for Update", as expected. Yumrepo 6.6basic only contains rpms from the CentOS 6.6 install media. It does not contain any updates. To set up a local yumserver that contains all updates as a normal yumrepo server out on internet, follow part 3 .

Recap

At this point, you would have yumserver01.local.loc with just 1 yumrepo, 6.6basic. When you set a CentOS to use that yumrepo 6.6basic (rename CentOS-Base.repo and add 66basic.repo), the yumclient can find rpms included on the install media. But it cannot find any of the rpms that were released as updates after. This is really a pointless setup in a production setting, but it should help you understand how the script works. If for some reason you need CentOS 6.6 running without any updates (dev/qa), the yumrepo 6.6basic should be useful.

Next blog, part 3 , will show you how to set up yumrepo 6.6 which will mirror a normal yumserver. Unlike yumrepo 6.6basic, yumrepo 6.6 will behave exactly as a normal yumrepo server you'd find out in the internet.

If the script doesn't work, please let me know below.

On to part 3, creating yumrepo 6.6.