LinkedInPrint
Posted on by

YUM on Exadata – Local mirror with docker

Heaving own local repository mirror is handy because you can use single source to update multiple database nodes. You will also save your time and avoid mounting ISO files individually on every node.

Consider these two repositories as minimal set for my Exadata db nodes (example taken from my previous post related do YUM):

  1. [exadata_dbserver_12.1.2.3.3_x86_64_base] – baseline of my current Exadata firmware
  2. [ol6_u8_x86_64_base] – base repository for Oracle Linux 6 Update 8.

Unfortunately first is not available on Public Oracle YUM server. It can be created out of ISO files downloaded from Oracle or mirrored from Unbreakable Linux Network (ULN). Second, generic repository is available on both: ULN and public YUM servers (for example here).

What methods can be used to prepare such local mirror with these two repositories?

  1. Use uln-yum-mirror to mirror both repositories from ULN.
  2. Use ISO files downloaded from MOS for Exadata baseline and Oracle Public Yum server for other packages.
  3. Use docker container .

First is well documented here and here, however it has one disadvantage: it requires system with Oracle Linux to host such ULN mirror. While it should not be a problem to create small virtual machine for OL – many Red Hat customers seems to have problem with it. They buy Oracle Exadata but they don’t want to use Oracle Linux anywhere else in their environment.

For such a case two other methods might be effective. Until recently – when I dealt with Red-Hat-only-customer – I used to create local mirror out of ISO file downloaded from MOS together with OL base repository mirrored from Oracle public YUM server using reposync (or also from ISO). This can be done on any Linux which have reposync and createrepo installed (even Debian-based system can deal with it). But I couldn’t figure out how to use reposync with ULN on Red Hat or any other non-Oracle linux distribution.

Recently I’ve discovered that I might use oraclelinux docker image on any Linux distribution to register it in the ULN (!) as an YUM server. The only requirement is the system with docker support, for example Red Hat 7 and ULN subscription (which you have if you are Exadata customer).

Setting up docker

It might be a case that you don’t have docker installed. On Red Hat 7 this can be done this way:

[root@rhel ~]# subscription-manager repos --enable=rhel-7-server-rpms
[root@rhel ~]# subscription-manager repos --enable=rhel-7-server-extras-rpms
[root@rhel ~]# yum install docker
[root@rhel ~]# systemctl enable docker
[root@rhel ~]# systemctl start docker
[root@rhel ~]# docker info

Create directory structure to hold persistenta data and repo files

Here I’m going to use directory on my host which will be mounted as a volume into docker container. This is just easier to start with but later I’ll show how to keep this data inside separate docker volume – which what you should actually do when working with docker. 

[root@rhel ~]# mkdir -p /yum/conf
[root@rhel ~]# mkdir -p /yum/repos
[root@rhel ~]# mkdir -p /yum/persist
[root@rhel ~]# mkdir -p /yum/cache
[root@rhel ~]# chcon -Rt svirt_sandbox_file_t /yum
[root@rhel ~]# cat >/yum/conf/yum.conf <<EOF
[main]
logfile=/var/log/yum.log
gpgcheck=1
plugins=1
pluginconfpath=/yum/conf
persistdir=/yum/persist
cachedir=/yum/cache
EOF
cat >/yum/conf/rhnplugin.conf <<EOF
[main]
enabled=1
gpgcheck=1
EOF
cat >/yum/conf/ulninfo.conf <<EOF
[main]
enabled=1
gpgcheck=1
EOF

FYI: highlighted command is one of the ways to avoid protection SELinux imposes on docker accessing host directory.

Create and start container

Now let’s start container using oraclelinux image. It should be automatically located in official docker repository, downloaded and used to run container called “uln-mirror”:

[root@rhel ~]# docker run -it \
  --name "uln-mirror" \
  -h "uln-mirror" \
  -v /yum:/yum \
  oraclelinux

Once started you get the prompt of the bash running inside new container. You should also have /yum directory mounted from host with files created in previous step.

[root@uln-mirror /]# df -h
Filesystem                  Size  Used Avail Use% Mounted on
overlay                      37G   15G   23G  39% /
tmpfs                       920M     0  920M   0% /dev
tmpfs                       920M     0  920M   0% /sys/fs/cgroup
/dev/mapper/rhel_rhel-root   37G   15G   23G  39% /yum
shm                          64M     0   64M   0% /dev/shm
tmpfs                       920M     0  920M   0% /proc/acpi
tmpfs                       920M     0  920M   0% /proc/scsi
tmpfs                       920M     0  920M   0% /sys/firmware
[root@uln-mirror /]# find /yum
/yum
/yum/conf
/yum/conf/yum.conf
/yum/conf/rhnplugin.conf
/yum/conf/ulninfo.conf
/yum/repos
/yum/persist
/yum/cache
[root@uln-mirror /]# cat /etc/oracle-release 
Oracle Linux Server release 7.6

Register docker container as YUM server

So, despite the fact that I’m running Red Hat Enterprise Linux 7.6 – I do have an container on it which identifies itself as Oracle Linux 7.6. Moreover uln_register command is available there, so why not try to register it in ULN. But before executing it, I need to make my registration persistent because everything changed inside container will be lost after its removal. I do this by modifying /etc/sysconfig/rhn/up2date config file where I change systemIdPath value to /yum/conf/systemid (and this step needs to be done every time new container is created from the image). The systemid file will be created upon successful registration and If I keep it – I will be able to recreate my setup with another container.

[root@uln-mirror /]# sed -i 's|^systemIdPath=.*$|systemIdPath=/yum/conf/systemid|g' /etc/sysconfig/rhn/up2date
[root@uln-mirror /]# uln_register
[root@uln-mirror /]# ls -l /yum/conf/systemid
-rw-------. 1 root root 1394 06-01 18:19 /yum/conf/systemid

Now, let’s modify my registration so that my container is treated as YUM server (necessary to subscribe any channel from ULN). Also let’s subscribe to repositories I require on my Exadata db nodes:

[root@uln-mirror /]# uln-channel --enable-yum-server
[root@uln-mirror /]# uln-channel -a -c exadata_dbserver_12.1.2.3.3_x86_64_base -c ol6_u8_x86_64_base
Username: *******
Password: *******
[root@uln-mirror /]# uln-channel -l
exadata_dbserver_12.1.2.3.3_x86_64_base
ol6_u8_x86_64_base
ol7_x86_64_UEKR5
ol7_x86_64_latest

As you can see now I’m subscribed to default OL7 repositories and to these two I explicitly added.

Synchronize repositories

Here is where my /yum/conf/yum.conf and other files created previously starts to be important. I need to download (differentially if this is consecutive try) repositories into /yum/repos directory and use persistent cache – so I don’t use default config files from /etc/yum* (that would be deleted anyway as I’m inside container).

[root@uln-mirror /]# reposync -c /yum/conf/yum.conf -l -m --download-metadata -p /yum/repos -r exadata_dbserver_12.1.2.3.3_x86_64_base -r ol6_u8_x86_64_base
...
Loaded plugins: rhnplugin
This system is receiving updates from ULN.
exadata_dbserver_12.1.2.3.3_x86_64_base            | 1.2 kB  00:00:00     
exadata_dbserver_12.1.2.3.3_x86_64_base/updateinfo |  153 B  00:00:00     
exadata_dbserver_12.1.2.3.3_x86_64_base/primary    | 477 kB  00:00:02     
ol6_u8_x86_64_base                                 | 1.4 kB  00:00:00     
ol6_u8_x86_64_base/group 
(1/446): Lib_Utils-1.00-09.noarch.rpm              | 1.5 MB  00:00:05     
(2/446): MAKEDEV-3.24-6.el6.x86_64.rpm             |  88 kB  00:00:00     
...

It works and repositories are being downloaded into /yum/repos. After process finishes the only task left is to create repository out of them:

[root@uln-mirror /]# createrepo -v /yum/repos/exadata_dbserver_12.1.2.3.3_x86_64_base
[root@uln-mirror /]# createrepo -v /yum/repos/ol6_u8_x86_64_base

and publish /yum/repos directory using web server, for example Apache as it is described in official documentation.

Better (automated) way

Entire process can be automated and improved, so it could be more aligned to docker philosophy. For example you can use one image for registration and download and another one for publishing repositories locally using HTTP service inside docker container.

For that purpose I created working example of such setup. There is also more information about usage under the link. Below is an example of how it can be used (this is done on my workstation which runs on Ubuntu)

Step 1 – Register

Register creates prepares the image and starts container which will ask you for ULN credentials and CSI number. You only provides docker volume name which will be created (if does not exists yet) or mounted (if you run registration multiple times with the same parameter).

michal@sunman:/tmp$ git clone https://github.com/wes-pro/docker-uln-mirror.git
Cloning into 'docker-uln-mirror'...
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 37 (delta 14), reused 27 (delta 9), pack-reused 0
Unpacking objects: 100% (37/37), done.
Checking connectivity... done.

michal@sunman:/tmp$ cd docker-uln-mirror
michal@sunman:/tmp$ echo exadata_dbserver_12.1.2.3.3_x86_64_base>repo_list
michal@sunman:/tmp$ echo ol6_u8_x86_64_base>>repo_list

michal@sunman:/tmp/docker-uln-mirror$ ./register.sh uln-vol
latest: Pulling from library/oraclelinux
a010df13e11b: Pull complete 
Digest: sha256:f36ae2dff1e4aa8420ca3c3403afa84ab1b77005d164213f5cd02ebd4735ea2c
Status: Downloaded newer image for oraclelinux:latest
uln-vol
Sending build context to Docker daemon  139.8kB
Step 1/6 : FROM oraclelinux:latest
 ---> ccafb8385951
Step 2/6 : RUN yum install -y createrepo
 ---> Running in 60c3e773f62e
Loaded plugins: ovl, ulninfo
Resolving Dependencies
--> Running transaction check
---> Package createrepo.noarch 0:0.9.9-28.el7 will be installed
--> Processing Dependency: python-deltarpm for package: createrepo-0.9.9-28.el7.noarch
--> Processing Dependency: deltarpm for package: createrepo-0.9.9-28.el7.noarch
--> Running transaction check
---> Package deltarpm.x86_64 0:3.6-3.el7 will be installed
---> Package python-deltarpm.x86_64 0:3.6-3.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package               Arch         Version              Repository        Size
================================================================================
Installing:
 createrepo            noarch       0.9.9-28.el7         ol7_latest        93 k
Installing for dependencies:
 deltarpm              x86_64       3.6-3.el7            ol7_latest        81 k
 python-deltarpm       x86_64       3.6-3.el7            ol7_latest        30 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Total download size: 205 k
Installed size: 558 k
Downloading packages:
--------------------------------------------------------------------------------
Total                                              377 kB/s | 205 kB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : deltarpm-3.6-3.el7.x86_64                                    1/3 
  Installing : python-deltarpm-3.6-3.el7.x86_64                             2/3 
  Installing : createrepo-0.9.9-28.el7.noarch                               3/3 
  Verifying  : createrepo-0.9.9-28.el7.noarch                               1/3 
  Verifying  : deltarpm-3.6-3.el7.x86_64                                    2/3 
  Verifying  : python-deltarpm-3.6-3.el7.x86_64                             3/3 

Installed:
  createrepo.noarch 0:0.9.9-28.el7                                              

Dependency Installed:
  deltarpm.x86_64 0:3.6-3.el7         python-deltarpm.x86_64 0:3.6-3.el7        

Complete!
Removing intermediate container 60c3e773f62e
 ---> dc0d8c389578
Step 3/6 : COPY entrypoint.sh /
 ---> eadb36308c45
Step 4/6 : COPY repo_list /
 ---> 36827290e9fa
Step 5/6 : ENTRYPOINT ["/entrypoint.sh"]
 ---> Running in 49c56043df57
Removing intermediate container 49c56043df57
 ---> f038f0b3b633
Step 6/6 : CMD ["download"]
 ---> Running in 4329ea7418ef
Removing intermediate container 4329ea7418ef
 ---> 58041752999c
Successfully built 58041752999c
Successfully tagged uln-mirror:latest
Your ULN username: ****
Your ULN password: ****
Your CSI: ****
Registering your system... Please wait...
Successfully registered your system. File: /uln/conf/systemid has been created.
List of available repositories has been written to /uln/conf/repo_available

Subscribed to repositories:
exadata_dbserver_12.1.2.3.3_x86_64_base
ol6_u8_x86_64_base

  Docker image "uln-mirror" and volume "uln-vol" created. 

  If you haven't done so yet, put channel names you want to download into 
  repo_list file and rerun register.sh script with the same volume name. 
  You can find list of all available repositories in repo_available file.
  Re-running register.sh script with existing volume - still asks for username
  and password - these are required by uln-channel to modify channel list.

  Then run below command to start docker and which will download selected repositories: 
  ./download.sh <volume-name>

  To publish downloaded repositories using HTTP server start another docker using:
  ./publish.sh <volume-name>

It is possible to execute register.sh multiple times with the same argument. The script behaves slightly differently when it finds registration data on volume. It only asks for Oracle credentials – but not for CSI – and use them to subscribe channels from repo_list file. This is for the case when you want to start downloading new channels which were not selected during initial registration.

Step 2 – Download

Download uses the same image and volume and assumes you already registered the system in ULN and registration data is written into systemid file on the volume. It just download packages from ULN (or update them differentially) and then it recreates local repositories.

michal@sunman:/tmp/docker-uln-mirror$ ./download.sh uln-vol
bf32078aab785acdb79c0204439e575f76e266a68cc93d29d12e8b1d8daabf3b
michal@sunman:/tmp/docker-uln-mirror$ docker logs -f bf32078aab785acdb79c0204439e575f76e266a68cc93d29d12e8b1d8daabf3b
Loaded plugins: ovl, rhnplugin
This system is receiving updates from ULN.
exadata_dbserver_12.1.2.3.3_x86_64_base                  | 1.2 kB     00:00     
exadata_dbserver_12.1.2.3.3_x86_64_base/updateinfo       |  153 B     00:00     
exadata_dbserver_12.1.2.3.3_x86_64_base/primary          | 477 kB     00:02     
ol6_u8_x86_64_base                                       | 1.4 kB     00:00     
ol6_u8_x86_64_base/group                                 | 1.2 MB     00:04     
ol6_u8_x86_64_base/updateinfo                            | 1.5 MB     00:05  
...

Step 3 – Publish

Publish does not use oraclelinux docker image. Instead it uses official nginx image which is just web server running inside docker. The container will start publishing downloaded repositories on port 8080.

michal@sunman:/tmp/docker-uln-mirror$ ./publish.sh uln-vol
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
743f2d6c1f65: Pull complete 
6bfc4ec4420a: Pull complete 
688a776db95f: Pull complete 
Digest: sha256:23b4dcdf0d34d4a129755fc6f52e1c6e23bb34ea011b315d87e193033bcd1b68
Status: Downloaded newer image for nginx:latest
1373451bb76aaf6dbda1a610d0f3dd6f4a7a5d77784b4b058f146c01c0a97cec

You can check that last step created docker container with NGINX web server which publishes downloaded repositories on port 8080:

michal@sunman:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
1373451bb76a        nginx               "nginx -g 'daemon of…"   13 hours ago        Up 13 hours         0.0.0.0:8080->80/tcp   uln-mirror-nginx
bf32078aab78        uln-mirror:latest   "/entrypoint.sh down…"   13 hours ago        Up 13 hours                                uln-mirror

Check it out using browser:

From now on you can just point your compute nodes to this local repository. For example:

[root@exa01vm03 ~]# cat /etc/yum.repos.d/exadata.repo
[exadata_ol6_base_repo_12.1.2.3.3]
name=Exadata release 12.1.2.3.3 db server installation packages (x86_64)
baseurl=http://yum-mirror:8080/exadata_dbserver_12.1.2.3.3_x86_64_base
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
[ol6_u8_x86_64_base]
name=Oracle Linux 6 Update 8 installation media copy (x86_64)
baseurl=http://yum-mirror:8080/ol6_u8_x86_64_base
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=0

Published by Michal

Leave a Reply

Your email address will not be published. Required fields are marked *

The following GDPR rules must be read and accepted:
This form collects your name and email so that I can keep track of the comments placed on the website. I do not share this data with any organisation or person. Your IP address is not collected and will not be displayed with your comment.