Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9867

Set up development environment on NCSA OpenStack

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      A development playground (multi-node, consisting of czar, several workers, and some test data) is needed for prototyping data distribution and replication. It is anticipated that this will be a non-containerized deployment on NCSA OpenStack VMs in order to facilitate rapid development.

      The general plan for this effort includes the following steps:

      1. create and configure 1 master and 10 worker nodes
      2. external access to the cluster via the Floating IP of the master node
      3. user accounts and security: create user qserv, configure ssh, etc.
      4. allocate 11 volumes (one per node) for data and databases
      5. setup an NFS server on master to export some local folder to all workers for sharing LSST Stack and other relevant LSST packages
      6. setup the minimally required *RPM*s to support the software deployment and development efforts
      7. build a local copy of the LSST stack and other relevant package on the exported volume
      8. configure and start Qserv (using Docker container) on all nodes
      9. IMPORTANT: make a snapshot of the VMs of all instances
      10. import a data set (a subset of chunks of PDAC's wise or stripe82)

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment -

            Finished the set up as planned. Loaded the cluster with 1/30th of the wise_00 catalog (of PDAC). Tested basic functionality of the setup.

            Show
            gapon Igor Gaponenko added a comment - Finished the set up as planned. Loaded the cluster with 1/30th of the wise_00 catalog (of PDAC ). Tested basic functionality of the setup.
            Hide
            gapon Igor Gaponenko added a comment -

            Extra steps

            iostat, etc.

            This tool provides very valuable stats for various performance testings. Installed it on all nodes of the cluster:

            % sudo yum install sysstat
            

            Login scripts

            Added the following line to user gapon's login script on all nodes of the cluster:

            % tail -1 /home/gapon/.bash_profile 
            exec scl enable devtoolset-3 sclo-git25 bash
            

            Show
            gapon Igor Gaponenko added a comment - Extra steps iostat, etc. This tool provides very valuable stats for various performance testings. Installed it on all nodes of the cluster: % sudo yum install sysstat Login scripts Added the following line to user gapon 's login script on all nodes of the cluster: % tail -1 /home/gapon/ .bash_profile exec scl enable devtoolset-3 sclo-git25 bash
            Hide
            gapon Igor Gaponenko added a comment - - edited

            2018-01-19: devtoolset-6 and the latest version of LSST Stack

            Upgraded to devtoolset-6 on all nodes of the cluster in order to support the latest version of the LSST Stack:

            % sudo yum install devtoolset-6
            

            After that one has to use:

            % exec scl enable devtoolset-6 sclo-git25 bash
            

            LSST Stack which includes XrootD SSI v2 has been reinstalled at:

            [gapon@lsst-gapon-qserv-worker-1 ~]$ ls -al /software/stack/
            total 84
            drwxr-xr-x. 6 gapon gapon  4096 Jan 18 02:10 .
            drwxr-xr-x. 4 root  root     36 Mar 22  2017 ..
            drwxrwxr-x. 3 gapon gapon    23 Jan 18 02:09 _build
            drwxrwxr-x. 3 gapon gapon    32 Jan 18 02:10 eups
            -rw-rw-r--. 1 gapon gapon 35442 Jan 18 02:10 eupsbuild.log
            -rw-rw-r--. 1 gapon gapon   486 Jan 18 02:10 loadLSST.bash
            -rw-rw-r--. 1 gapon gapon   522 Jan 18 02:10 loadLSST.csh
            -rw-rw-r--. 1 gapon gapon   478 Jan 18 02:10 loadLSST.ksh
            -rw-rw-r--. 1 gapon gapon   450 Jan 18 02:10 loadLSST.zsh
            -rw-rw-r--. 1 gapon gapon 28034 Jan 18 02:02 newinstall.sh
            drwxrwxr-x. 3 gapon gapon    44 Jan 18 02:07 python
            drwxrwxr-x. 3 gapon gapon    52 Jan 18 02:10 stack
            

            NOTE: this folder is ls available on all worker nodes via NFS mount points.

            Show
            gapon Igor Gaponenko added a comment - - edited 2018-01-19: devtoolset-6 and the latest version of LSST Stack Upgraded to devtoolset-6 on all nodes of the cluster in order to support the latest version of the LSST Stack: % sudo yum install devtoolset-6 After that one has to use: % exec scl enable devtoolset-6 sclo-git25 bash LSST Stack which includes XrootD SSI v2 has been reinstalled at: [gapon@lsst-gapon-qserv-worker-1 ~]$ ls -al /software/stack/ total 84 drwxr-xr-x. 6 gapon gapon 4096 Jan 18 02:10 . drwxr-xr-x. 4 root root 36 Mar 22 2017 .. drwxrwxr-x. 3 gapon gapon 23 Jan 18 02:09 _build drwxrwxr-x. 3 gapon gapon 32 Jan 18 02:10 eups -rw-rw-r--. 1 gapon gapon 35442 Jan 18 02:10 eupsbuild.log -rw-rw-r--. 1 gapon gapon 486 Jan 18 02:10 loadLSST. bash -rw-rw-r--. 1 gapon gapon 522 Jan 18 02:10 loadLSST.csh -rw-rw-r--. 1 gapon gapon 478 Jan 18 02:10 loadLSST.ksh -rw-rw-r--. 1 gapon gapon 450 Jan 18 02:10 loadLSST.zsh -rw-rw-r--. 1 gapon gapon 28034 Jan 18 02:02 newinstall.sh drwxrwxr-x. 3 gapon gapon 44 Jan 18 02:07 python drwxrwxr-x. 3 gapon gapon 52 Jan 18 02:10 stack NOTE : this folder is ls available on all worker nodes via NFS mount points.
            Hide
            gapon Igor Gaponenko added a comment -

            Replacing master node, shrinking the cluster by 2 worker nodes and reconfiguring storage

            2018-03-06: Replacing lsst-gapon-qserv-master-01 with lsst-gapon-qserv-master-03

            NOTE: this change was made after an unsuccessful attempt to resize the instance. As a result of the incident Nebula reset the original state of master node dated back to March 2017. It turned out this operation is not supported in Nebula. Fortunately, a few days earlier (2018-03-02) before the incident I did a snapshot of the master node which was called lsst-gapon-qserv-master-01. Hence a solution was to create another master node and initialize it from that snapshot.

            Actions:

            • build a larger instance (160 GB disk, 16 GB RAM, 8 cores) to be initialized from a snapshot of lsst-gapon-qserv-master-01.
            • detach storage volume lsst-gapon-qserv-master from lsst-gapon-qserv-master-01 and attach it to lsst-gapon-qserv-master-03.
            • reconfigure /etc/hosts on all nodes of the cluster due to a change in the name and IP address of the master node
            • reconfigure shmux-based Qserv management tools on /software/development/qserv/admin/tools/docker/deployment/os/

            2018-03-12: Rebuilding lsst-gapon-qserv-worker-04

            NOTE: this change had to be made after a regular maintenance of Nebula at NCSA which turned this node into an unrecoverable state. This issue was reported in [IHS-593]. As of today no progress has been made in that area.

            Actions:

            • rename lsst-gapon-qserv-worker-04 into lsst-gapon-qserv-worker-04-saved
            • created a new version of lsst-gapon-qserv-worker-04
            • detach storage volume lsst-gapon-qserv-db-4 from the older node and attach it to the new one
            • update RPMs, install minimally required RPMs, install devtoolset-6
            • reconfigure /etc/hosts on all nodes of the cluster due to a change in the name and IP address of the node

            2018-03-29: Temporarily evicting lsst-gapon-qserv-worker-1 and lsst-gapon-qserv-worker-8

            NOTE: this change had to be made after a regular maintenance of Nebula at NCSA which turned this node into an unrecoverable state. This issue was also reported in [IHS-593]. As of today no progress has been made in that area.

            Actions:

            • reconfigure shmux-based Qserv management tools on /software/development/qserv/admin/tools/docker/deployment/os/
            • disable these nodes in the Replication system configuration

            2018-03-29: Migrating /qserv storage of worker nodes 2,3,4,5,6,7,9,10 to larger volumes

            This step is needed to allow more options when testing the replication system. Here is the volume replacement map (note that two failed nodes are presently excluded from that map):

            host old volume size new volume size
            lsst-gapon-qserv-worker-1 lsst-gapon-qserv-db-1 128 GB n/a  
            lsst-gapon-qserv-worker-2 lsst-gapon-qserv-db-2 128 GB lsst-gapon-qserv-db-12 256 GB
            lsst-gapon-qserv-worker-3 lsst-gapon-qserv-db-3 128 GB lsst-gapon-qserv-db-13 256 GB
            lsst-gapon-qserv-worker-4 lsst-gapon-qserv-db-4 128 GB lsst-gapon-qserv-db-14 256 GB
            lsst-gapon-qserv-worker-5 lsst-gapon-qserv-db-5 128 GB lsst-gapon-qserv-db-15 256 GB
            lsst-gapon-qserv-worker-6 lsst-gapon-qserv-db-6 128 GB lsst-gapon-qserv-db-16 256 GB
            lsst-gapon-qserv-worker-7 lsst-gapon-qserv-db-7 128 GB lsst-gapon-qserv-db-17 256 GB
            lsst-gapon-qserv-worker-8 lsst-gapon-qserv-db-8 128 GB n/a
            lsst-gapon-qserv-worker-9 lsst-gapon-qserv-db-9 128 GB lsst-gapon-qserv-db-19 256 GB
            lsst-gapon-qserv-worker-10 lsst-gapon-qserv-db-10 128 GB lsst-gapon-qserv-db-20 256 GB

            The main idea was to:

            • stop Qserv or any other activities using old volume /qserv before the operation
            • rename mount points for older volumes from /qserv to /qserv_old
            • mount the larger volumes as /qserv

            Storage volumes are attached to each node as block device /dev/vdc:

            % ls -al /dev/vd*
            brw-rw----. 1 root disk 253,  0 Mar 29 18:58 /dev/vda
            brw-rw----. 1 root disk 253,  1 Mar 29 18:58 /dev/vda1
            brw-rw----. 1 root disk 253, 16 Mar 29 18:58 /dev/vdb
            brw-rw----. 1 root disk 253, 32 Mar 29 19:29 /dev/vdc
            

            Re-mount the old volume with the different path:

            sudo umount /qserv
            sudo mkdir /qserv_old
            sudo chown qserv:qserv /qserv_old
            sudo mount /dev/vdb  /qserv_old
            

            Make mods to this file (using lsst-gapon-qserv-worker-2 as an example):

            cat /etc/fstab | grep qserv_old
            UUID=14bd49fb-28b9-4a49-8352-d2768fb7bde3 /qserv_old    ext4 defaults 0 0
            

            Initialize the new file system, mount it on a folder owned by user qserv (assuming sudo below):

            mkfs.ext4 /dev/vdc
            mount /dev/vdc  /qserv
            df -h | grep qserv | grep vd
            /dev/vdb                              126G   92G   28G  77% /qserv_old
            /dev/vdc                              252G   61M  239G   1% /qserv
            

            Add the new file to /etc/fstab to ensure it's automatically mounted after system reboot (using the lsst-gapon-qserv-worker-2 node as an example. Do the same on all worker nodes):

            sudo dumpe2fs /dev/vdc | grep UUID
            dumpe2fs 1.42.9 (28-Dec-2013)
            Filesystem UUID:          dad5999c-ee22-46e8-b691-e969ca063b1e
             
            sudo vim /etc/fstab
            ..
            UUID=14bd49fb-28b9-4a49-8352-d2768fb7bde3 /qserv_old    ext4 defaults 0 0
            UUID=dad5999c-ee22-46e8-b691-e969ca063b1e /qserv        ext4 defaults 0 0
            ..
            sudo umount /qserv
            sudo mount -a
            df -h | grep qserv | grep vd
            /dev/vdb                              126G   92G   28G  77% /qserv_old
            /dev/vdc                              252G   61M  239G   1% /qserv
            

            Change the owner:

            sudo chown qserv:qserv /qserv
            

            Tested the performance of the storage volume by:

            /bin/sudo -u qserv -i
            cd /qserv
             
            dd bs=1M count=1000 if=/dev/zero of=001.dat oflag=direct
            1048576000 bytes (1.0 GB) copied, 6.95146 s, 151 MB/s
             
            ls -al 001.dat
            -rw-rw-r--. 1 qserv qserv 1048576000 Mar 29 20:58 001.dat
             
            rm 001.dat
            

            Copy over the content of the old disk to the new one:

            /bin/sudo -i
            pwd
            /root
             
            nohup cp -arvf /qserv_old/* /qserv >& cp_qserv_old_qserv.log&
            

            REPEAT: the same sequence of steps for nodes 3,4,5,6,7,9,10

            Show
            gapon Igor Gaponenko added a comment - Replacing master node, shrinking the cluster by 2 worker nodes and reconfiguring storage 2018-03-06: Replacing lsst-gapon-qserv-master-01 with lsst-gapon-qserv-master-03 NOTE : this change was made after an unsuccessful attempt to resize the instance. As a result of the incident Nebula reset the original state of master node dated back to March 2017. It turned out this operation is not supported in Nebula. Fortunately, a few days earlier (2018-03-02) before the incident I did a snapshot of the master node which was called lsst-gapon-qserv-master-01 . Hence a solution was to create another master node and initialize it from that snapshot. Actions: build a larger instance (160 GB disk, 16 GB RAM, 8 cores) to be initialized from a snapshot of lsst-gapon-qserv-master-01 . detach storage volume lsst-gapon-qserv-master from lsst-gapon-qserv-master-01 and attach it to lsst-gapon-qserv-master-03 . reconfigure /etc/hosts on all nodes of the cluster due to a change in the name and IP address of the master node reconfigure shmux -based Qserv management tools on /software/development/qserv/admin/tools/docker/deployment/os/ 2018-03-12: Rebuilding lsst-gapon-qserv-worker-04 NOTE : this change had to be made after a regular maintenance of Nebula at NCSA which turned this node into an unrecoverable state. This issue was reported in [IHS-593] . As of today no progress has been made in that area. Actions: rename lsst-gapon-qserv-worker-04 into lsst-gapon-qserv-worker-04-saved created a new version of lsst-gapon-qserv-worker-04 detach storage volume lsst-gapon-qserv-db-4 from the older node and attach it to the new one update RPMs, install minimally required RPMs, install devtoolset-6 reconfigure /etc/hosts on all nodes of the cluster due to a change in the name and IP address of the node 2018-03-29: Temporarily evicting lsst-gapon-qserv-worker-1 and lsst-gapon-qserv-worker-8 NOTE : this change had to be made after a regular maintenance of Nebula at NCSA which turned this node into an unrecoverable state. This issue was also reported in [IHS-593] . As of today no progress has been made in that area. Actions: reconfigure shmux -based Qserv management tools on /software/development/qserv/admin/tools/docker/deployment/os/ disable these nodes in the Replication system configuration 2018-03-29: Migrating /qserv storage of worker nodes 2,3,4,5,6,7,9,10 to larger volumes This step is needed to allow more options when testing the replication system. Here is the volume replacement map (note that two failed nodes are presently excluded from that map): host old volume size new volume size lsst-gapon-qserv-worker-1 lsst-gapon-qserv-db-1 128 GB n/a   lsst-gapon-qserv-worker-2 lsst-gapon-qserv-db-2 128 GB lsst-gapon-qserv-db-12 256 GB lsst-gapon-qserv-worker-3 lsst-gapon-qserv-db-3 128 GB lsst-gapon-qserv-db-13 256 GB lsst-gapon-qserv-worker-4 lsst-gapon-qserv-db-4 128 GB lsst-gapon-qserv-db-14 256 GB lsst-gapon-qserv-worker-5 lsst-gapon-qserv-db-5 128 GB lsst-gapon-qserv-db-15 256 GB lsst-gapon-qserv-worker-6 lsst-gapon-qserv-db-6 128 GB lsst-gapon-qserv-db-16 256 GB lsst-gapon-qserv-worker-7 lsst-gapon-qserv-db-7 128 GB lsst-gapon-qserv-db-17 256 GB lsst-gapon-qserv-worker-8 lsst-gapon-qserv-db-8 128 GB n/a lsst-gapon-qserv-worker-9 lsst-gapon-qserv-db-9 128 GB lsst-gapon-qserv-db-19 256 GB lsst-gapon-qserv-worker-10 lsst-gapon-qserv-db-10 128 GB lsst-gapon-qserv-db-20 256 GB The main idea was to: stop Qserv or any other activities using old volume /qserv before the operation rename mount points for older volumes from /qserv to /qserv_old mount the larger volumes as /qserv Storage volumes are attached to each node as block device /dev/vdc : % ls -al /dev/vd * brw-rw----. 1 root disk 253, 0 Mar 29 18:58 /dev/vda brw-rw----. 1 root disk 253, 1 Mar 29 18:58 /dev/vda1 brw-rw----. 1 root disk 253, 16 Mar 29 18:58 /dev/vdb brw-rw----. 1 root disk 253, 32 Mar 29 19:29 /dev/vdc Re-mount the old volume with the different path: sudo umount /qserv sudo mkdir /qserv_old sudo chown qserv:qserv /qserv_old sudo mount /dev/vdb /qserv_old Make mods to this file (using lsst-gapon-qserv-worker-2 as an example): cat /etc/fstab | grep qserv_old UUID=14bd49fb-28b9-4a49-8352-d2768fb7bde3 /qserv_old ext4 defaults 0 0 Initialize the new file system, mount it on a folder owned by user qserv (assuming sudo below): mkfs.ext4 /dev/vdc mount /dev/vdc /qserv df -h | grep qserv | grep vd /dev/vdb 126G 92G 28G 77% /qserv_old /dev/vdc 252G 61M 239G 1% /qserv Add the new file to /etc/fstab to ensure it's automatically mounted after system reboot (using the lsst-gapon-qserv-worker-2 node as an example. Do the same on all worker nodes): sudo dumpe2fs /dev/vdc | grep UUID dumpe2fs 1.42.9 (28-Dec-2013) Filesystem UUID: dad5999c-ee22-46e8-b691-e969ca063b1e   sudo vim /etc/fstab .. UUID=14bd49fb-28b9-4a49-8352-d2768fb7bde3 /qserv_old ext4 defaults 0 0 UUID=dad5999c-ee22-46e8-b691-e969ca063b1e /qserv ext4 defaults 0 0 .. sudo umount /qserv sudo mount -a df -h | grep qserv | grep vd /dev/vdb 126G 92G 28G 77% /qserv_old /dev/vdc 252G 61M 239G 1% /qserv Change the owner: sudo chown qserv:qserv /qserv Tested the performance of the storage volume by: /bin/sudo -u qserv -i cd /qserv   dd bs=1M count=1000 if = /dev/zero of=001.dat oflag=direct 1048576000 bytes (1.0 GB) copied, 6.95146 s, 151 MB /s   ls -al 001.dat -rw-rw-r--. 1 qserv qserv 1048576000 Mar 29 20:58 001.dat   rm 001.dat Copy over the content of the old disk to the new one: /bin/sudo -i pwd /root   nohup cp -arvf /qserv_old/ * /qserv >& cp_qserv_old_qserv.log& REPEAT : the same sequence of steps for nodes 3,4,5,6,7,9,10
            Hide
            gapon Igor Gaponenko added a comment - - edited

            2018-03-30: Replacing lsst-gapon-qserv-worker-1 and lsst-gapon-qserv-worker-8 with new instances

            Replacing instances

            The old instances were renamed:

            • lsst-gapon-qserv-worker-1 to lsst-gapon-qserv-worker-1-bad
            • lsst-gapon-qserv-worker-8 to lsst-gapon-qserv-worker-8-bad

            New instances were created and sized similarly to the bad ones.

            Attaching volumes

            NOTE: ideally it was much simple to transfer (detach from the older ones and attach to the new ones) volumes lsst-gapon-qserv-db-1 and lsst-gapon-qserv-db-8 (those which have properly configured Qserv data directory /qserv). Unfortunately this was not possible due to a strange state in which both bad instances are within the OpenStack service. Hence, the idea is to borrow Qserv data snapshot of some other (healthy) workers.

            Attached existing (borrowed) volumes as /dev/vdb to the instances.
            Created two 256 GB volumes for new instances and attached them as /dev/vdc to the instances

            Here is a map of instances

            host borrowed volume: /qserv_old size own new volume: /qserv size
            lsst-gapon-qserv-worker-1 lsst-gapon-qserv-db-2 128 GB lsst-gapon-qserv-db-11 256 GB
            lsst-gapon-qserv-worker-8 lsst-gapon-qserv-db-3 128 GB lsst-gapon-qserv-db-18 256 GB

            Configuring the master node

            • update IP addresses of the new instances at /etc/hosts

            Configuring the instances as explained earlier

            •   update RPMs
            •  install RPMs
            •  install devtoolset-6
            •  setup accounts
            •  configure /etc/hosts
            •  mount and initialize external volumes
            •   configure /etc/fstab
            •  configure NFS
            •  setup and configure the latest version of Docker

            Borrowing data from /qserv_old to /qserv

            •   copy files from /qserv_old to /qserv
            • reconfigure Qserv to add these nodes back
            •  pull Docker containers to the nodes
            • start Qserv
            • fix worker ID on the nodes
            • rebuild the empty chunks list for database wise_00
            • restart Qserv
            • test Qserv by launching a few queries
            Show
            gapon Igor Gaponenko added a comment - - edited 2018-03-30: Replacing lsst-gapon-qserv-worker-1 and lsst-gapon-qserv-worker-8 with new instances Replacing instances The old instances were renamed: lsst-gapon-qserv-worker-1 to lsst-gapon-qserv-worker-1-bad lsst-gapon-qserv-worker-8 to lsst-gapon-qserv-worker-8-bad New instances were created and sized similarly to the bad ones. Attaching volumes NOTE : ideally it was much simple to transfer (detach from the older ones and attach to the new ones) volumes lsst-gapon-qserv-db-1 and lsst-gapon-qserv-db-8 (those which have properly configured Qserv data directory /qserv ). Unfortunately this was not possible due to a strange state in which both bad instances are within the OpenStack service. Hence, the idea is to borrow Qserv data snapshot of some other (healthy) workers. Attached existing (borrowed) volumes as /dev/vdb to the instances. Created two 256 GB volumes for new instances and attached them as /dev/vdc to the instances Here is a map of instances host borrowed volume: /qserv_old size own new volume: /qserv size lsst-gapon-qserv-worker-1 lsst-gapon-qserv-db-2 128 GB lsst-gapon-qserv-db-11 256 GB lsst-gapon-qserv-worker-8 lsst-gapon-qserv-db-3 128 GB lsst-gapon-qserv-db-18 256 GB Configuring the master node update IP addresses of the new instances at /etc/hosts Configuring the instances as explained earlier   update RPMs  install RPMs  install devtoolset-6  setup accounts  configure /etc/hosts  mount and initialize external volumes   configure /etc/fstab  configure NFS  setup and configure the latest version of Docker Borrowing data from /qserv_old to /qserv   copy files from /qserv_old to /qserv reconfigure Qserv to add these nodes back  pull Docker containers to the nodes start Qserv fix worker ID on the nodes rebuild the empty chunks list for database wise_00 restart Qserv test Qserv by launching a few queries

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                fritzm Fritz Mueller
                Watchers:
                Fritz Mueller, Igor Gaponenko
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel