Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8119

Incorrect time zone settings within Qserv Docker containers

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:

      Description

      I have noticed that the timezone is not set correctly within the latest version of the Qserv containers. This is an example:

      % hostname
      lsst-qserv-db01
       
      % date
      Fri Oct 28 11:33:00 CDT 2016
       
      % docker exec -it qserv date
      Fri Oct 28 16:33:10 UTC 2016
      

      This may have various (some of them would be unpleasant) side effects, such as (to count just a few):

      • incorrect timing in the log files (which will make it hard to investigate problems)
      • shifted monitoring data (hard to interpret/analyze the monitoring data)
      • anything else which is time-dependent within the application (such as scheduling, etc.)

      It looks like the timezone configuration within the container is set to:

      % docker exec -it qserv cat /etc/timezone
      Etc/UTC
      

      Should this be solved by running the container with this extra option which would override the default setting of Etc/UTC set at a container build time?

      % ls -l /etc/localtime
      lrwxrwxrwx 1 root root 35 Sep 23 09:43 /etc/localtime -> /usr/share/zoneinfo/America/Chicago
       
      % docker run ... -v /etc/localtime:/etc/localtime:ro
      

      This would add the read-only map from the container's configuration file to the hosts's one.

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment -

            Investigating a proposed option of adding a map for /etc/localtime

            It's been discovered upon further investigation of the site-specific deployment procedure for the container that the container is run by the following script (which is a part of package qserv):

            % cat qserv/admin/tools/docker/deployment/ncsa/run.sh
            ..
            echo
            echo "Launch Qserv containers on master"
            echo "================================="
            echo
            shmux -Bm -c "
            docker run --detach=true \
                -e "QSERV_MASTER=$MASTER" \
                $DATA_VOLUME_OPT \
                $LOG_VOLUME_OPT \
                $ULIMIT_OPT \
                --name $CONTAINER_NAME --net=host \
                $MASTER_IMAGE" "$SSH_MASTER"
             
            echo
            echo "Launch Qserv containers on worker"
            echo "================================="
            echo
            shmux -Bm -S all -c "docker run --detach=true \
                -e "QSERV_MASTER=$MASTER" \
                $DATA_VOLUME_OPT \
                $LOG_VOLUME_OPT \
                $ULIMIT_OPT \
                --name $CONTAINER_NAME --net=host \
                $WORKER_IMAGE" $SSH_WORKERS
            

            Hence an obvious solution to the problem was to add the following line to both master and worker sections of the script:

            shmux -Bm -S all -c "docker run --detach=true \
                ...
                -v /etc/localtime:/etc/localtime:ro \
                ...
            

            The new configuration has been tested by restarting the (Qserv) cluster:

            ./stop.sh
            ./run.sh
            

            And inspecting new settings on the master and one of the worker nodes by:

            % hostname
            lsst-qserv-master01
             
            % date
            Fri Oct 28 16:29:09 CDT 2016
             
            % docker exec -it qserv date
            Fri Oct 28 16:29:11 CDT 2016
             
            % hostname
            lsst-qserv-db01
             
            % date
            Fri Oct 28 16:29:44 CDT 2016
             
            % docker exec -it qserv date
            Fri Oct 28 16:29:45 CDT 2016
            

            Show
            gapon Igor Gaponenko added a comment - Investigating a proposed option of adding a map for /etc/localtime It's been discovered upon further investigation of the site-specific deployment procedure for the container that the container is run by the following script (which is a part of package qserv ): % cat qserv /admin/tools/docker/deployment/ncsa/run .sh .. echo echo "Launch Qserv containers on master" echo "=================================" echo shmux -Bm -c " docker run --detach= true \ -e "QSERV_MASTER=$MASTER" \ $DATA_VOLUME_OPT \ $LOG_VOLUME_OPT \ $ULIMIT_OPT \ --name $CONTAINER_NAME --net=host \ $MASTER_IMAGE " " $SSH_MASTER"   echo echo "Launch Qserv containers on worker" echo "=================================" echo shmux -Bm -S all -c "docker run --detach= true \ -e "QSERV_MASTER=$MASTER" \ $DATA_VOLUME_OPT \ $LOG_VOLUME_OPT \ $ULIMIT_OPT \ --name $CONTAINER_NAME --net=host \ $WORKER_IMAGE" $SSH_WORKERS Hence an obvious solution to the problem was to add the following line to both master and worker sections of the script: shmux -Bm -S all -c "docker run --detach= true \ ... - v /etc/localtime : /etc/localtime :ro \ ... The new configuration has been tested by restarting the (Qserv) cluster: . /stop .sh . /run .sh And inspecting new settings on the master and one of the worker nodes by: % hostname lsst-qserv-master01   % date Fri Oct 28 16:29:09 CDT 2016   % docker exec -it qserv date Fri Oct 28 16:29:11 CDT 2016   % hostname lsst-qserv-db01   % date Fri Oct 28 16:29:44 CDT 2016   % docker exec -it qserv date Fri Oct 28 16:29:45 CDT 2016
            Hide
            gapon Igor Gaponenko added a comment -

            Fabrice,

            I've made a small change to the container startup script for the NCSA deployment so that containers would inherit the timezone settings of the host environment rather than assuming the default settings of Etc/UTC. I believe this is the right thing to do, at least for the PDAC effort.

            I would appreciate if you had some time to review my changes and let me know if you'd agree with them.

            Thanks,
            Igor

            Show
            gapon Igor Gaponenko added a comment - Fabrice, I've made a small change to the container startup script for the NCSA deployment so that containers would inherit the timezone settings of the host environment rather than assuming the default settings of Etc/UTC. I believe this is the right thing to do, at least for the PDAC effort. I would appreciate if you had some time to review my changes and let me know if you'd agree with them. Thanks, Igor
            Hide
            ktl Kian-Tat Lim added a comment -

            Timezones in logs are meant to be handled according to (IETF) RFC 3339 as discussed in DM-1203, unless UTC-only systems can be configured.

            Show
            ktl Kian-Tat Lim added a comment - Timezones in logs are meant to be handled according to (IETF) RFC 3339 as discussed in DM-1203 , unless UTC-only systems can be configured.

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                Fabrice Jammes
                Watchers:
                Fabrice Jammes, Fritz Mueller, Igor Gaponenko, John Gates, Kian-Tat Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel