Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14870

eups.lsst.codes sync from s3 does not update objects of identical size

    Details

    • Templates:
    • Story Points:
      1
    • Epic Link:
    • Team:
      SQuaRE

      Description

      Fritz Mueller reported yesterday that the qserv-dev eups distrib tag was not updating after being published. The s3 object was confirmed to be correct but was not syncing to the k8s service. It was assumed at the time that this was a random case of s3 eventual consistency taking an excessively long time and the k8s pod was always getting an old version of the object. However, > 12 hours seems excessive for this.

      Upon further investigation this morning, it appears that aws s3 sync from awscli, which is used to to perform the sync, does not checksum the local file to determine if it is in-sync with s3. All it does it look at the file size by default, and can optionally compare timestamps (which isn't enabled) – there is no option to force checksums (ie., rsync -c). This is rather unfortunate as s3 does have an ETag (md5) for all objects. Eg.,

      $ aws s3api head-object  --bucket eups.lsst.codes --key stack/src/tags/qserv-dev.list 
      {
          "AcceptRanges": "bytes",
          "LastModified": "Fri, 22 Jun 2018 01:14:45 GMT",
          "ContentLength": 2495,
          "ETag": "\"04d0d2da6b4b1107bb03453177813201\"",
          "VersionId": "null",
          "ContentType": "binary/octet-stream",
          "Metadata": {}
      }
       $ aws s3 cp s3://eups.lsst.codes/stack/src/tags/qserv-dev.list .
      download: s3://eups.lsst.codes/stack/src/tags/qserv-dev.list to ./qserv-dev.list
       $ md5sum qserv-dev.list 
      04d0d2da6b4b1107bb03453177813201  qserv-dev.list
      

      Demonstration that the s3 object and the stale eups.lsst.codes file are the same size:

      [root@pkgroot-rc-jh4lf /]# grep BUILD= /var/www/html/stack/src/tags/qserv-dev.list
      #BUILD=b3668
      [root@pkgroot-rc-jh4lf /]# ls -la /var/www/html/stack/src/tags/qserv-dev.list
      -rw-r--r-- 1 root root 2495 Jun 21 12:42 /var/www/html/stack/src/tags/qserv-dev.list
       
      $ grep BUILD= qserv-dev.list 
      #BUILD=b3670
      $ ls -la qserv-dev.list 
      -rw-rw-r--. 1 jhoblitt jhoblitt 2495 Jun 21 18:14 qserv-dev.list
      

        Attachments

          Container Issues

            Activity

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Fritz Mueller
                Watchers:
                Fritz Mueller, Frossie Economou, Gabriele Comoretto, Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel