Fritz Mueller reported yesterday that the qserv-dev eups distrib tag was not updating after being published. The s3 object was confirmed to be correct but was not syncing to the k8s service. It was assumed at the time that this was a random case of s3 eventual consistency taking an excessively long time and the k8s pod was always getting an old version of the object. However, > 12 hours seems excessive for this.
Upon further investigation this morning, it appears that aws s3 sync from awscli, which is used to to perform the sync, does not checksum the local file to determine if it is in-sync with s3. All it does it look at the file size by default, and can optionally compare timestamps (which isn't enabled) – there is no option to force checksums (ie., rsync -c). This is rather unfortunate as s3 does have an ETag (md5) for all objects. Eg.,
Demonstration that the s3 object and the stale eups.lsst.codes file are the same size: