Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ci_hsc_gen3
-
Labels:None
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
Note: This is not a problem with trying to use newer version to clean up repository run by older software version. This is using completely same version of code to run and then try to clean up and run again. After running scons -c after letting scons -j 8 run entire pipeline:
$ find DATA -not -type d | wc -l
|
1107
|
$ find DATA -maxdepth 3 -type d
|
DATA
|
DATA/HSC
|
DATA/HSC/calib
|
DATA/HSC/calib/camera
|
DATA/HSC/calib/sky
|
DATA/HSC/calib/bias
|
DATA/HSC/calib/transmission_optics
|
DATA/HSC/calib/transmission_filter
|
DATA/HSC/calib/transmission_atmosphere
|
DATA/HSC/calib/flat
|
DATA/HSC/calib/bfKernel
|
DATA/HSC/calib/transmission_sensor
|
DATA/HSC/calib/defects
|
DATA/HSC/calib/dark
|
DATA/HSC/raw
|
DATA/HSC/raw/all
|
DATA/HSC/masks
|
DATA/HSC/masks/brightObjectMask
|
|
|
$ scons -j 8
|
scons: Reading SConscript files ...
|
EUPS integration: enabled
|
scons: done reading SConscript files.
|
scons: Building targets ...
|
rewrite_shebang(["bin/exportExternalData.py"], ["bin.src/exportExternalData.py"])
|
changing mode of bin/exportExternalData.py from 420 to 493
|
python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler create /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA
|
python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler register-instrument /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA lsst.obs.subaru.HyperSuprimeCam
|
python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler write-curated-calibrations /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA -i HSC
|
Error: An error occurred during command execution:
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/cli/utils.py", line 266, in cli_handle_exception
|
return func(*args, **kwargs)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/script/writeCuratedCalibrations.py", line 55, in writeCuratedCalibrations
|
instr.writeCuratedCalibrations(butler, run=output_run)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 271, in writeCuratedCalibrations
|
self.writeCameraGeom(butler, run=run)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 325, in writeCameraGeom
|
butler.put(camera, datasetType, unboundedDataId, run=run)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner
|
return func(self, *args, **kwargs)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/_butler.py", line 650, in put
|
self.datastore.put(obj, ref)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner
|
return func(self, *args, **kwargs)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/fileLikeDatastore.py", line 1096, in put
|
storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
|
File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/posixDatastore.py", line 168, in _write_in_memory_to_artifact
|
raise FileExistsError(f"Cannot write file for ref {ref} as "
|
FileExistsError: Cannot write file for ref camera@{instrument: HSC, calibration_label: unbounded} (id=1) as output file /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA/HSC/calib/camera/camera_unbounded_HSC_HSC_calib.fits already exists
|
I think this has been happening for at least one previous weekly, but never stopped to verify until current week w_2020_28. The workaround is to rm -rf ${CI_HSC_GEN3_DIR}/DATA between scons executions.