[DM-25927] ci_hsc_gen3 scons -c does not clean all data causing problems running scons again - Jira

XML

Word

Printable

Details

Type: Story
Status: Won't Fix
Resolution: Done
Fix Version/s: None
Component/s: ci_hsc_gen3
Labels:
None

Epic Link:
s21b-drp-emergent
Team:
Data Release Production
Urgent?:
No

Description

Note: This is not a problem with trying to use newer version to clean up repository run by older software version. This is using completely same version of code to run and then try to clean up and run again. After running scons -c after letting scons -j 8 run entire pipeline:

$ find DATA -not -type d | wc -l

$ find DATA -maxdepth 3 -type d

DATA

DATA/HSC

DATA/HSC/calib

DATA/HSC/calib/camera

DATA/HSC/calib/sky

DATA/HSC/calib/bias

DATA/HSC/calib/transmission_optics

DATA/HSC/calib/transmission_filter

DATA/HSC/calib/transmission_atmosphere

DATA/HSC/calib/flat

DATA/HSC/calib/bfKernel

DATA/HSC/calib/transmission_sensor

DATA/HSC/calib/defects

DATA/HSC/calib/dark

DATA/HSC/raw

DATA/HSC/raw/all

DATA/HSC/masks

DATA/HSC/masks/brightObjectMask

$ scons -j 8

scons: Reading SConscript files ...

EUPS integration: enabled

scons: done reading SConscript files.

scons: Building targets ...

rewrite_shebang(["bin/exportExternalData.py"], ["bin.src/exportExternalData.py"])

changing mode of bin/exportExternalData.py from 420 to 493

python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler create /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA

python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler register-instrument /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA lsst.obs.subaru.HyperSuprimeCam

python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler write-curated-calibrations /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA -i HSC

Error: An error occurred during command execution:

Traceback (most recent call last):

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/cli/utils.py", line 266, in cli_handle_exception

    return func(*args, **kwargs)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/script/writeCuratedCalibrations.py", line 55, in writeCuratedCalibrations

    instr.writeCuratedCalibrations(butler, run=output_run)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 271, in writeCuratedCalibrations

    self.writeCameraGeom(butler, run=run)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 325, in writeCameraGeom

    butler.put(camera, datasetType, unboundedDataId, run=run)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner

    return func(self, *args, **kwargs)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/_butler.py", line 650, in put

    self.datastore.put(obj, ref)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner

    return func(self, *args, **kwargs)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/fileLikeDatastore.py", line 1096, in put

    storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)

  File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/posixDatastore.py", line 168, in _write_in_memory_to_artifact

    raise FileExistsError(f"Cannot write file for ref {ref} as "

FileExistsError: Cannot write file for ref camera@{instrument: HSC, calibration_label: unbounded} (id=1) as output file /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA/HSC/calib/camera/camera_unbounded_HSC_HSC_calib.fits already exists

I think this has been happening for at least one previous weekly, but never stopped to verify until current week w_2020_28. The workaround is to rm -rf ${CI_HSC_GEN3_DIR}/DATA between scons executions.

Attachments

Activity

Ascending order - Click to sort in descending order

No builds found.

Michelle Gower created issue - 13/Jul/20 5:39 PM

Hide

Permalink

Tim Jenness added a comment - 13/Jul/20 7:40 PM

Can you do git clean -fdx to work around it. That's what I always do and it's more reliable than scons -c or manually trying to remember what to remove.

Show

Tim Jenness added a comment - 13/Jul/20 7:40 PM Can you do git clean -fdx to work around it. That's what I always do and it's more reliable than scons -c or manually trying to remember what to remove.

Hide

Permalink

Michelle Gower added a comment - 13/Jul/20 8:37 PM

Yes. git clean -fdx works much better than scons -c; rm -rf DATA

Show

Michelle Gower added a comment - 13/Jul/20 8:37 PM Yes. git clean -fdx works much better than scons -c; rm -rf DATA

Michelle Gower made changes - 13/Jul/20 9:06 PM

Field	Original Value	New Value
Description	Note: This is not a problem with trying to use newer version to clean up repository run by older software version. This is using completely same version of code to run and then try to clean up and run again. After running scons -c after letting scons -j 8 run entire pipeline: {noformat} $ find DATA -not -type d \| wc -l 1107 $ find DATA -maxdepth 3 -type d DATA DATA/HSC DATA/HSC/calib DATA/HSC/calib/camera DATA/HSC/calib/sky DATA/HSC/calib/bias DATA/HSC/calib/transmission_optics DATA/HSC/calib/transmission_filter DATA/HSC/calib/transmission_atmosphere DATA/HSC/calib/flat DATA/HSC/calib/bfKernel DATA/HSC/calib/transmission_sensor DATA/HSC/calib/defects DATA/HSC/calib/dark DATA/HSC/raw DATA/HSC/raw/all DATA/HSC/masks DATA/HSC/masks/brightObjectMask $ scons -j 8 scons: Reading SConscript files ... EUPS integration: enabled scons: done reading SConscript files. scons: Building targets ... rewrite_shebang(["bin/exportExternalData.py"], ["bin.src/exportExternalData.py"]) changing mode of bin/exportExternalData.py from 420 to 493 python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler create /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler register-instrument /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA lsst.obs.subaru.HyperSuprimeCam python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler write-curated-calibrations /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA -i HSC Error: An error occurred during command execution: Traceback (most recent call last): File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/cli/utils.py", line 266, in cli_handle_exception return func(args, kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/script/writeCuratedCalibrations.py", line 55, in writeCuratedCalibrations instr.writeCuratedCalibrations(butler, run=output_run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 271, in writeCuratedCalibrations self.writeCameraGeom(butler, run=run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 325, in writeCameraGeom butler.put(camera, datasetType, unboundedDataId, run=run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner return func(self, args, *kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/_butler.py", line 650, in put self.datastore.put(obj, ref) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner return func(self, args, **kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/fileLikeDatastore.py", line 1096, in put storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/posixDatastore.py", line 168, in _write_in_memory_to_artifact raise FileExistsError(f"Cannot write file for ref {ref} as " FileExistsError: Cannot write file for ref camera@{instrument: HSC, calibration_label: unbounded} (id=1) as output file /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA/HSC/calib/camera/camera_unbounded_HSC_HSC_calib.fits already exists {noformat} I think this has been happening for at least one previous weekly, but never stopped to verify until current week w_2020_28. The workaround is to rm -rf ${CI_HSC_GEN3_DIR}/DATA between scons executions.	Note: This is not a problem with trying to use newer version to clean up repository run by older software version. This is using completely same version of code to run and then try to clean up and run again. After running scons -c after letting scons -j 8 run entire pipeline: {noformat}$ find DATA -not -type d \| wc -l 1107 $ find DATA -maxdepth 3 -type d DATA DATA/HSC DATA/HSC/calib DATA/HSC/calib/camera DATA/HSC/calib/sky DATA/HSC/calib/bias DATA/HSC/calib/transmission_optics DATA/HSC/calib/transmission_filter DATA/HSC/calib/transmission_atmosphere DATA/HSC/calib/flat DATA/HSC/calib/bfKernel DATA/HSC/calib/transmission_sensor DATA/HSC/calib/defects DATA/HSC/calib/dark DATA/HSC/raw DATA/HSC/raw/all DATA/HSC/masks DATA/HSC/masks/brightObjectMask $ scons -j 8 scons: Reading SConscript files ... EUPS integration: enabled scons: done reading SConscript files. scons: Building targets ... rewrite_shebang(["bin/exportExternalData.py"], ["bin.src/exportExternalData.py"]) changing mode of bin/exportExternalData.py from 420 to 493 python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler create /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler register-instrument /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA lsst.obs.subaru.HyperSuprimeCam python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler write-curated-calibrations /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA -i HSC Error: An error occurred during command execution: Traceback (most recent call last): File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/cli/utils.py", line 266, in cli_handle_exception return func(args, kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/script/writeCuratedCalibrations.py", line 55, in writeCuratedCalibrations instr.writeCuratedCalibrations(butler, run=output_run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 271, in writeCuratedCalibrations self.writeCameraGeom(butler, run=run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 325, in writeCameraGeom butler.put(camera, datasetType, unboundedDataId, run=run) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner return func(self, args, *kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/_butler.py", line 650, in put self.datastore.put(obj, ref) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner return func(self, args, **kwargs) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/fileLikeDatastore.py", line 1096, in put storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref) File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/posixDatastore.py", line 168, in _write_in_memory_to_artifact raise FileExistsError(f"Cannot write file for ref {ref} as " FileExistsError: Cannot write file for ref camera@{instrument: HSC, calibration_label: unbounded} (id=1) as output file /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA/HSC/calib/camera/camera_unbounded_HSC_HSC_calib.fits already exists {noformat} I think this has been happening for at least one previous weekly, but never stopped to verify until current week w_2020_28. The workaround is to {{rm -rf $\{CI_HSC_GEN3_DIR}/DATA}} between scons executions.

John Swinbank made changes - 20/Jul/20 3:07 PM

Labels

gen3-middleware

John Swinbank made changes - 20/Jul/20 3:07 PM

Epic Link

~~DM-25271~~ [ 435631 ]

John Swinbank made changes - 20/Jul/20 3:07 PM

Team

Data Release Production [ 10301 ]

Yusra AlSayyad made changes - 02/Oct/20 7:18 PM

Epic Link

~~DM-25271~~ [ 435631 ]

~~DM-26789~~ [ 439739 ]

Yusra AlSayyad made changes - 14/Dec/20 8:01 PM

Epic Link

~~DM-26789~~ [ 439739 ]

~~DM-27974~~ [ 442759 ]

Hide

Permalink

Christopher Waters added a comment - 17/Mar/21 9:07 PM

Can you confirm if this is still a problem? As part of a DRP look at outstanding/unassigned tickets, I ran

git pull && scons -j 4 && scons -c

and this appears to have cleaned everything up, including the DATA directory.

Show

Christopher Waters added a comment - 17/Mar/21 9:07 PM Can you confirm if this is still a problem? As part of a DRP look at outstanding/unassigned tickets, I ran git pull && scons -j 4 && scons -c and this appears to have cleaned everything up, including the DATA directory.

Christopher Waters made changes - 17/Mar/21 9:08 PM

Assignee

Christopher Waters [ cwaters ]

Christopher Waters made changes - 17/Mar/21 9:09 PM

Epic Link

~~DM-27974~~ [ 442759 ]

~~DM-29152~~ [ 458509 ]

Hide

Permalink

Michelle Gower added a comment - 18/Mar/21 4:17 PM

I have the git clean command in all my scripts, so I wouldn't have noticed. I ran a test with w_2021_12 and confirm that this is no longer a problem.

Show

Michelle Gower added a comment - 18/Mar/21 4:17 PM I have the git clean command in all my scripts, so I wouldn't have noticed. I ran a test with w_2021_12 and confirm that this is no longer a problem.

Hide

Permalink

Christopher Waters added a comment - 18/Mar/21 5:39 PM

With confirmation it's no longer an issue, I'm closing the ticket.

Show

Christopher Waters added a comment - 18/Mar/21 5:39 PM With confirmation it's no longer an issue, I'm closing the ticket.

Christopher Waters made changes - 18/Mar/21 5:39 PM

Resolution		Done [ 10000 ]
Status	To Do [ 10001 ]	Won't Fix [ 10405 ]

People

Assignee:: Christopher Waters

Reporter:: Michelle Gower

Watchers:: Christopher Waters, Michelle Gower, Tim Jenness

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 13/Jul/20 5:39 PM

Updated:: 18/Mar/21 5:39 PM

Resolved:: 18/Mar/21 5:39 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Activity

People

Dates

Jenkins