Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25927

ci_hsc_gen3 scons -c does not clean all data causing problems running scons again

    XMLWordPrintable

Details

    • Story
    • Status: Won't Fix
    • Resolution: Done
    • None
    • ci_hsc_gen3
    • None

    Description

      Note: This is not a problem with trying to use newer version to clean up repository run by older software version. This is using completely same version of code to run and then try to clean up and run again. After running scons -c after letting scons -j 8 run entire pipeline:

      $ find DATA -not -type d | wc -l
      1107
      $ find DATA -maxdepth 3 -type d
      DATA
      DATA/HSC
      DATA/HSC/calib
      DATA/HSC/calib/camera
      DATA/HSC/calib/sky
      DATA/HSC/calib/bias
      DATA/HSC/calib/transmission_optics
      DATA/HSC/calib/transmission_filter
      DATA/HSC/calib/transmission_atmosphere
      DATA/HSC/calib/flat
      DATA/HSC/calib/bfKernel
      DATA/HSC/calib/transmission_sensor
      DATA/HSC/calib/defects
      DATA/HSC/calib/dark
      DATA/HSC/raw
      DATA/HSC/raw/all
      DATA/HSC/masks
      DATA/HSC/masks/brightObjectMask
       
      $ scons -j 8
      scons: Reading SConscript files ...
      EUPS integration: enabled
      scons: done reading SConscript files.
      scons: Building targets ...
      rewrite_shebang(["bin/exportExternalData.py"], ["bin.src/exportExternalData.py"])
      changing mode of bin/exportExternalData.py from 420 to 493
      python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler create /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA
      python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler register-instrument /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA lsst.obs.subaru.HyperSuprimeCam
      python /software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/bin/butler write-curated-calibrations /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA -i HSC
      Error: An error occurred during command execution:
      Traceback (most recent call last):
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/cli/utils.py", line 266, in cli_handle_exception
          return func(*args, **kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/script/writeCuratedCalibrations.py", line 55, in writeCuratedCalibrations
          instr.writeCuratedCalibrations(butler, run=output_run)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 271, in writeCuratedCalibrations
          self.writeCameraGeom(butler, run=run)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/obs_base/20.0.0-17-g41c5faf/python/lsst/obs/base/_instrument.py", line 325, in writeCameraGeom
          butler.put(camera, datasetType, unboundedDataId, run=run)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner
          return func(self, *args, **kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/_butler.py", line 650, in put
          self.datastore.put(obj, ref)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/core/utils.py", line 249, in inner
          return func(self, *args, **kwargs)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/fileLikeDatastore.py", line 1096, in put
          storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
        File "/software/lsstsw/stack_20200515/stack/miniconda3-4.7.12-46b24e8/Linux64/daf_butler/19.0.0-114-gca9ed1bd/python/lsst/daf/butler/datastores/posixDatastore.py", line 168, in _write_in_memory_to_artifact
          raise FileExistsError(f"Cannot write file for ref {ref} as "
      FileExistsError: Cannot write file for ref camera@{instrument: HSC, calibration_label: unbounded} (id=1) as output file /scratch/mgower/weekly_ci_hsc_gen3/postgres/git/ci_hsc_gen3/DATA/HSC/calib/camera/camera_unbounded_HSC_HSC_calib.fits already exists
      

      I think this has been happening for at least one previous weekly, but never stopped to verify until current week w_2020_28. The workaround is to rm -rf ${CI_HSC_GEN3_DIR}/DATA between scons executions.

      Attachments

        Activity

          tjenness Tim Jenness added a comment -

          Can you do git clean -fdx to work around it. That's what I always do and it's more reliable than scons -c or manually trying to remember what to remove.

          tjenness Tim Jenness added a comment - Can you do git clean -fdx to work around it. That's what I always do and it's more reliable than scons -c or manually trying to remember what to remove.

          Yes. git clean -fdx works much better than scons -c; rm -rf DATA

          mgower Michelle Gower added a comment - Yes. git clean -fdx works much better than scons -c; rm -rf DATA

          Can you confirm if this is still a problem?  As part of a DRP look at outstanding/unassigned tickets, I ran

          git pull && scons -j 4 && scons -c

          and this appears to have cleaned everything up, including the DATA directory.

          czw Christopher Waters added a comment - Can you confirm if this is still a problem?  As part of a DRP look at outstanding/unassigned tickets, I ran git pull && scons -j 4 && scons -c and this appears to have cleaned everything up, including the DATA directory.

          I have the git clean command in all my scripts, so I wouldn't have noticed. I ran a test with w_2021_12 and confirm that this is no longer a problem.

          mgower Michelle Gower added a comment - I have the git clean command in all my scripts, so I wouldn't have noticed. I ran a test with w_2021_12 and confirm that this is no longer a problem.

          With confirmation it's no longer an issue, I'm closing the ticket.

          czw Christopher Waters added a comment - With confirmation it's no longer an issue, I'm closing the ticket.

          People

            czw Christopher Waters
            mgower Michelle Gower
            Christopher Waters, Michelle Gower, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.