Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11284

Butler writes many copies of identical entries _parents to repositoryCfg.yaml

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None

      Description

      I just noticed that one of the calibration scripts is loading the same Posix calibration registry around 40 times (which isn't cheap).

      The problem seems to be that rerun/rhl/tmp/repositoryCfg.yaml has many copies of the parent in its _parents list:

      !RepositoryCfg_v1
      _mapper: !!python/name:lsst.obs.pfs.pfsMapper.PfsMapper ''
      _mapperArgs: {calibRoot: /datasets/PFS/LAM/XCALIB}
      _parents: [../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM,
        ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM, ../../../../../../../datasets/PFS/LAM]
      _policy: null
      _root: null
      dirty: true
      

      While poking at this, I noted that the

      python/lsst/daf/persistence/butler.py", line 909, in _setAndVerifyParentsLists
          parents, repoData.cfg.parents, e))
      RuntimeError: Inputs of this Butler: ...
      

      error message doesn't give the paths to the butlers in question – if it did, it'd make the problem much easier to debug.

        Attachments

          Activity

          Hide
          npease Nate Pease added a comment -

          yikes

          Show
          npease Nate Pease added a comment - yikes
          Hide
          rhl Robert Lupton added a comment - - edited

          Here's a how-to-repeat on lsst-dev. If you're not familiar with this feature of eups, the -t ~rlupton/DM-11284.tags says "use the versions in this file as if they had a tag (such as w_1958_05) and I specified that tag instead" (and I wrote it by redirecting eups list -s). You can ignore the warnings – they are caused by my only processing one input image for speed.

          The use of --calib appears to be essential to reproducing this. I gave you write permission on $COMCAM_REPO_DIR/rerun/rhl/DM-11284, but you should be able to use your own output directory.

          $ setup pipe_drivers -t ~rlupton/DM-11284.tags
          $ setup obs_comCam -t ~rlupton/DM-11284.tags
          $ COMCAM_REPO_DIR=/home/mfl/comCam_repo_temp
          $ constructBias.py $COMCAM_REPO_DIR --calib $COMCAM_REPO_DIR/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --clobber-versions
          root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py'
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py'
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          root INFO: Running: /home/rlupton/LSST/pipe/drivers/bin/constructBias.py /home/mfl/comCam_repo_temp --calib /home/mfl/comCam_repo_temp/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --clobber-versions
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          bias INFO: Scatter processing
          bias INFO: Processing DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) on lsst-dev01.ncsa.illinois.edu:1723220
          bias.isr INFO: Performing ISR on sensor DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([]))
          assembleCcd WARN: No WCS found in input exposure
          CameraMapper INFO: Loading Posix exposure registry from .
          CameraMapper INFO: Loading Posix calib registry from .
          CameraMapper INFO: darkTime is NaN/Inf; using exposureTime
          bias.isr.assembleCcd WARN: No WCS found in input exposure
          bias INFO: Scale on lsst-dev01.ncsa.illinois.edu:1723220
          bias INFO: Scatter combination
          bias INFO: Combining {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723220
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2889: RuntimeWarning: Mean of empty slice.
            out=out, **kwargs)
          /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in true_divide
            ret = ret.dtype.type(ret / rcount)
          bias INFO: Writing {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723220
          $ cat $COMCAM_REPO_DIR/rerun/rhl/DM-11284/repositoryCfg.yaml
          !RepositoryCfg_v1
          _mapper: !!python/name:lsst.obs.comCam.comCamMapper.ComCamMapper ''
          _mapperArgs: {calibRoot: /home/mfl/comCam_repo_temp/NCALIB}
          _parents: [../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp,
            ../../../../../home/mfl/comCam_repo_temp]
          _policy: null
          _root: null
          dirty: true
          $ constructBias.py $COMCAM_REPO_DIR --calib $COMCAM_REPO_DIR/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --clobber-versions
          root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py'
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py'
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          root INFO: Running: /home/rlupton/LSST/pipe/drivers/bin/constructBias.py /home/mfl/comCam_repo_temp --calib /home/mfl/comCam_repo_temp/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --clobber-versions
          CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284
          bias INFO: Scatter processing
          bias INFO: Processing DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) on lsst-dev01.ncsa.illinois.edu:1723364
          bias.isr INFO: Performing ISR on sensor DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([]))
          assembleCcd WARN: No WCS found in input exposure
          CameraMapper INFO: Loading Posix exposure registry from .
          CameraMapper INFO: Loading Posix calib registry from .
          CameraMapper INFO: darkTime is NaN/Inf; using exposureTime
          bias.isr.assembleCcd WARN: No WCS found in input exposure
          bias INFO: Scale on lsst-dev01.ncsa.illinois.edu:1723364
          bias INFO: Scatter combination
          bias INFO: Combining {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723364
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header
          /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2889: RuntimeWarning: Mean of empty slice.
            out=out, **kwargs)
          /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in true_divide
            ret = ret.dtype.type(ret / rcount)
          bias INFO: Writing {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723364
          $ cat $COMCAM_REPO_DIR/rerun/rhl/DM-11284/repositoryCfg.yaml
          !RepositoryCfg_v1
          _mapper: !!python/name:lsst.obs.comCam.comCamMapper.ComCamMapper ''
          _mapperArgs: {calibRoot: /home/mfl/comCam_repo_temp/NCALIB}
          _parents: [../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp,
            ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp,
            ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp]
          _policy: null
          _root: null
          dirty: true
          $ 
          

          Show
          rhl Robert Lupton added a comment - - edited Here's a how-to-repeat on lsst-dev. If you're not familiar with this feature of eups , the -t ~rlupton/ DM-11284 .tags says "use the versions in this file as if they had a tag (such as w_1958_05 ) and I specified that tag instead" (and I wrote it by redirecting eups list -s ). You can ignore the warnings – they are caused by my only processing one input image for speed. The use of --calib appears to be essential to reproducing this. I gave you write permission on $COMCAM_REPO_DIR/rerun/rhl/ DM-11284 , but you should be able to use your own output directory. $ setup pipe_drivers -t ~rlupton/DM-11284.tags $ setup obs_comCam -t ~rlupton/DM-11284.tags $ COMCAM_REPO_DIR=/home/mfl/comCam_repo_temp $ constructBias.py $COMCAM_REPO_DIR --calib $COMCAM_REPO_DIR/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --clobber-versions root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py' CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py' CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 root INFO: Running: /home/rlupton/LSST/pipe/drivers/bin/constructBias.py /home/mfl/comCam_repo_temp --calib /home/mfl/comCam_repo_temp/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --clobber-versions CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 bias INFO: Scatter processing bias INFO: Processing DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) on lsst-dev01.ncsa.illinois.edu:1723220 bias.isr INFO: Performing ISR on sensor DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) assembleCcd WARN: No WCS found in input exposure CameraMapper INFO: Loading Posix exposure registry from . CameraMapper INFO: Loading Posix calib registry from . CameraMapper INFO: darkTime is NaN/Inf; using exposureTime bias.isr.assembleCcd WARN: No WCS found in input exposure bias INFO: Scale on lsst-dev01.ncsa.illinois.edu:1723220 bias INFO: Scatter combination bias INFO: Combining {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723220 afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2889: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in true_divide ret = ret.dtype.type(ret / rcount) bias INFO: Writing {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723220 $ cat $COMCAM_REPO_DIR/rerun/rhl/DM-11284/repositoryCfg.yaml !RepositoryCfg_v1 _mapper: !!python/name:lsst.obs.comCam.comCamMapper.ComCamMapper '' _mapperArgs: {calibRoot: /home/mfl/comCam_repo_temp/NCALIB} _parents: [../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp] _policy: null _root: null dirty: true $ constructBias.py $COMCAM_REPO_DIR --calib $COMCAM_REPO_DIR/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --clobber-versions root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py' CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 root INFO: Loading config overrride file u'/home/rlupton/LSST/obs/comCam/config/bias.py' CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 root INFO: Running: /home/rlupton/LSST/pipe/drivers/bin/constructBias.py /home/mfl/comCam_repo_temp --calib /home/mfl/comCam_repo_temp/NCALIB --rerun rhl/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --clobber-versions CameraMapper INFO: Loading Posix calib registry from /home/mfl/comCam_repo_temp/rerun/rhl/DM-11284 bias INFO: Scatter processing bias INFO: Processing DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) on lsst-dev01.ncsa.illinois.edu:1723364 bias.isr INFO: Performing ISR on sensor DataId(initialdata={'raft': 'R00', 'ccd': 'S11', u'run': '4417', 'visit': 270024215}, tag=set([])) assembleCcd WARN: No WCS found in input exposure CameraMapper INFO: Loading Posix exposure registry from . CameraMapper INFO: Loading Posix calib registry from . CameraMapper INFO: darkTime is NaN/Inf; using exposureTime bias.isr.assembleCcd WARN: No WCS found in input exposure bias INFO: Scale on lsst-dev01.ncsa.illinois.edu:1723364 bias INFO: Scatter combination bias INFO: Combining {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723364 afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header afw.image.ExposureInfo INFO: Empty WCS extension, using FITS header /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2889: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /ssd/lsstsw/stack/Linux64/miniconda2/4.2.12.lsst2/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in true_divide ret = ret.dtype.type(ret / rcount) bias INFO: Writing {'filter': 'NONE', 'ccd': 'S11', 'calibDate': '2017-05-24'} on lsst-dev01.ncsa.illinois.edu:1723364 $ cat $COMCAM_REPO_DIR/rerun/rhl/DM-11284/repositoryCfg.yaml !RepositoryCfg_v1 _mapper: !!python/name:lsst.obs.comCam.comCamMapper.ComCamMapper '' _mapperArgs: {calibRoot: /home/mfl/comCam_repo_temp/NCALIB} _parents: [../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp, ../../../../../home/mfl/comCam_repo_temp] _policy: null _root: null dirty: true $
          Hide
          price Paul Price added a comment - - edited

          I hit something that seems to be related while running singleFrameDriver.py in w_2017_28:

          Traceback (most recent call last):
            File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 113, in wrapper
              return func(*args, **kwargs)
            File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 1067, in run
              while not menu[command]():
            File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 237, in wrapper
              return func(*args, **kwargs)
            File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 1093, in reduce
              job = self.comm.recv(tag=tags.work, source=self.root)
            File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 268, in recv
              return super(Comm, self).recv(buf=obj, source=sts.source, tag=sts.tag, status=status)
            File "MPI/Comm.pyx", line 1192, in mpi4py.MPI.Comm.recv (src/mpi4py.MPI.c:106889)
            File "MPI/msgpickle.pxi", line 292, in mpi4py.MPI.PyMPI_recv (src/mpi4py.MPI.c:43053)
            File "MPI/msgpickle.pxi", line 143, in mpi4py.MPI.Pickle.load (src/mpi4py.MPI.c:41248)
            File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 1570, in _unreduce
              butler = Butler(**initArgs)
            File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 543, in __init__
              repoData.repo = Repository(repoData)
            File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/repository.py", line 136, in __init__
              self._storage.putRepositoryCfg(repoData.cfg, repoData.cfgRoot)
            File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py", line 166, in putRepositoryCfg
              datasetType=None))
            File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 84, in write
              raise RuntimeError("Can not extend existing repository cfg because: {}".format(e))
          RuntimeError: (RuntimeError("Can not extend existing repository cfg because: The beginning of the passed-in parents list: ['../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..'] does not match the existing parents list in this RepositoryCfg: ['../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..']",), <function _unreduce at 0x7f107f728cf8>, ({'mapper': None, 'inputs': {'root': '/scratch/pprice/abell2319/DATA', 'mapperArgs': {'calibRoot': '/datasets/hsc/calib/20170105'}}, 'root': None, 'outputs': {'root': '/scratch/pprice/abell2319/DATA/rerun/20170719', 'mode': 'rw', 'mapperArgs': {'calibRoot': '/datasets/hsc/calib/20170105'}}}, {}))
          

          I ran:

          pprice@lsst-dev01:/scratch/pprice/abell2319 $ singleFrameDriver.py DATA --rerun 20170719 --id visit=7802..7806:2 --calib /datasets/hsc/calib/20170105/ --cores 112 --batch-type slurm --mpiexec '-bind-to socket' --time 600 --job abell2319
          

          Show
          price Paul Price added a comment - - edited I hit something that seems to be related while running singleFrameDriver.py in w_2017_28 : Traceback (most recent call last): File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 113, in wrapper return func(*args, **kwargs) File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 1067, in run while not menu[command](): File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 237, in wrapper return func(*args, **kwargs) File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 1093, in reduce job = self.comm.recv(tag=tags.work, source=self.root) File "/software/lsstsw/stack/Linux64/ctrl_pool/13.0-5-g9cf35e0+7/python/lsst/ctrl/pool/pool.py", line 268, in recv return super(Comm, self).recv(buf=obj, source=sts.source, tag=sts.tag, status=status) File "MPI/Comm.pyx", line 1192, in mpi4py.MPI.Comm.recv (src/mpi4py.MPI.c:106889) File "MPI/msgpickle.pxi", line 292, in mpi4py.MPI.PyMPI_recv (src/mpi4py.MPI.c:43053) File "MPI/msgpickle.pxi", line 143, in mpi4py.MPI.Pickle.load (src/mpi4py.MPI.c:41248) File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 1570, in _unreduce butler = Butler(**initArgs) File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py", line 543, in __init__ repoData.repo = Repository(repoData) File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/repository.py", line 136, in __init__ self._storage.putRepositoryCfg(repoData.cfg, repoData.cfgRoot) File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py", line 166, in putRepositoryCfg datasetType=None)) File "/software/lsstsw/stack/Linux64/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 84, in write raise RuntimeError("Can not extend existing repository cfg because: {}".format(e)) RuntimeError: (RuntimeError("Can not extend existing repository cfg because: The beginning of the passed-in parents list: ['../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..'] does not match the existing parents list in this RepositoryCfg: ['../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..', '../..']",), <function _unreduce at 0x7f107f728cf8>, ({'mapper': None, 'inputs': {'root': '/scratch/pprice/abell2319/DATA', 'mapperArgs': {'calibRoot': '/datasets/hsc/calib/20170105'}}, 'root': None, 'outputs': {'root': '/scratch/pprice/abell2319/DATA/rerun/20170719', 'mode': 'rw', 'mapperArgs': {'calibRoot': '/datasets/hsc/calib/20170105'}}}, {})) I ran: pprice@lsst-dev01:/scratch/pprice/abell2319 $ singleFrameDriver.py DATA --rerun 20170719 --id visit=7802..7806:2 --calib /datasets/hsc/calib/20170105/ --cores 112 --batch-type slurm --mpiexec '-bind-to socket' --time 600 --job abell2319
          Hide
          rhl Robert Lupton added a comment - - edited

          Here's a how-to-repeat using /datasets/comCam/repo directly. If I got the file permissions wrong, complain on slack!

          $ setup lsst_distrib -t w_2017_28
          $ setup -j -r ~rlupton/LSST/obs/base          # this is master, and will be w_2017_29
          $ setup -j -r ~rlupton/LSST/obs/comCam   # this is tickets/DM-11163
           
          $ export COMCAM_REPO_DIR=/datasets/comCam/repo
          $ mkdir -p ~/tmp
           
          $ rm -rf $COMCAM_REPO_DIR/rerun/npease/DM-11284
          $ rm -rf ~/tmp/NCALIB
           
          $ constructBias.py $COMCAM_REPO_DIR --calib ~/tmp/NCALIB --rerun npease/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --no-versions
          $ cat $COMCAM_REPO_DIR/rerun/npease/DM-11284/repositoryCfg.yaml
          

          Every time you rerun constructBias.py you'll get three more entries in _parent (there are already three after this command).

          Show
          rhl Robert Lupton added a comment - - edited Here's a how-to-repeat using /datasets/comCam/repo directly. If I got the file permissions wrong, complain on slack! $ setup lsst_distrib -t w_2017_28 $ setup -j -r ~rlupton/LSST/obs/base # this is master, and will be w_2017_29 $ setup -j -r ~rlupton/LSST/obs/comCam # this is tickets/DM-11163   $ export COMCAM_REPO_DIR=/datasets/comCam/repo $ mkdir -p ~/tmp   $ rm -rf $COMCAM_REPO_DIR/rerun/npease/DM-11284 $ rm -rf ~/tmp/NCALIB   $ constructBias.py $COMCAM_REPO_DIR --calib ~/tmp/NCALIB --rerun npease/DM-11284 --id raft=R00 ccd=S11 visit=270024215 --batch-type none --no-versions $ cat $COMCAM_REPO_DIR/rerun/npease/DM-11284/repositoryCfg.yaml Every time you rerun constructBias.py you'll get three more entries in _parent (there are already three after this command).
          Hide
          npease Nate Pease added a comment -

          note there's 2 branches, 2 PR's (one of them is not showing up in Jira yet)

          Show
          npease Nate Pease added a comment - note there's 2 branches, 2 PR's (one of them is not showing up in Jira yet)
          Hide
          npease Nate Pease added a comment -

          Andy Salnikov is reviewing, but here I'll ping @rhl who may want to take a look as well.

          Show
          npease Nate Pease added a comment - Andy Salnikov is reviewing, but here I'll ping @rhl who may want to take a look as well.
          Hide
          rhl Robert Lupton added a comment -

          I checked that the code worked for me (with a new --calib), so I'll leave the picky comments to Andy!

          Show
          rhl Robert Lupton added a comment - I checked that the code worked for me (with a new --calib ), so I'll leave the picky comments to Andy!
          Hide
          salnikov Andy Salnikov added a comment -

          Looks OK, few minor comments on PR.

          Show
          salnikov Andy Salnikov added a comment - Looks OK, few minor comments on PR.

            People

            • Assignee:
              npease Nate Pease
              Reporter:
              rhl Robert Lupton
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Kian-Tat Lim, Nate Pease, Paul Price, Robert Lupton
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel