Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17731

the dimensions of PSF model skyrocket

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      With lsst_distrib w_2019_05 and obs_lsst ade4ffd (master as of Feb 4),
      coadding imsim Run 1.2i data at tract=4849 failed by

      makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: Adjusted dimensions of reference PSF model from (23, 23) to (9675, 9675)
      

      followed by OOM kill.

      This can be reproduced by

      makeCoaddTempExp.py  /datasets/DC2/repo/ --rerun w_2019_04/sfm:private/username/yours -c doApplyUberCal=False --id tract=4849 patch=4,3 filter=g   --selectId visit=1707144 
      

      Or, starting with processCcd:

      processCcd.py /datasets/DC2/repo/ --rerun private/username/yours --id  visit=1707144 raftName=R24 detectorName=S02^S12
      makeSkyMap.py /datasets/DC2/repo --rerun private/username/yours
      makeCoaddTempExp.py  /datasets/DC2/repo/ --rerun private/username/yours -c doApplyUberCal=False --id tract=4849 patch=4,3 filter=g   --selectId visit=1707144
      

      (This would run ~8 min on verification cluster before killed)

        Attachments

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          FWIW, this issue wasn't seen in the same data processed by w_2018_49 last year; those old outputs are at /datasets/DC2/repo/rerun/w49multiband

          Show
          hchiang2 Hsin-Fang Chiang added a comment - FWIW, this issue wasn't seen in the same data processed by w_2018_49 last year; those old outputs are at  /datasets/DC2/repo/rerun/w49multiband
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Came across another example...

          To reproduce:

          makeCoaddTempExp.py  /datasets/DC2/repo/ --rerun w_2019_04/sfm:private/username/yours  -c doApplyUberCal=False --id tract=4850 patch=6,4 filter=i    --selectId visit=204661 
          

           

          makeCoaddTempExp INFO: Processing calexp 4 of 4 for this Warp: id=DataId(initialdata={'visit': 204661, 'filter': 'i', 'raftName': 'R12', 'detectorName': 'S12', 'detector': 50, 'tract': 4850}, tag=set())
          makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: compute Psf-matching kernel
          makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: Adjusted dimensions of reference PSF model from (23, 23) to (6165, 6165)
          makeCoaddTempExp.warpAndPsfMatch INFO: Cannot PSF-Match: std::bad_alloc
          

          This one failed more gently. The maximum resident set size was ~225 GB.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Came across another example... To reproduce: makeCoaddTempExp.py /datasets/DC2/repo/ --rerun w_2019_04/sfm:private/username/yours -c doApplyUberCal=False --id tract=4850 patch=6,4 filter=i --selectId visit=204661   makeCoaddTempExp INFO: Processing calexp 4 of 4 for this Warp: id=DataId(initialdata={'visit': 204661, 'filter': 'i', 'raftName': 'R12', 'detectorName': 'S12', 'detector': 50, 'tract': 4850}, tag=set()) makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: compute Psf-matching kernel makeCoaddTempExp.warpAndPsfMatch.psfMatch INFO: Adjusted dimensions of reference PSF model from (23, 23) to (6165, 6165) makeCoaddTempExp.warpAndPsfMatch INFO: Cannot PSF-Match: std::bad_alloc This one failed more gently. The maximum resident set size was ~225 GB.
          Hide
          swinbank John Swinbank added a comment -

          Yusra AlSayyad, Jim Bosch  — I assume the DRP team is taking this?

          Show
          swinbank John Swinbank added a comment - Yusra AlSayyad , Jim Bosch  — I assume the DRP team is taking this?

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, John Swinbank, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel