Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20536

Can't count on `obj_idxs` being in bounds

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_association
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Sprint:
      AP F19-2
    • Team:
      Alert Production

      Description

      While trying to run ap_pipe with preConvolution=True, some dataIds failed on the ap_association stage. eg:

      FATAL 2019-07-11T13:02:53.312-0500 apPipe ({'visit': 1238, 'ccd': 98, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0})(cmdLineTask.py:397)- Failed on dataId={'visit': 1238, 'ccd': 98, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}: IndexError: index 38 is out of bounds for axis 0 with size 38
      FATAL 2019-07-11T13:04:15.720-0500 apPipe ({'visit': 1246, 'ccd': 103, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0})(cmdLineTask.py:397)- Failed on dataId={'visit': 1246, 'ccd': 103, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}: RuntimeError: No patches found!
      FATAL 2019-07-11T13:11:07.675-0500 apPipe ({'visit': 1238, 'ccd': 99, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0})(cmdLineTask.py:397)- Failed on dataId={'visit': 1238, 'ccd': 99, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}: IndexError: index 34 is out of bounds for axis 0 with size 34
      FATAL 2019-07-11T13:18:27.189-0500 apPipe ({'visit': 1238, 'ccd': 100, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0})(cmdLineTask.py:397)- Failed on dataId={'visit': 1238, 'ccd': 100, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}: IndexError: index 224 is out of bounds for axis 0 with size 224
      

      which is from https://github.com/lsst/ap_association/blob/master/python/lsst/ap/association/association.py#L612

      This looks like some failure mode where the distance is returned as inf for some diaSources, (despite your not having set an upper bound) which produces the expected behavior of returning a flag value = n points (in this case 4):

      (Pdb) len(dia_objects)
      4
      (Pdb) dists
      array([           inf, 3.39470366e-04, 2.87712120e-04, 3.24986976e-04,
             2.09797388e-04, 5.00443603e-04, 4.77567395e-04, 4.99864336e-04,
             1.33310026e-04, 3.58528102e-04, 1.12004350e-04, 3.54529474e-04,
                        inf, 1.81771009e-04, 4.18082426e-04, 1.33462009e-04,
             1.48876950e-04, 2.58281375e-04, 6.19140327e-04, 4.38035230e-04,
             9.45884785e-05, 2.12677353e-04, 4.38683345e-04, 6.46062325e-04,
             1.88193056e-04, 4.13965388e-04, 6.06120497e-04, 4.06044351e-04,
             5.83160070e-04, 3.95516906e-04, 2.74059920e-04, 2.21553182e-04,
             2.26415781e-04, 5.64072359e-04, 5.79131387e-04, 6.63503591e-04,
             8.34488898e-04, 4.25167947e-04, 8.32268198e-04, 4.34112545e-04,
             2.04662618e-04, 3.73198069e-04, 6.91330856e-04, 7.06692744e-04,
             8.75070394e-05, 9.04586832e-04, 1.64555013e-04, 4.46136353e-04,
             2.84480881e-04, 1.05083764e-03, 1.17729476e-03, 7.32235889e-04,
             4.84333806e-04, 6.72715508e-04, 3.60535417e-04, 5.07654976e-04,
             1.03698729e-03, 1.22701689e-03, 1.31727626e-03, 1.10645719e-03,
             9.48651990e-04, 5.26152766e-04, 5.42898829e-04, 5.53111663e-04,
             8.86445016e-04, 1.33463361e-03, 1.05134358e-03, 7.10162430e-04,
             1.31023614e-03,            inf, 1.20190626e-03, 8.42393869e-04,
             8.90818870e-04, 1.61531283e-03, 1.30757514e-03, 9.61624113e-04,
             1.10305108e-03, 1.41183753e-03, 1.67478608e-03, 1.12513779e-03,
             1.07850540e-03, 1.18741103e-03, 1.15155631e-03, 1.93253143e-03,
             1.33435522e-03, 1.88592052e-03, 1.34288681e-03,            inf,
             1.59098384e-03, 1.44037035e-03, 1.45541046e-03, 1.97276848e-03,
             1.83746994e-03, 1.98287152e-03, 2.20175350e-03, 1.89541959e-03,
             1.81040203e-03, 1.98075464e-03, 2.31995026e-03, 2.28856630e-03,
             2.30179302e-03, 2.29817933e-03, 2.37023074e-03, 2.13633563e-03,
             2.09017129e-03,            inf, 2.52452285e-03, 2.63064197e-03,
             2.50920251e-03, 2.16545745e-03, 2.61624181e-03, 2.39901930e-03,
             2.25197331e-03, 2.63819654e-03, 2.48899074e-03, 5.01476338e-04])
      (Pdb) obj_idxs
      array([4, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 4, 2, 2, 0, 0, 2, 0, 0, 2, 2,
             0, 0, 2, 0, 0, 0, 0, 3, 2, 3, 3, 0, 0, 0, 0, 3, 0, 3, 3, 3, 0, 0,
             3, 0, 3, 3, 3, 0, 0, 3, 3, 3, 3, 3, 3, 0, 0, 3, 3, 3, 3, 3, 3, 0,
             3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4,
             3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3,
             3, 3, 3, 3, 3, 0])
      

      I'm not sure WHY it thinks that inf is greater than inf, but regardless, I'd expect this to just return no matches.

      Can reproduce from /datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe/w_2019_26_preConv with {{
      ap_pipe.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB/ --template /datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_templates_noSkyCorr --rerun RC/w_2019_26/DM-19560-sfm:private/yusra/RC2/ap_pipe/w_2019_26_preConv --configfile ./configfile.py --no-versions  --longlog --id visit=19712 ccd=49 --reuse-outputs-from=all --clobber-config}}

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  cmorrison Chris Morrison
                  Reporter:
                  yusra Yusra AlSayyad
                  Reviewers:
                  Yusra AlSayyad
                  Watchers:
                  Chris Morrison, Yusra AlSayyad
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel