Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4454

Fix multiple patch catalog sorting for forcedPhotCcd.py

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_base
    • Labels:
      None

      Description

      forcedPhotCcd.py is currently broken due to the requirement of the lsst.afw.table.getChildren() function that the SourceCatalog is sorted by the parent key (i.e. lsst.afw.table.SourceTable.getParentKey()). This occurs naturally in the case of SourceCatalogs produced by the detection and deblending tasks, but it may not be true when concatenating multiple such catalogs. This is indeed the case for forcedPhotCcd.py as a given CCD can be overlapped by multiple patches, thus requiring a concatenation of the reference catalogs of all overlapping patches.

      There two places in the running of forcedPhotCcd.py where calls to getChildren() can cause a failure: one in the subset() function in references.py, and the other in the run function of SingleFrameMeasurementTask in sfm.py.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -
          Show
          price Paul Price added a comment - Is this relevant: https://hsc-jira.astro.princeton.edu/jira/browse/HSC-1215 ?
          Hide
          lauren Lauren MacArthur added a comment -

          That has already been ported in DM-2606. This is actually a sorting of the catalog by parent id's (i.e. not the sorting of the peak's by value). Jim Bosch is under the impression that this requirement is new to LSST (in anticipation of having to do more catalog concatenation), and thus there is no associated fix on the HSC side. Do you concur? I'm about to push a branch with a fix. Would you mind giving it a review Paul Price?

          Show
          lauren Lauren MacArthur added a comment - That has already been ported in DM-2606 . This is actually a sorting of the catalog by parent id's (i.e. not the sorting of the peak's by value). Jim Bosch is under the impression that this requirement is new to LSST (in anticipation of having to do more catalog concatenation), and thus there is no associated fix on the HSC side. Do you concur? I'm about to push a branch with a fix. Would you mind giving it a review Paul Price ?
          Hide
          lauren Lauren MacArthur added a comment -

          Paul, would you mind giving this a look. With this fix forcedPhotCcd.py now runs through. I've checked that it is getting the appropriate list of sources from the reference catalogs in a case where there were 4 overlapping patches for a given CCD. The attached plot shows the ref catalogs from the four patches as circles (cyan, gray, magenta, green) and the output catalog from forcedPhotCcd.py as blue crosses. You may notice that there are a number of sources from the former that do not have entries in the latter. I've confirmed that they are being removed due to the True setting of the config parameter removePatchOverlaps. I'm not sure if the spattering of points at the bottom left that seem to be outside the CCD boundaries should have made it into the catalog, but they all get Runtime or Length Errors in the measurement stage, so do not get measurements and have appropriate flags set.

          A Jenkins build was successful.

          Show
          lauren Lauren MacArthur added a comment - Paul, would you mind giving this a look. With this fix forcedPhotCcd.py now runs through. I've checked that it is getting the appropriate list of sources from the reference catalogs in a case where there were 4 overlapping patches for a given CCD. The attached plot shows the ref catalogs from the four patches as circles (cyan, gray, magenta, green) and the output catalog from forcedPhotCcd.py as blue crosses. You may notice that there are a number of sources from the former that do not have entries in the latter. I've confirmed that they are being removed due to the True setting of the config parameter removePatchOverlaps . I'm not sure if the spattering of points at the bottom left that seem to be outside the CCD boundaries should have made it into the catalog, but they all get Runtime or Length Errors in the measurement stage, so do not get measurements and have appropriate flags set. A Jenkins build was successful.
          Hide
          price Paul Price added a comment -

          Straight-forward patch with good explanation.

          Show
          price Paul Price added a comment - Straight-forward patch with good explanation.
          Hide
          lauren Lauren MacArthur added a comment -

          Thanks Paul. Merged to master.

          Show
          lauren Lauren MacArthur added a comment - Thanks Paul. Merged to master.

            People

            • Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Paul Price
              Watchers:
              Lauren MacArthur, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel