Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16392

Aperture correction field keys not guaranteed to point the same offsets within a given reprocessing



    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • meas_base
    • None
    • 0.5
    • DRP F18-5
    • Data Release Production


      In perusing the pipe_analysis outputs of the RC2 reprocessing runs to check that our results are all looking as they should, I noticed that the aperture corrections were comparing differently between reprocessing runs (e.g. w42 vs. w41). While, in principle, this could happen as a result of code changes, this did not appear to be the case as the flux comparisons were all identical. Digging into it, I discovered that the key offsets point to a different place for different patches/visits for all aperture correction fields within the same processing run. I.e. the catalog schemas don't match each other, nor the one defined in the schema files in the repo.

      E.g. in /datasets/hsc/repo/rerun/RC/w_2018_42/DM-16095/, the meas cat schema in schema/deepCoadd_meas.fits has:

      base_GaussianFlux_apCorr Key<D>(offset=1832, nElements=1)
      base_PsfFlux_apCorr Key<D>(offset=1992, nElements=1)

      But for two patches, we find:

      dataId = {"patch": "8,0", "tract": 9615, "filter":"HSC-I"}
      base_GaussianFlux_apCorr Key<D>(offset=2040, nElements=1)
      base_PsfFlux_apCorr Key<D>(offset=1992, nElements=1)

      dataId = {"patch": "3,7", "tract": 9615, "filter":"HSC-I"}
      base_GaussianFlux_apCorr Key<D>(offset=1880, nElements=1)
      base_PsfFlux_apCorr Key<D>(offset=1928, nElements=1)

      This is most undesirable as it precludes concatenating all patches in one tract into a single catalog (which requires them all to have identical schemas).  The fix may be as simple as looping over a sorted list (as opposed to one where order is not guaranteed) when adding the apCorr keys to the schema.



          lauren Lauren MacArthur created issue -
          lauren Lauren MacArthur made changes -
          Field Original Value New Value
          Epic Link DM-14405 [ 79812 ]
          jbosch Jim Bosch made changes -
          Assignee Jim Bosch [ jbosch ]
          Status To Do [ 10001 ] In Progress [ 3 ]
          jbosch Jim Bosch made changes -
          Story Points 0.5
          jbosch Jim Bosch made changes -
          Sprint DRP F18-5 [ 790 ]
          jbosch Jim Bosch made changes -
          Reviewers Lauren MacArthur [ lauren ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          lauren Lauren MacArthur made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          jbosch Jim Bosch made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]


            jbosch Jim Bosch
            lauren Lauren MacArthur
            Lauren MacArthur
            Jim Bosch, John Swinbank, Lauren MacArthur, Yusra AlSayyad
            0 Vote for this issue
            4 Start watching this issue




                No builds found.