Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28985

Update Gen2 calibration policy formats in obs_subaru

    XMLWordPrintable

Details

    • 4
    • DRP S21a (Dec Jan)
    • Data Release Production
    • No

    Description

      The Gen2 obs_subaru policy file policy/HscMapper.yaml defines bias and dark calibrations to be of type ImageF, which, by definition, does not have mask or variance planes. On read, these are passed through a standardization step that converts ImageF to an ExposureF by adding empty mask and variance planes. However, when running a comparison test using Gen3 middleware (on DM-28936), it was revealed that these files do have variance and masks planes, it turns out these calibration files do have mask and variance planes and uses them when updating the variance planes throughout the processing. So, as it stands, Gen2 and Gen3 are behaving slightly differently leading to small (and very likely scientifically insignificant) differences, putting a dent in our efforts towards acheiving Gen2/Gen3 parity through Single Frame Processing. The fix is to update the bias and dark formats in the policy file from ImageF to MaskedImageF, which will be done here.

      Attachments

        Issue Links

          Activity

            As a validation of this fix, I ran two Gen2 processCcd  processing of a single HSC visit (1228 which is HSC-I and in the COSMOS 9813 tract) with the latest w_2021_09 master stack, and against this change.  I also ran the same visit with Gen3.  The latter Gen2 and Gen3 visits now compare bitwise identical for all image planes (image, variance, mask) in both the icExp and calexp outputs for all 103 CCDs.  The src catalogs also compare almost identically, but for the minor differences noted in DM-28858 (based on ci_hsc_gen2 vs. ci_hsc_gen3 comparisons) and currently under investigation (but blocked by this ticket) on DM-28936.

            I also compared the two Gen2 runs. While there are very small differences in the data products due to the minor differences in the variance planes (resulting in slight thresholding differences --> slightly different numbers of CRs detected and source footprint sizes --> slight differences in centroids for deblended sources, etc...), these were discussed during Princeton's regular Monday Meeting on Mar 1, 2021, and it was agreed they were indeed at too small a level to cause any concern and that this change should be made. Direct comparison plots can be perused (for the time being!) here and I've attached a few examples to the ticket for posterity (make particular note of the scales/units on these comparison plots...differences are tiny!).

            lauren Lauren MacArthur added a comment - As a validation of this fix, I ran two Gen2  processCcd   processing of a single HSC visit (1228 which is HSC-I and in the COSMOS 9813 tract) with the latest w_2021_09  master stack, and against this change.  I also ran the same visit with Gen3.  The latter Gen2 and Gen3 visits now compare bitwise identical for all image planes (image, variance, mask) in both the icExp and calexp outputs for all 103 CCDs.  The src catalogs also compare almost identically, but for the minor differences noted in DM-28858 (based on  ci_hsc_gen2  vs.  ci_hsc_gen3 comparisons) and currently under investigation (but blocked by this ticket) on DM-28936 . I also compared the two Gen2 runs. While there are very small differences in the data products due to the minor differences in the variance planes (resulting in slight thresholding differences --> slightly different numbers of CRs detected and source footprint sizes --> slight differences in centroids for deblended sources, etc...), these were discussed during Princeton's regular Monday Meeting on Mar 1, 2021, and it was agreed they were indeed at too small a level to cause any concern and that this change should be made. Direct comparison plots can be perused (for the time being!) here and I've attached a few examples to the ticket for posterity (make particular note of the scales/units on these comparison plots...differences are tiny!).

            Would you mind giving this a look? I think you were present at the meeting this morning where this issue was discussed (at some length!), but do let me know if you would like any further detail.

            Jenkins + ci_hsc is running.

            lauren Lauren MacArthur added a comment - Would you mind giving this a look? I think you were present at the meeting this morning where this issue was discussed (at some length!), but do let me know if you would like any further detail. Jenkins + ci_hsc is running .

            This all looks fine to me (especially when I double checked scales).  Mostly for my own curiosity, do you know if the outlier detectors in the final two focal plane plots are always different, or if the set changes depending on visit?

            I don't have any concerns, so feel free to merge once Jenkins is happy.

            czw Christopher Waters added a comment - This all looks fine to me (especially when I double checked scales).  Mostly for my own curiosity, do you know if the outlier detectors in the final two focal plane plots are always different, or if the set changes depending on visit? I don't have any concerns, so feel free to merge once Jenkins is happy.

            Thanks for the speedy turnaround, Chris!  No, I don't know if the outlier detectors are always as such as I have only run the one visit.  It's a good question though, so I'll run a few more visits (and in different bands) and have a look.  It could also vary for different CALIBs, so, to start, I'll try to pick visits that will all use the same master bias and dark frames to take that element out of the equation.

            lauren Lauren MacArthur added a comment - Thanks for the speedy turnaround, Chris!  No, I don't know if the outlier detectors are always as such as I have only run the one visit.  It's a good question though, so I'll run a few more visits (and in different bands) and have a look.  It could also vary for different CALIBs, so, to start, I'll try to pick visits that will all use the same master bias and dark frames to take that element out of the equation.

            So...the answer is no.  Having checked that the same CALIBs would be in play:

            rootDirGen2 = "/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654"
            butlerGen2 = dafPersist.Butler(rootDirGen2)
            ccd = 49
            dataType = "bias"
            for visit in [1228, 1220, 1194, 1882,]:
                print(butlerGen2.getUri(dataType, visit=visit, ccd=ccd))
            for visit in [1228, 1220, 1194, 1882,]:
                print(butlerGen2.getUri("calexp", visit=visit, ccd=ccd))
            /datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
            /datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
            /datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
            /datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
            /datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-I/corr/CORR-0001228-049.fits
            /datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-R/corr/CORR-0001220-049.fits
            /datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-Z/corr/CORR-0001194-049.fits
            /datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00823/HSC-Y/corr/CORR-0001882-049.fits
            

            I ran those visits on the ticket branch and made the same comparison plots against the w_20201_06 RC2 run.
            The same general pattern of some outlier CCDs is present, but it seems to be a different set of CCDs each time. The good news is that all differences in the above new runs were all at the same very low levels.

            Jenkins was indeed happy, so merged and done!

            lauren Lauren MacArthur added a comment - So...the answer is no.  Having checked that the same CALIBs would be in play: rootDirGen2 = "/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654" butlerGen2 = dafPersist.Butler(rootDirGen2) ccd = 49 dataType = "bias" for visit in [ 1228 , 1220 , 1194 , 1882 ,]: print (butlerGen2.getUri(dataType, visit = visit, ccd = ccd)) for visit in [ 1228 , 1220 , 1194 , 1882 ,]: print (butlerGen2.getUri( "calexp" , visit = visit, ccd = ccd)) / datasets / hsc / repo / CALIB / BIAS / 2014 - 03 - 29 / NONE / BIAS - 2014 - 03 - 29 - 049.fits / datasets / hsc / repo / CALIB / BIAS / 2014 - 03 - 29 / NONE / BIAS - 2014 - 03 - 29 - 049.fits / datasets / hsc / repo / CALIB / BIAS / 2014 - 03 - 29 / NONE / BIAS - 2014 - 03 - 29 - 049.fits / datasets / hsc / repo / CALIB / BIAS / 2014 - 03 - 29 / NONE / BIAS - 2014 - 03 - 29 - 049.fits / datasets / hsc / repo / rerun / RC / w_2021_06 / DM - 28654 - sfm / 00817 / HSC - I / corr / CORR - 0001228 - 049.fits / datasets / hsc / repo / rerun / RC / w_2021_06 / DM - 28654 - sfm / 00817 / HSC - R / corr / CORR - 0001220 - 049.fits / datasets / hsc / repo / rerun / RC / w_2021_06 / DM - 28654 - sfm / 00817 / HSC - Z / corr / CORR - 0001194 - 049.fits / datasets / hsc / repo / rerun / RC / w_2021_06 / DM - 28654 - sfm / 00823 / HSC - Y / corr / CORR - 0001882 - 049.fits I ran those visits on the ticket branch and made the same comparison plots against the w_20201_06 RC2 run. The same general pattern of some outlier CCDs is present, but it seems to be a different set of CCDs each time. The good news is that all differences in the above new runs were all at the same very low levels. Jenkins was indeed happy, so merged and done!

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Christopher Waters
              Christopher Waters, Jim Bosch, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.