Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28858

Compare the data products of the gen2 vs. gen3 ci_hsc's up to Single Frame Processing

    XMLWordPrintable

    Details

    • Story Points:
      7
    • Epic Link:
    • Sprint:
      DRP S21a (Dec Jan)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As part of the acceptance criteria towards deprecating the "gen2" middleware in favor of the almost-ready-for-prime-time "gen3" middleware, detailed consistency checks between the data products produced by each must be made. This may not require bitwise-esque identity in all cases, but any differences should be understood and deemed acceptable/preferable on the gen3 side of things. As a first step, on this ticket, processing outputs will be assessed up to and including the Single Frame Processing stage using the outputs of the two HSC continuous integration packages (ci_hsc_gen2 vs. ci_hsc_gen3), which probe a limited dataset (in testdata_ci_hsc), but provide a nice starting point for these comparisons. In particular, the outputs created by ci_hsc_gen2 include a conversion of its gen2 repo into a gen3 one. This may ease certain types of comparisons (and could provide another sanity check on the middleware flavor conversion scripts).

      This ticket is a precursor to a more complete and thorough consistency check to be done on DM-21872.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Note to self: Keep DM-28806 in mind while becoming more familiar with the structure and execution of these scripts.

            Show
            lauren Lauren MacArthur added a comment - Note to self: Keep DM-28806 in mind while becoming more familiar with the structure and execution of these scripts.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            I have made some detailed comparisons of the outputs from the two ci_hsc repos based on w_2020_08. This includes comparisons of the following for each visit/ccd combination for both the ic and calexp versions of the following:

            • image arrays (image, variance, mask planes)
            • photoCalib objects
            • PSFs
            • WCSs
            • every column in the source tables

            I think the main conclusion is that things look pretty consistent!

            Here is a quick summary:

            All images and (central CCD position) PSFs are bitwise identical for both ic and calexp.

            Differences appear in some of the photoCalib objects associated with the calexp objects, e.g.

            Comparing gen2 vs. gen3 calexps for HSC-R 903334 16:
            photoCalib diff: calMean: 3.339e-08 calErr: 1.086e-11 instFluxAtMag0: -2.502e+04
            

            Differences appear in the calexp WCSs, e.g.:

            gen2 vs. gen3 calexp WCSs for HSC-R 903334  16: Pixel Scale Diff -0.0582 (mas)
            

            Consequently, differences appear in the calexp src coords, e.g.

            Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:
            coord_ra   Absolute diff (mas): mean: 2.273  min: 0.001  max 25.897
            coord_dec  Absolute diff (mas): mean: 1.596  min: 0.000  max 12.967
            

            All deblend_peakId values in the calexp src catalogs that are not 0 are offset by a constant integer, e.g:

            Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:
            deblend_peakId id number offset: 284224
            Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 22:
            deblend_peakId id number offset: 196013
            

            The actual sources "used" in astrom/calib numbers can differ by small(ish) numbers ("total" here is the total number of sources in the catalog, the number of actual calib "used" sources ranges from ~110 - 280), e.g.

            Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 16:
            calib_astrometry_used Number of differences: 12 of 1337 total
            calib_photometry_used Number of differences:  4 of 1337 total
            Comparing gen2 vs. gen3 src catalogs for HSC-R 903334 22:
            calib_astrometry_used Number of differences:  3 of 1256 total
            

            The above differences likely explain the WCS differences.

            I'm not sure if it's of interest, but the ordering of schema.getOrderedNames() in the icSrc catalogs differs from gen2 to gen3 (are the plugins not being executed in a deterministic order in ISR as they are for SFP?). The set s are the same, though.

            For (much) more detail, I attach a rendering of the notebook I wrote to make these comparisons which shows the full output for all visit/ccd comparisons.

            Show
            lauren Lauren MacArthur added a comment - - edited I have made some detailed comparisons of the outputs from the two ci_hsc repos based on w_2020_08 . This includes comparisons of the following for each visit/ccd combination for both the ic and calexp versions of the following: image arrays (image, variance, mask planes) photoCalib objects PSFs WCSs every column in the source tables I think the main conclusion is that things look pretty consistent! Here is a quick summary: All images and (central CCD position) PSFs are bitwise identical for both ic and calexp . Differences appear in some of the photoCalib objects associated with the calexp objects, e.g. Comparing gen2 vs. gen3 calexps for HSC - R 903334 16 : photoCalib diff: calMean: 3.339e - 08 calErr: 1.086e - 11 instFluxAtMag0: - 2.502e + 04 Differences appear in the calexp WCSs, e.g.: gen2 vs. gen3 calexp WCSs for HSC - R 903334 16 : Pixel Scale Diff - 0.0582 (mas) Consequently, differences appear in the calexp src coords, e.g. Comparing gen2 vs. gen3 src catalogs for HSC - R 903334 16 : coord_ra Absolute diff (mas): mean: 2.273 min : 0.001 max 25.897 coord_dec Absolute diff (mas): mean: 1.596 min : 0.000 max 12.967 All deblend_peakId values in the calexp src catalogs that are not 0 are offset by a constant integer, e.g: Comparing gen2 vs. gen3 src catalogs for HSC - R 903334 16 : deblend_peakId id number offset: 284224 Comparing gen2 vs. gen3 src catalogs for HSC - R 903334 22 : deblend_peakId id number offset: 196013 The actual sources "used" in astrom/calib numbers can differ by small(ish) numbers ("total" here is the total number of sources in the catalog, the number of actual calib "used" sources ranges from ~110 - 280), e.g. Comparing gen2 vs. gen3 src catalogs for HSC - R 903334 16 : calib_astrometry_used Number of differences: 12 of 1337 total calib_photometry_used Number of differences: 4 of 1337 total Comparing gen2 vs. gen3 src catalogs for HSC - R 903334 22 : calib_astrometry_used Number of differences: 3 of 1256 total The above differences likely explain the WCS differences. I'm not sure if it's of interest, but the ordering of schema.getOrderedNames() in the icSrc catalogs differs from gen2 to gen3 (are the plugins not being executed in a deterministic order in ISR as they are for SFP?). The set s are the same, though. For (much) more detail, I attach a rendering of the notebook I wrote to make these comparisons which shows the full output for all visit/ccd comparisons.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            I'm throwing this one to you, Jim, but feel free to redirect if appropriate! I am interested in the thresholds/tolerances for the differences here and if they require follow-up. My initial instinct is that they are, for the most part, quite small and mostly explained by small differences in the sources that get used in the astrometric and photometric calibrations. I suspect we should be able to make these consistent in both cases(?). If that is desired, please let me know and I'd be happy to look into that (here or on another ticket).

            Show
            lauren Lauren MacArthur added a comment - - edited I'm throwing this one to you, Jim, but feel free to redirect if appropriate! I am interested in the thresholds/tolerances for the differences here and if they require follow-up. My initial instinct is that they are, for the most part, quite small and mostly explained by small differences in the sources that get used in the astrometric and photometric calibrations. I suspect we should be able to make these consistent in both cases(?). If that is desired, please let me know and I'd be happy to look into that (here or on another ticket).
            Hide
            jbosch Jim Bosch added a comment -

            Thanks for the detailed investigation, and I'm very glad to see that the results are already so close (and especially that the images are already identical).

            Matching to the reference catalog is something I expect to produce slightly different results, because it's a very different code path to determine which initial reference catalog objects are provided to the Tasks. Since we're already so close to being exactly identical in other respects, I'm tempted to try to see if we can make that difference go away as well, and make our coadds identical, too. I suspect the order of the reference objects passed in is at least different, so deterministic sorting on both sides might help. And it's probable that the Gen3 reference objects being passed in are a superset of those Gen2 is getting; if that's the case, we ought to be able to make the Gen3 code path do the exact same down-selection Gen2 is doing, and then we'll really be in business. Would you mind opening a new ticket to look into that, and taking it on? Might involve getting in there with a debugger to see exactly how the reference catalog objects differ, but I think it'll be worth it if we can make the coadd images the same (or at least give Jointcal and FGCM the exact same inputs).

            I'm not quite sure what's going on with the deblend_peakId values; that's a new field that I'm not super familiar with yet, and I know the logic that generates the visit+detector integer IDs that gets mangled into many of our source IDs hasn't settled down yet. I'm working on my own tickets to deal with the latter, so let's revisit this when those are done. In any case, a constant integer offset is an easy difference to deal with even if it stays around and propagates through.

            The schema ordering difference is one I don't understand. The plugins are sorted by name using the same (subtask) code in Gen2 and Gen3, and it's the same code in CharacterizeImageTask and CalibrateTask. So I'm guessing the ordering is related to when fields from one or more other subtasks are added (propagated flags? stuff from astrometry or deblending?) relative to each other or the measurement plugins. Could you look at icSrcSchema.getOrderedNames() again to see if you can spot any blocks of names that have been swapped, reflecting a task-initialization reordering that we might be able to revert?

            Show
            jbosch Jim Bosch added a comment - Thanks for the detailed investigation, and I'm very glad to see that the results are already so close (and especially that the images are already identical). Matching to the reference catalog is something I expect to produce slightly different results, because it's a very different code path to determine which initial reference catalog objects are provided to the Tasks. Since we're already so close to being exactly identical in other respects, I'm tempted to try to see if we can make that difference go away as well, and make our coadds identical, too. I suspect the order of the reference objects passed in is at least different, so deterministic sorting on both sides might help. And it's probable that the Gen3 reference objects being passed in are a superset of those Gen2 is getting; if that's the case, we ought to be able to make the Gen3 code path do the exact same down-selection Gen2 is doing, and then we'll really be in business. Would you mind opening a new ticket to look into that, and taking it on? Might involve getting in there with a debugger to see exactly how the reference catalog objects differ, but I think it'll be worth it if we can make the coadd images the same (or at least give Jointcal and FGCM the exact same inputs). I'm not quite sure what's going on with the deblend_peakId values; that's a new field that I'm not super familiar with yet, and I know the logic that generates the visit+detector integer IDs that gets mangled into many of our source IDs hasn't settled down yet. I'm working on my own tickets to deal with the latter, so let's revisit this when those are done. In any case, a constant integer offset is an easy difference to deal with even if it stays around and propagates through. The schema ordering difference is one I don't understand. The plugins are sorted by name using the same (subtask) code in Gen2 and Gen3, and it's the same code in CharacterizeImageTask and CalibrateTask . So I'm guessing the ordering is related to when fields from one or more other subtasks are added (propagated flags? stuff from astrometry or deblending?) relative to each other or the measurement plugins. Could you look at icSrcSchema.getOrderedNames() again to see if you can spot any blocks of names that have been swapped, reflecting a task-initialization reordering that we might be able to revert?
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks for the review Jim. I've created DM-28936 to try to sync up the calibration source selections and will get started on that soon. I've also looked more deeply into the schema ordering issue. It was stemming from the aperture correction "sets" getting populated in different orders, e.g.:
            Gen2:

            _ApCorrNameSet =  {'ext_convolved_ConvolvedFlux_2_kron', 'modelfit_CModel_initial', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_0_kron', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_3_4_5', 'base_GaussianFlux', 'modelfit_CModel_exp', 'ext_photometryKron_KronFlux', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_1_3_3', 'ext_convolved_ConvolvedFlux_3_3_3', 'ext_convolved_ConvolvedFlux_3_kron', 'ext_convolved_ConvolvedFlux_3_6_0', 'ext_convolved_ConvolvedFlux_0_4_5', 'base_PsfFlux', 'modelfit_CModel_dev', 'ext_convolved_ConvolvedFlux_1_kron', 'modelfit_CModel', 'ext_convolved_ConvolvedFlux_0_3_3'}
            

            Gen3:

            _ApCorrNameSet =  {'ext_photometryKron_KronFlux', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_3_4_5', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_0_kron', 'modelfit_CModel_dev', 'modelfit_CModel_exp', 'ext_convolved_ConvolvedFlux_0_4_5', 'ext_convolved_ConvolvedFlux_3_6_0', 'modelfit_CModel_initial', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_3_3_3', 'base_GaussianFlux', 'ext_convolved_ConvolvedFlux_0_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_2_kron', 'ext_convolved_ConvolvedFlux_3_kron', 'modelfit_CModel', 'ext_convolved_ConvolvedFlux_1_3_3', 'base_PsfFlux', 'ext_convolved_ConvolvedFlux_1_kron'}
            

            I was puzzled having seen your commit to make the apCorr ordering deterministic here.
            However, I then found that the same sorting is not being done where the apcorr_{name}_used columns get set here.
            This also explains why this only turned up in the icSrc catalogs as these columns do not get added to the src catalogs.

            As a fix, I've pushed a meas_algorithms branch to add the sorting. I also tried to add a "best effort" to the unittest along the lines of what you did in meas_base. Could you give this PR a review as well? I reran the two ci_hsc s against this branch and can confirm that I am now seeing the same ordering of the gen2 and gen3 icSrc schemas.  I've also got a Jenkins + ci_hsc running, but it seems that is destined to fail on ci_hsc until an unrelated bug gets fixed (I'll run it again when that happens).

            Show
            lauren Lauren MacArthur added a comment - Thanks for the review Jim. I've created DM-28936 to try to sync up the calibration source selections and will get started on that soon. I've also looked more deeply into the schema ordering issue. It was stemming from the aperture correction "sets" getting populated in different orders, e.g.: Gen2: _ApCorrNameSet = { 'ext_convolved_ConvolvedFlux_2_kron' , 'modelfit_CModel_initial' , 'ext_convolved_ConvolvedFlux_1_6_0' , 'ext_convolved_ConvolvedFlux_2_3_3' , 'ext_convolved_ConvolvedFlux_1_4_5' , 'ext_convolved_ConvolvedFlux_0_kron' , 'ext_convolved_ConvolvedFlux_2_4_5' , 'ext_convolved_ConvolvedFlux_0_6_0' , 'ext_convolved_ConvolvedFlux_3_4_5' , 'base_GaussianFlux' , 'modelfit_CModel_exp' , 'ext_photometryKron_KronFlux' , 'ext_convolved_ConvolvedFlux_2_6_0' , 'ext_convolved_ConvolvedFlux_1_3_3' , 'ext_convolved_ConvolvedFlux_3_3_3' , 'ext_convolved_ConvolvedFlux_3_kron' , 'ext_convolved_ConvolvedFlux_3_6_0' , 'ext_convolved_ConvolvedFlux_0_4_5' , 'base_PsfFlux' , 'modelfit_CModel_dev' , 'ext_convolved_ConvolvedFlux_1_kron' , 'modelfit_CModel' , 'ext_convolved_ConvolvedFlux_0_3_3' } Gen3: _ApCorrNameSet = { 'ext_photometryKron_KronFlux' , 'ext_convolved_ConvolvedFlux_0_6_0' , 'ext_convolved_ConvolvedFlux_3_4_5' , 'ext_convolved_ConvolvedFlux_2_4_5' , 'ext_convolved_ConvolvedFlux_2_3_3' , 'ext_convolved_ConvolvedFlux_0_kron' , 'modelfit_CModel_dev' , 'modelfit_CModel_exp' , 'ext_convolved_ConvolvedFlux_0_4_5' , 'ext_convolved_ConvolvedFlux_3_6_0' , 'modelfit_CModel_initial' , 'ext_convolved_ConvolvedFlux_1_6_0' , 'ext_convolved_ConvolvedFlux_3_3_3' , 'base_GaussianFlux' , 'ext_convolved_ConvolvedFlux_0_3_3' , 'ext_convolved_ConvolvedFlux_1_4_5' , 'ext_convolved_ConvolvedFlux_2_6_0' , 'ext_convolved_ConvolvedFlux_2_kron' , 'ext_convolved_ConvolvedFlux_3_kron' , 'modelfit_CModel' , 'ext_convolved_ConvolvedFlux_1_3_3' , 'base_PsfFlux' , 'ext_convolved_ConvolvedFlux_1_kron' } I was puzzled having seen your commit to make the apCorr ordering deterministic here . However, I then found that the same sorting is not being done where the apcorr_{name}_used columns get set here . This also explains why this only turned up in the icSrc catalogs as these columns do not get added to the src catalogs. As a fix, I've pushed a meas_algorithms branch to add the sorting. I also tried to add a "best effort" to the unittest along the lines of what you did in meas_base . Could you give this PR a review as well? I reran the two ci_hsc s against this branch and can confirm that I am now seeing the same ordering of the gen2 and gen3 icSrc schemas.  I've also got a Jenkins + ci_hsc running , but it seems that is destined to fail on ci_hsc until an unrelated bug gets fixed (I'll run it again when that happens).
            Hide
            lauren Lauren MacArthur added a comment -

            In addressing your review comments, I've added another PR just to keep things consistent (sorry!) I've also kicked off another Jenkins + ci_hsc which I think has a shot at passing this time.

            Show
            lauren Lauren MacArthur added a comment - In addressing your review comments, I've added another PR just to keep things consistent (sorry!) I've also kicked off another Jenkins + ci_hsc which I think has a shot at passing this time.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks again Jim. Jenkins passed, so both branches have been merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks again Jim. Jenkins passed, so both branches have been merged to master.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.