Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32624

Override source selection criteria in SFM calibration stages for LSSTCam-imSim

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      In the analysis done on DM-16875, the suggestion was made to override the default config settings for the source selection criteria for consideration of a given source as a suitable candidate for the PSF modeling. In particular, the default setting of a minimum flux of 12500 (raw counts, I believe) seems way too high and, ideally, the cut would not be based on flux, but rather on S/N. Justification for this based on HSC data as well as the S/N thresholding functionality was added on DM-17043. [Note that the fluxMin value has long been overridden to 4000 in obs_subaru].

      It seemed this change was widely agreed upon by those involved in the imsim/DC2 Run2.2i processing and was adopted – in addition to several other source selection overrides for other calibration phases – in at least some of the DESC workflows (see, e.g. the overrides in their "dc2/run2.2" branch run: https://github.com/lsst/obs_lsst/blob/dc2/run2.2/config/imsim/processCcd.py#L39-L42
      and note the other overrides in that file). Unfortunately, during an investigation on PREOPS-885, it was noted that these overrides have not found their way into the master/main config files in obs_lsst, so they are not currently in effect for any processing based off the master/main configs. This situation should be rectified.

      Some care may be required in the actual S/N threshold set and whether the other updates in the file pointed to above should also be adopted. In particular, I would love to hear from anyone involved the DC2 processing campaigns as to the validation of those choices (I added a few watchers as a first guess, but please do add/remove as you see fit!) I also have no idea what was used in the DP0.1 processing run, so if anyone could chime in on that, it would be greatly appreciated!

        Attachments

          Issue Links

            Activity

            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue relates to DM-16785 [ DM-16785 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-17043 [ DM-17043 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to PREOPS-885 [ PREOPS-885 ]
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            Description In the analysis done on DM-16875, the suggestion was made to override the default config settings for the source selection criteria for consideration of a given source as a suitable candidate for the PSF modeling. In particular, the default setting of a minimum flux of 12500 (raw counts, I believe) seems way too high and, ideally, the cut would not be based on flux, but rather on S/N. Justification for this based on HSC data as well as the S/N thresholding functionality was added on DM-17043. [Note that the {{fluxMin}} value has long been overridden to 4000 in {{obs_subaru}}].

            It seemed this change was widely agreed upon by those involved in the {{imsim}}/DC2 Run2.2i processing and was adopted – in addition to several other source selection overrides for other calibration phases – in at least some of the DESC workflows (see, e.g. the overrides in their "dc2/run2.2" branch run: [https://github.com/lsst/obs_lsst/blob/dc2/run2.2/config/imsim/processCcd.py#L39-L42]
            and note the other overrides in that file). Unfortunately, during an investigation on PREOPS-885, it was noted that these overrides have not found their way into the master/main config files in {{obs_lsst}}, so they are not currently in effect for any processing based off the master/main configs. This situation should be rectified.

            Some care may be required in the actual S/N threshold set and whether the other updates in the file pointed to above should also be adopted. In particular, I would love to hear from anyone involved the DC2 processing campaigns as to the validation of those choices (I added a few watchers as a first guess, but please do add/remove as you see fit!) I also have no idea what was used in the DP0.1 processing run, so if anyone could chime in on that, it would be greatly appreciated!
            In the analysis done on DM-16875, the suggestion was made to override the default config settings for the source selection criteria for consideration of a given source as a suitable candidate for the PSF modeling. In particular, the default setting of a minimum flux of 12500 (raw counts, I believe) seems way too high and, ideally, the cut would not be based on flux, but rather on S/N. Justification for this based on HSC data as well as the S/N thresholding functionality was added on DM-17043. \[Note that the {{fluxMin}} value has long been overridden to 4000 in {{obs_subaru}}\].

            It seemed this change was widely agreed upon by those involved in the {{imsim}}/DC2 Run2.2i processing and was adopted – in addition to several other source selection overrides for other calibration phases – in at least some of the DESC workflows (see, e.g. the overrides in their "dc2/run2.2" branch run: [https://github.com/lsst/obs_lsst/blob/dc2/run2.2/config/imsim/processCcd.py#L39-L42]
            and note the other overrides in that file). Unfortunately, during an investigation on PREOPS-885, it was noted that these overrides have not found their way into the master/main config files in {{obs_lsst}}, so they are not currently in effect for any processing based off the master/main configs. This situation should be rectified.

            Some care may be required in the actual S/N threshold set and whether the other updates in the file pointed to above should also be adopted. In particular, I would love to hear from anyone involved the DC2 processing campaigns as to the validation of those choices (I added a few watchers as a first guess, but please do add/remove as you see fit!) I also have no idea what was used in the DP0.1 processing run, so if anyone could chime in on that, it would be greatly appreciated!
            lauren Lauren MacArthur made changes -
            Summary Override source selection criteria in SFM calibration stages fo LSSTCam-imSim Override source selection criteria in SFM calibration stages for LSSTCam-imSim
            ctslater Colin Slater made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31518 ]
            lauren Lauren MacArthur made changes -
            Attachment plot-v2334-u-base_GausssianFluxSn_raw-hist.png [ 54891 ]
            lauren Lauren MacArthur made changes -
            Assignee Lauren MacArthur [ lauren ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            Attachment plot-v2334-u-base_GausssianFluxSn_raw-hist.png [ 54891 ]
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            Attachment psfEllipRediduals_test-med-1.pdf [ 54936 ]
            lauren Lauren MacArthur made changes -
            Attachment psfScaledSizeScatter_test-med-1.pdf [ 54937 ]
            lauren Lauren MacArthur made changes -
            Attachment psfScaledSizeScatter_test-med-1.pdf [ 54937 ]
            lauren Lauren MacArthur made changes -
            Attachment psfEllipRediduals_test-med-1.pdf [ 54936 ]
            lauren Lauren MacArthur made changes -
            Attachment psfEllipRediduals_test-med-1.png [ 54938 ]
            Attachment psfScaledSizeScatter_test-med-1.png [ 54939 ]
            lauren Lauren MacArthur made changes -
            Attachment psfScaledSizeScatter_HSC_RC2.png [ 54940 ]
            Attachment psfEllipRediduals_HSC_RC2.png [ 54941 ]
            lauren Lauren MacArthur made changes -
            Reviewers Yusra AlSayyad [ yusra ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            lauren Lauren MacArthur made changes -
            Labels SciencePipelines SciencePipelines backport-v23
            yusra Yusra AlSayyad made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            yusra Yusra AlSayyad made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Labels SciencePipelines backport-v23 SciencePipelines backport-approved backport-v23
            ctslater Colin Slater made changes -
            Remote Link This issue links to "Page (Confluence)" [ 31568 ]
            lauren Lauren MacArthur made changes -
            Labels SciencePipelines backport-approved backport-v23 SciencePipelines backport-approved backport-done backport-v23
            lauren Lauren MacArthur made changes -
            Story Points 5
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30540 [ 511198 ]

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Colin Slater, Eli Rykoff, Heather Kelly, James Chiang, Jim Bosch, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.