Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21052

Update the pipe_analysis scripts to have the option to apply fgcm calibrations

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      10
    • Epic Link:
    • Sprint:
      DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan), DRP S20-3 (Feb)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The fgcmcal "Forward Global Calibration Method" calibration is adding a new dataset type and product in the form of the usual photoCalib object (called fgcm_tract_photoCalib but currently only defined in obs_subaru, see DM-16704). The scripts in pipe_analysis require updating to allow for the option of applying fgcm photometric calibrations in lieu of the jointcal ones.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23579 visitAnslysis for HSC-PDR2

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23580 CompareVisitAnalysis for HSC-PDR2

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20161 Run tract-based fgcmcal on RC2 dataset

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16704 Add fgcmcal-tract mode to fgcmcal for small-scale processing with a reference catalog

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            The application here will also follow the naming scheme for configs established in DM-21308.

            Show
            lauren Lauren MacArthur added a comment - The application here will also follow the naming scheme for configs established in DM-21308 .
            Hide
            Permalink
            lauren Lauren MacArthur added a comment - - edited

            The scripts on this branch now allow the user to choose which external calibrations to apply to the visit-level src catalogs. Their application and which flavor to apply are determined by config parameters (as in pipe_tasks for coaddition):

            doApplyExternalPhotoCalib
            		 
            externalPhotoCalibName
            		 
            doApplyExternalSkyWcs
            		 
            externalSkyWcsName

            (there is still accommodation for any vestigial meas_mosaic runs...but I doubt they will ever be used going forward).

            The compareVisitAnalysis.py script can be run with, e.g. 

            externalPhotoCalibName1="fgcm" externalPhotoCalibName2="jointcal"

            in order to directly compare the mapping between the two calibrations. As an example, this is the mapping between the fgcm & jointcal photometric calibrations for a COSMOS HSC-I band visit:

            Most uses of the term "uber" in regards to calibrations have been replaced with "external". The one exception, as described in the commit message, is a variable named uberCalLabel as doing this change now would cause some serious rebasing nightmares for a few far-reaching tickets that our soon to land. It will be updated to externalCalLabel once a less disruptive phase is reached.

            Show
            lauren Lauren MacArthur added a comment - - edited The scripts on this branch now allow the user to choose which external calibrations to apply to the visit-level src catalogs. Their application and which flavor to apply are determined by config parameters (as in pipe_tasks for coaddition): doApplyExternalPhotoCalib externalPhotoCalibName doApplyExternalSkyWcs externalSkyWcsName (there is still accommodation for any vestigial meas_mosaic runs...but I doubt they will ever be used going forward). The compareVisitAnalysis.py script can be run with, e.g. externalPhotoCalibName1 = "fgcm" externalPhotoCalibName2 = "jointcal" in order to directly compare the mapping between the two calibrations. As an example, this is the mapping between the fgcm & jointcal photometric calibrations for a COSMOS HSC-I band visit: Most uses of the term "uber" in regards to calibrations have been replaced with "external". The one exception, as described in the commit message, is a variable named uberCalLabel as doing this change now would cause some serious rebasing nightmares for a few far-reaching tickets that our soon to land. It will be updated to externalCalLabel once a less disruptive phase is reached.
            Hide
            Permalink
            lauren Lauren MacArthur added a comment - - edited

            Would you mind reviewing this?  You have seen many of the relevant plots demonstrating the functionality already, but you can dig deeper if you'd like looking at the plots in https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-21052/plots (some don't use the functionality directly, but I ran all scripts to make sure nothing outright broke and that the labels are appropriate and reasonably placed).

            Oh, and a question for you in particular:  what do you think the default should be for externalPhotoCalibName : "jointcal", "fgcm", or "fgcm_tract"?

            PR is: https://github.com/lsst-dm/pipe_analysis/pull/49

            Show
            lauren Lauren MacArthur added a comment - - edited Would you mind reviewing this?  You have seen many of the relevant plots demonstrating the functionality already, but you can dig deeper if you'd like looking at the plots in https://lsst-web.ncsa.illinois.edu/~lauren/lauren/DM-21052/plots  (some don't use the functionality directly, but I ran all scripts to make sure nothing outright broke and that the labels are appropriate and reasonably placed). Oh, and a question for you in particular:  what do you think the default should be for externalPhotoCalibName : "jointcal", "fgcm", or "fgcm_tract"? PR is: https://github.com/lsst-dm/pipe_analysis/pull/49
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Looks good.

            Show
            erykoff Eli Rykoff added a comment - Looks good.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Eli Rykoff
              Watchers:
              Eli Rykoff, Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.