Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6815

Update LSST full-stack processing configuration to match best practice from HSC

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 10
    • DRP F16-2, DRP F16-3, DRP F16-4
    • Data Release Production

    Description

      In preparation for running an end-to-end comparison of large scale processing with the HSC and LSST stacks, we need to update the configuration to reflect currently understood best practice.

      In general, we expect the default HSC configuration to be better understood and "battle-tested" given that it has been used for science-grade data releases.

      Audit the default configuration of the full LSST stack (from ProcessCcdTask through multiband coadd processing). Where LSST defaults differ from HSC, update the LSST configuration to match the HSC equivalent unless there's a clear reason why LSST's default should be different. When it's not appropriate to update the LSST configuration, add an override to obs_subaru.

      In some cases, the LSST and HSC stacks have diverged so that a direct transfer of configuration options isn't possible. Where an equivalent can be found, take advantage of it. Otherwise, stick with existing LSST defaults.

      Attachments

        1. demoSpatialStructureUseApprox.png
          demoSpatialStructureUseApprox.png
          79 kB
        2. demoSpatialStructure2.png
          demoSpatialStructure2.png
          75 kB
        3. demoSpatialStructure.png
          demoSpatialStructure.png
          212 kB
        4. demoPsfFlux.png
          demoPsfFlux.png
          15 kB
        5. demoJunkSuppression.png
          demoJunkSuppression.png
          30 kB

        Issue Links

          Activity

            rhl, quick update: having now set useApprox=True the apparent gradient is gone. Deep interpretation of any differences is difficult as the demo previously ran with no junk suppression and performing background estimation via interpolation.

            lauren Lauren MacArthur added a comment - rhl , quick update: having now set useApprox=True the apparent gradient is gone. Deep interpretation of any differences is difficult as the demo previously ran with no junk suppression and performing background estimation via interpolation.
            price Paul Price added a comment -

            Some comments on GitHub PRs.

            I'm concerned about making applyUberCal=True the default because it will break cameras for which jointcal or meas_mosaic isn't working.

            price Paul Price added a comment - Some comments on GitHub PRs. I'm concerned about making applyUberCal=True the default because it will break cameras for which jointcal or meas_mosaic isn't working.

            price: I've updated to keep applyUberCal=False as the codebase default, but am overriding it to True in obs_subaru by default for HSC coaddition. Also, because the QA analysis script makes use of the extendedness value in the forced coadd results, I've added the following to config/forcedPhotCoadd.py:

            config.catalogCalculation.plugins.names=["base_ClassificationExtendedness"]
            

            Are you ok with these changes?

            lauren Lauren MacArthur added a comment - price : I've updated to keep applyUberCal=False as the codebase default, but am overriding it to True in obs_subaru by default for HSC coaddition. Also, because the QA analysis script makes use of the extendedness value in the forced coadd results, I've added the following to config/forcedPhotCoadd.py : config.catalogCalculation.plugins.names=["base_ClassificationExtendedness"] Are you ok with these changes?
            price Paul Price added a comment -

            Yes, thanks!

            price Paul Price added a comment - Yes, thanks!

            Thanks to everyone for all of the input on this one.

            Merged to master.

            lauren Lauren MacArthur added a comment - Thanks to everyone for all of the input on this one. Merged to master.

            People

              lauren Lauren MacArthur
              swinbank John Swinbank
              Paul Price
              Bob Armstrong, Jim Bosch, John Swinbank, Lauren MacArthur, Paul Price, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.