Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6815

Update LSST full-stack processing configuration to match best practice from HSC

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      10
    • Sprint:
      DRP F16-2, DRP F16-3, DRP F16-4
    • Team:
      Data Release Production

      Description

      In preparation for running an end-to-end comparison of large scale processing with the HSC and LSST stacks, we need to update the configuration to reflect currently understood best practice.

      In general, we expect the default HSC configuration to be better understood and "battle-tested" given that it has been used for science-grade data releases.

      Audit the default configuration of the full LSST stack (from ProcessCcdTask through multiband coadd processing). Where LSST defaults differ from HSC, update the LSST configuration to match the HSC equivalent unless there's a clear reason why LSST's default should be different. When it's not appropriate to update the LSST configuration, add an override to obs_subaru.

      In some cases, the LSST and HSC stacks have diverged so that a direct transfer of configuration options isn't possible. Where an equivalent can be found, take advantage of it. Otherwise, stick with existing LSST defaults.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Robert Lupton, quick update: having now set useApprox=True the apparent gradient is gone. Deep interpretation of any differences is difficult as the demo previously ran with no junk suppression and performing background estimation via interpolation.

            Show
            lauren Lauren MacArthur added a comment - Robert Lupton , quick update: having now set useApprox=True the apparent gradient is gone. Deep interpretation of any differences is difficult as the demo previously ran with no junk suppression and performing background estimation via interpolation.
            Hide
            price Paul Price added a comment -

            Some comments on GitHub PRs.

            I'm concerned about making applyUberCal=True the default because it will break cameras for which jointcal or meas_mosaic isn't working.

            Show
            price Paul Price added a comment - Some comments on GitHub PRs. I'm concerned about making applyUberCal=True the default because it will break cameras for which jointcal or meas_mosaic isn't working.
            Hide
            lauren Lauren MacArthur added a comment -

            Paul Price: I've updated to keep applyUberCal=False as the codebase default, but am overriding it to True in obs_subaru by default for HSC coaddition. Also, because the QA analysis script makes use of the extendedness value in the forced coadd results, I've added the following to config/forcedPhotCoadd.py:

            config.catalogCalculation.plugins.names=["base_ClassificationExtendedness"]
            

            Are you ok with these changes?

            Show
            lauren Lauren MacArthur added a comment - Paul Price : I've updated to keep applyUberCal=False as the codebase default, but am overriding it to True in obs_subaru by default for HSC coaddition. Also, because the QA analysis script makes use of the extendedness value in the forced coadd results, I've added the following to config/forcedPhotCoadd.py : config.catalogCalculation.plugins.names=["base_ClassificationExtendedness"] Are you ok with these changes?
            Hide
            price Paul Price added a comment -

            Yes, thanks!

            Show
            price Paul Price added a comment - Yes, thanks!
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks to everyone for all of the input on this one.

            Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks to everyone for all of the input on this one. Merged to master.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Paul Price
                Watchers:
                Bob Armstrong, Jim Bosch, John Swinbank, Lauren MacArthur, Paul Price, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel