Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4887

Refactor measurement afterburners into a new plugin system

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_base
    • Labels:
      None

      Description

      Some of the operations we currently run as part of measurement (or would like to) share some features that make them a bit different from most plugin algorithms:

      • They must be run after at least some other high-level plugins, and may be run after all of them.
      • They do not require access to pixel data, as they derive their outputs entirely from other plugins' catalog outputs.
      • They may require an aggregation stage of some sort to be run on the regular plugin output before they can be run.

      Some examples include:

      • Star/Galaxy classification (with training done after measurement and before classification).
      • Applying aperture corrections (estimating the correction must be done first).
      • BFD's P, Q, R statistics (requires a prior estimated from deep data).

      We should move these algorithms to a new plugin system that's run by a new subtask, allowing these plugins to be run entirely separately from SingleFrameMeasurementTask. This will simplify some of the currently contorted logic required to make S/G classification happen after aperture correction, while making room for hierarchical inference algorithms like BFD and Bayesian S/G classification in the future.

      (We will not be able to support BFD immediately, as this will also require changes to our parallelization approach, but this will be a step in the right direction).

      This work should probably be delayed until after the HSC merge and Russell Owen's rewrite of ProcessCcdTask are complete, but it's conceivable that this refactoring could solve emergent problems there and be worth doing earlier as a result.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Looking at what we have in place, the barrier to do this is much lower than I thought, and the short-term importance much higher. I'm lowering my SP estimate and will submit an RFC shortly.

            Show
            jbosch Jim Bosch added a comment - Looking at what we have in place, the barrier to do this is much lower than I thought, and the short-term importance much higher. I'm lowering my SP estimate and will submit an RFC shortly.
            Hide
            price Paul Price added a comment -

            Please forgive the delay. I've finally completed the review on github. Nothing serious, but I've suggested refactoring in a few places.

            Show
            price Paul Price added a comment - Please forgive the delay. I've finally completed the review on github . Nothing serious, but I've suggested refactoring in a few places.
            Hide
            swinbank John Swinbank added a comment - - edited

            Nate Lust – looks like it's release notes time!

            Show
            swinbank John Swinbank added a comment - - edited Nate Lust – looks like it's release notes time!
            Hide
            swinbank John Swinbank added a comment -

            Hey Nate Lust – can you give me a status update on the release notes, please?

            Show
            swinbank John Swinbank added a comment - Hey Nate Lust – can you give me a status update on the release notes, please?
            Hide
            nlust Nate Lust added a comment -

            Added a small blurb to the confluence page

            Show
            nlust Nate Lust added a comment - Added a small blurb to the confluence page
            Hide
            swinbank John Swinbank added a comment -

            Thanks Nate!

            Show
            swinbank John Swinbank added a comment - Thanks Nate!

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Paul Price
              Watchers:
              Bob Armstrong, Jim Bosch, John Swinbank, Lauren MacArthur, Nate Lust, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.