Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16268

Rewrite meas_extensions_ngmix for better functionality and upstream collaboration

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_extensions_ngmix
    • Labels:
      None

      Description

      The old meas_extensions_ngmix tried to wrap ngmix functionality within the constraints of DM's existing SingleFrameMeasurement framework, which immediately rules out a lot of what ngmix does well (fitting multiple bands or epochs together).  It also never really got off the ground.

      Blow away the SFM plugins that currently exist, and replace them with (to start) a CmdLineTask that processes all bands in a patch at once, and loops over optionally-deblended objects without trying to package that up in some kind of plugin framework.  This task will depend on all current pipe_tasks multiband outputs (or at least deepCoadd_ref).

      Once we've gained some experience with the actual APIs of measurements like this have, we can consider factoring that out into a new measurement framework in e.g. obs_base.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Yusra AlSayyad, John Swinbank: I'm planning to try to get this done early this week myself, as I think it's a good effort-multiplier to do it ASAP (Erin is eager to start working on this himself) and I think it'll be compatible with some other wait-for-tests-flavored work I have to do in Gen3 right now.

          I think it can be motivated well as any of Emergent, Galaxy-Modeling, or SST-side science collab support, and am happy to call it External/Science Time if any of this makes you nervous; please feel free to reclassify.

          Show
          jbosch Jim Bosch added a comment - Yusra AlSayyad , John Swinbank : I'm planning to try to get this done early this week myself, as I think it's a good effort-multiplier to do it ASAP (Erin is eager to start working on this himself) and I think it'll be compatible with some other wait-for-tests-flavored work I have to do in Gen3 right now. I think it can be motivated well as any of Emergent, Galaxy-Modeling, or SST-side science collab support, and am happy to call it External/Science Time if any of this makes you nervous; please feel free to reclassify.
          Hide
          jbosch Jim Bosch added a comment -

          John Swinbank, as per the (very brief) discussion on Slack last week, most of the code on this ticket has already landed on master of meas_extensions_ngmix, which has been moved from the "lsst" GitHub org to the "lsst-dm" one, reflecting the fact that I don't want DM procedures to get in Erin Sheldon's way in getting this usable for DESC.  I've also moved LSST's TAP ngmix repo to lsst-dm, but do not plan to merge the branch there for this ticket to master; Erin and I have agreed that it's better for anyone who wants to use meas_extensions_ngmix to install ngmix via pip themselves, and avoid having a DM packaging middleman inside what is for now mostly DESC work.

          What remains is a PR on obs_base, to add some dataset type definitions for the new CmdLineTasks in meas_extensions_ngmix.  I don't think we have much choice about where to put them, so I hope it's not a problem to introduce there some datasets used by an extension package.  Mind reviewing that?  The definitions themselves are of course trivial.

          (Kian-Tat Lim, you may be interested in what's going on here, too - and I hope all of the above meets your approval as well; you were on vacation when we originally discussed this plan.)

          Show
          jbosch Jim Bosch added a comment - John Swinbank , as per the (very brief) discussion on Slack last week, most of the code on this ticket has already landed on master of meas_extensions_ngmix, which has been moved from the "lsst" GitHub org to the "lsst-dm" one, reflecting the fact that I don't want DM procedures to get in Erin Sheldon's way in getting this usable for DESC.  I've also moved LSST's TAP ngmix repo to lsst-dm, but do not plan to merge the branch there for this ticket to master; Erin and I have agreed that it's better for anyone who wants to use meas_extensions_ngmix to install ngmix via pip themselves, and avoid having a DM packaging middleman inside what is for now mostly DESC work. What remains is a PR on obs_base , to add some dataset type definitions for the new CmdLineTasks in meas_extensions_ngmix.  I don't think we have much choice about where to put them, so I hope it's not a problem to introduce there some datasets used by an extension package.  Mind reviewing that?  The definitions themselves are of course trivial. ( Kian-Tat Lim , you may be interested in what's going on here, too - and I hope all of the above meets your approval as well; you were on vacation when we originally discussed this plan.)
          Hide
          swinbank John Swinbank added a comment -

          I think this is fine.

          I wondered a little about the dataset names. In part, this is due to my own ignorance about the algorithms. Specifically: how generic are the metacal dataset types? Would the same datasets be used for any metacalibration implementation, or are they ngmix-specific? If the latter, it seems like it might be helpful to include the string “ngmix” in the name.

          Show
          swinbank John Swinbank added a comment - I think this is fine. I wondered a little about the dataset names. In part, this is due to my own ignorance about the algorithms. Specifically: how generic are the metacal dataset types? Would the same datasets be used for any metacalibration implementation, or are they ngmix-specific? If the latter, it seems like it might be helpful to include the string “ngmix” in the name.
          Hide
          jbosch Jim Bosch added a comment -

          I mostly just think of the names here as temporaries; I'm hoping to refactor this code and probably include ngmix/metacal output columns as parts of other datasets well before we have a second metacalibration implementation (which may be never).

          Show
          jbosch Jim Bosch added a comment - I mostly just think of the names here as temporaries; I'm hoping to refactor this code and probably include ngmix/metacal output columns as parts of other datasets well before we have a second metacalibration implementation (which may be never).
          Hide
          jbosch Jim Bosch added a comment -

          obs_base merged to master.

          Show
          jbosch Jim Bosch added a comment - obs_base merged to master.

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            John Swinbank
            Watchers:
            Jim Bosch, John Swinbank, Robert Lupton, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.