Fix Version/s: None
The old meas_extensions_ngmix tried to wrap ngmix functionality within the constraints of DM's existing SingleFrameMeasurement framework, which immediately rules out a lot of what ngmix does well (fitting multiple bands or epochs together). It also never really got off the ground.
Blow away the SFM plugins that currently exist, and replace them with (to start) a CmdLineTask that processes all bands in a patch at once, and loops over optionally-deblended objects without trying to package that up in some kind of plugin framework. This task will depend on all current pipe_tasks multiband outputs (or at least deepCoadd_ref).
Once we've gained some experience with the actual APIs of measurements like this have, we can consider factoring that out into a new measurement framework in e.g. obs_base.
John Swinbank, as per the (very brief) discussion on Slack last week, most of the code on this ticket has already landed on master of meas_extensions_ngmix, which has been moved from the "lsst" GitHub org to the "lsst-dm" one, reflecting the fact that I don't want DM procedures to get in Erin Sheldon's way in getting this usable for DESC. I've also moved LSST's TAP ngmix repo to lsst-dm, but do not plan to merge the branch there for this ticket to master; Erin and I have agreed that it's better for anyone who wants to use meas_extensions_ngmix to install ngmix via pip themselves, and avoid having a DM packaging middleman inside what is for now mostly DESC work.
What remains is a PR on obs_base, to add some dataset type definitions for the new CmdLineTasks in meas_extensions_ngmix. I don't think we have much choice about where to put them, so I hope it's not a problem to introduce there some datasets used by an extension package. Mind reviewing that? The definitions themselves are of course trivial.
(Kian-Tat Lim, you may be interested in what's going on here, too - and I hope all of the above meets your approval as well; you were on vacation when we originally discussed this plan.)
I think this is fine.
I wondered a little about the dataset names. In part, this is due to my own ignorance about the algorithms. Specifically: how generic are the metacal dataset types? Would the same datasets be used for any metacalibration implementation, or are they ngmix-specific? If the latter, it seems like it might be helpful to include the string “ngmix” in the name.
I mostly just think of the names here as temporaries; I'm hoping to refactor this code and probably include ngmix/metacal output columns as parts of other datasets well before we have a second metacalibration implementation (which may be never).
Yusra AlSayyad, John Swinbank: I'm planning to try to get this done early this week myself, as I think it's a good effort-multiplier to do it ASAP (Erin is eager to start working on this himself) and I think it'll be compatible with some other wait-for-tests-flavored work I have to do in Gen3 right now.
I think it can be motivated well as any of Emergent, Galaxy-Modeling, or SST-side science collab support, and am happy to call it External/Science Time if any of this makes you nervous; please feel free to reclassify.