Dan Taranu brought up a good point in a discussion the other day, which is that right now we are storing only the convolved scarlet models. The problem with this is that if anyone wants to make measurements on the deconvolved models they are SOL, since those are discarded. One motivation for storing the scarlet deconvolved models is for future processing. For example, in DR2 we will likely reach faster convergence using the DR1 models for sources (when possible) since they will already be deconvolved and the galaxy models shouldn't change much except in the low SNR regions. As far as I can tell, the only task that loads the deblender models in DRP is lsst.pipe.tasks.multiBand.MeasureMergedCoaddSourcesTask, so it looks to me like convolutions and re-distributing flux based on scarlet models would only done once, if at all.

So I want to propose the following:

• The deblender will output a single catalog (deblendedModel) that has an entry for each component. Each component record will have a 1D SED array column (it already exists) and a 2D morphology HeavyFootprint that is the deconvolved scarlet model for the component. There will also be an entry in the catalog for each source, where a source is a parent of one or more components. The sources will have a parent that is the parent of the entire blend, and each component will have a parent that is the source containing that component.
• A SourceCatalog.renderCatalog(Exposure exposure, bool convserveFlux) method will be created to convert the footprints in the deblendedModel catalog into a catalog for a given band, where exposure is the single band exposure to render the models into (needed for its PSF) and conserveFlux is a boolean value that determines whether or not flux should be re-distributed from the exposure using the models as templates. An isRendered property will be added to SourceCatalog to let the user know if the catalog has already been rendered and prevent multiple convolutions, and an isConserved parameter will let the user know if the catalog contains raw models or flux-conserved models.
• The deblendedModel catalog will be the input for lsst.pipe.tasks.multiBand.MeasureMergedCoaddSourcesTask, which will have config options to specify whether or not to convolveScarletModels or conserveFlux and will call deblenedModel.renderCatalog. Once the rendered catalog has been loaded into memory it can be passed to the measurement algorithms as necessary.

Thoughts?

Using previous-DR outputs to "seed" the current DR is potentially fraught with provenance and reproducibility problems, e.g. if a DAC offers only the latest DR. It also has the potential for locking in incorrect local minima that might otherwise be avoided with more data. So we should be very careful before using this as a performance enhancement.

I'm in favor of the new proposal, and those really feel like the only deblender output we should consider storing for delivery to science users (I think making deblender outputs a purely virtual dataset, i.e. something we always recompute from the image and catalog upon request, is also still in play).

Like Kian-Tat Lim, I am very skeptical of using those across DRs; note that we won't even have the same set of proto-objects detected, so matching would be problematic, even if provenance and local minima were not (and think they are a problem from the perspective of exact reproducibility, even if they aren't necessarily a problem from the perspective of scientific biases). But storing the source of truth is important in its own right, and may be relevant for downstream pipeline steps, like projecting deblending to short-period coadds for forced photometry. And I bet those deblended models will be (or can be made to be) much smaller than any of the PSF-convolved derived HeavyFootprints.

All that doesn't mean saving derived HeavyFootprints is out of the question - I think it's pretty likely that we'll want them as intermediates to avoid recomputing repeatedly in multiple downstream tasks. But I think we should try to think of that as an optimization in CPU/storage tradeoff space, so let's start without saving them and see how bad the CPU costs are.

We can talk more about processing across DRs another time. I'll just note that one good way to do source detection in future DRs might be to subtract all of the scarlet models from the previous DR and look for missing sources in the residuals in a type of difference imaging. Of course there are complications to this but it still might be worth looking into.

And yes Jim Bosch, the scarlet outputs are guaranteed to be smaller by the PSF radius in all directions, which is significant for faint and point-like sources (which is the majority of objects).

If there are no objections by Monday I'll adopt this RFC and begin to update DM-32436 accordingly to implement the single catalog deblender output with a renderCatalog method as outlined above in https://jira.lsstcorp.org/browse/RFC-825?focusedCommentId=500780&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-500780.