Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
On Dec. 7 there was a meeting announced in the #scarlet-hsc-test slack channel to review the QA plots from the HSC RC2 data, comparing the results from meas_deblender and meas_extensions_scarlet. While there is still work to be done on scarlet (as evidenced by the issues in DM-27972 and ongoing development by Peter Melchior's group at Princeton) the overall consensus seemed to be that we are ready to move forward with making scarlet the default deblender in the stack (which essentially involves changing `config.deblendCoaddSources.simultaneous=True` in multiBandDriver).
This was based on the fact that the HSC RC2 QA plots (available at https://lsst.ncsa.illinois.edu/~fred3m/qaRC2/scarletRC2_3/ which can be compared with the current run using meas_deblender at https://lsst.ncsa.illinois.edu/~emorgan2/w_2020_42_qaplots/) showed no signs of large unexplained errors in the scarlet runs. Notes from the meeting are located at https://docs.google.com/document/d/1eJfUdbFVG05AC6IXq5N0-C_2nZTIhKSkBVr9WzrEfIE/edit , which identify some of the plots are noticeably different.
Recent work by Dan Taranu (eg. DM-27590) has tested multiband driver using scarlet on DC2 data and has found that the performance of the current version of scarlet is mostly comparable to the current deblender with fewer outliers, especially on the faint end. There was a bug found in scarlet during Dan's testing, but the cause is specific to DC2 data only, was identified, and is currently being removed in DM-28017.
The only potential holdup is that the gen3 middleware does not yet work with a multiband deblender, which is an issue that Nate Lust is looking into.
Attachments
Issue Links
Activity
These are good points to bring up and we looked into both of these in DM-25744. The newer version of scarlet doesn't appear to use significantly more memory than meas_deblender, and is not the bottleneck, where something in the measurement portion of the multiband pipeline uses more memory (the run in DM-275744 did show a slight ~10% increase in memory usage that was mostly due to that version of scarlet using footprints for sources that were too big, and has mostly been corrected).
It is true that scarlet is ~3 times slower than the current deblender when run using the default settings, with two caveats:
1. meas_deblender conserves flux, meaning the templates calculated by the deblender are used to re-apportion the flux from the parent footprint so that all of the flux in the parent footprint are accounted for. For this reason there is no need to run meas_deblender on isolated sources and it simply skips them. Since scarlet builds a model for each source the default is to run scarlet on isolated footprints in order to have consistent models for blended and isolated sources. So scarlet is running on more "blends," which results in 30-40% more sources and adds to the overall runtime.
2. By default multi-threading is turned off in the science pipelines, which means the (single band) meas_deblender has access to 5 times as many cores, since the (multiband) scarlet is processed on a single core for 5 bands. So, for example, I typically run multiband driver with 15 processes per node, so that three multiband datasets can be processed on one node, running measurements on all 15 catalogs simultaneously. But that means that 80% of the cores are idle during deblending. When we do get scarlet ported to the gen3 middleware we hope to gain the ability to use multiple threads in the deblender only.
From a practical standpoint, this means that running the full multiband driver, with the current version of meas_extensions_scarlet, you shouldn't see a significant increase in memory usage and will see a ~20% increase in runtime if you have multi-threading turned off and run on the isolated sources.
Fred Moolekamp
added a comment - - edited These are good points to bring up and we looked into both of these in DM-25744 . The newer version of scarlet doesn't appear to use significantly more memory than meas_deblender, and is not the bottleneck, where something in the measurement portion of the multiband pipeline uses more memory (the run in DM-275744 did show a slight ~10% increase in memory usage that was mostly due to that version of scarlet using footprints for sources that were too big, and has mostly been corrected).
It is true that scarlet is ~3 times slower than the current deblender when run using the default settings, with two caveats:
1. meas_deblender conserves flux, meaning the templates calculated by the deblender are used to re-apportion the flux from the parent footprint so that all of the flux in the parent footprint are accounted for. For this reason there is no need to run meas_deblender on isolated sources and it simply skips them. Since scarlet builds a model for each source the default is to run scarlet on isolated footprints in order to have consistent models for blended and isolated sources. So scarlet is running on more "blends," which results in 30-40% more sources and adds to the overall runtime.
2. By default multi-threading is turned off in the science pipelines, which means the (single band) meas_deblender has access to 5 times as many cores, since the (multiband) scarlet is processed on a single core for 5 bands. So, for example, I typically run multiband driver with 15 processes per node, so that three multiband datasets can be processed on one node, running measurements on all 15 catalogs simultaneously. But that means that 80% of the cores are idle during deblending. When we do get scarlet ported to the gen3 middleware we hope to gain the ability to use multiple threads in the deblender only.
From a practical standpoint, this means that running the full multiband driver, with the current version of meas_extensions_scarlet, you shouldn't see a significant increase in memory usage and will see a ~20% increase in runtime if you have multi-threading turned off and run on the isolated sources. John Parejko did the previous post answer your potential objection?
I'm not seeing any other objections, so as long as there are no other objections raised by this evening the question becomes when do we want to flip the switch? Do we do it tonight so that it is in the next weekly? Should we wait for the gen3 middleware to support scarlet and change it there too (which will potentially get finished this week in DM-27722)? Should we wait for next months RC2 processing? The last time I spoke with Jim Bosch he thought that Yusra AlSayyad might have a preference in terms of implementing scarlet when there aren't too many other features being merged simultaneously, so it would be nice to get some opinions on when we should move forward.
Fred Moolekamp
added a comment - John Parejko did the previous post answer your potential objection?
I'm not seeing any other objections, so as long as there are no other objections raised by this evening the question becomes when do we want to flip the switch? Do we do it tonight so that it is in the next weekly? Should we wait for the gen3 middleware to support scarlet and change it there too (which will potentially get finished this week in DM-27722 )? Should we wait for next months RC2 processing? The last time I spoke with Jim Bosch he thought that Yusra AlSayyad might have a preference in terms of implementing scarlet when there aren't too many other features being merged simultaneously, so it would be nice to get some opinions on when we should move forward. Should have commented earlier, but this gets a big 
from me; we've been developing scarlet as the deblender of the future for a long time now, and this RFC is out precisely because the DRP people involved believe it's now enough better (in both current scientific quality and potential for future improvements) and similar enough in computational cost that it's time to flip the switch.
I support this as a change to the obs_subaru and obs_lsst packages, so that we can test it at scale in DRP's regular monthly RC2 and DC2 reprocessings. (i.e. this RFC isn't a proposal for what to use for LSST DR1)
Yes, I would like to wait until we get the Gaia reference catalog errors cleared up and the metrics back to baseline so that we can get a good comparison of the effect of Scarlet. But given that w02 gets tagged tonight, in practice your earliest rerun would be w08 regardless. Let's talk about this today at team meeting.
There have been no objections, and we discussed proceeding with this at Monday's metrics meeting. DM-28323 has been created to implement this RFC.
Fred Moolekamp
added a comment - There have been no objections, and we discussed proceeding with this at Monday's metrics meeting. DM-28323 has been created to implement this RFC. 
My only concern would be whether this causes a significant increase in system resource use during normal processing. I remember some discussion of scarlet's memory/CPU footprint, but I don't remember how that compares with the existing deblender.