Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32436

Implement scarlet lite in meas_extensions_scarlet

    XMLWordPrintable

Details

    • 22
    • DRP S21b, DRP S22A
    • Data Release Production
    • No

    Description

      This ticket is to implement the lite version of scarlet in meas_extensions_scarlet and test it on the RC2 and DC2 datasets.

      Attachments

        1. all blends.png
          all blends.png
          112 kB
        2. isolated.png
          isolated.png
          112 kB
        3. large blends.png
          large blends.png
          107 kB

        Issue Links

          Activity

            No builds found.
            fred3m Fred Moolekamp created issue -
            fred3m Fred Moolekamp made changes -
            Field Original Value New Value
            Status To Do [ 10001 ] In Progress [ 3 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30469 [ 509194 ] DM-30541 [ 511199 ]
            fred3m Fred Moolekamp made changes -
            Story Points 10 12
            fred3m Fred Moolekamp made changes -
            Link This issue is blocked by DM-30756 [ DM-30756 ]
            fred3m Fred Moolekamp made changes -
            Attachment Screen Shot 2021-11-22 at 2.42.20 PM.png [ 54966 ]
            fred3m Fred Moolekamp made changes -
            Attachment Screen Shot 2021-11-22 at 2.42.20 PM.png [ 54966 ]
            fred3m Fred Moolekamp made changes -
            Attachment isolated.png [ 54967 ]
            fred3m Fred Moolekamp made changes -
            Attachment all blends.png [ 54968 ]
            fred3m Fred Moolekamp made changes -
            Attachment large blends.png [ 54969 ]

            I tested the scarlet lite implementation on a random HSC patch (tract 9697, patch 40) to fix bugs in both the scarlet lite codebase and implementation in the stack. The runtime plots below show the runtime per blend for isolated sources, all blended sources, and all blends with more than 5 sources to show how the improvements in speed are highly non-linear and increase significantly for larger blends. The logL plots show the log likelihood histograms for each blends in all three datasets, illustrating that while the current scarlet and the lite branch and the scarlet lite with ADAM achieve slightly better results, the overall fits are not that different for the much faster FISTA implementations. On DC2 data it is expected that the results using redistributed flux will show even smaller differences from the truth between FISTA and ADAM, making it likely that we can use the faster (and slightly less memory intensive) FISTA algorithm.

            In all of the plots there are 4 different runs:

            • ADAM: scarlet lite with the proximal ADAM optimizer and chi^2 coadd for initialization
            • FISTA: scarlet lite with the FISTA proximal gradient method optimizer and wavelet initialization
            • short: The same as FISTA but using 10^-2 relative error for convergence instead of the currrent 10^-4
            • main: The current main branch using full scarlet

            The reason ADAM and FISTA have different initialization is that testing in DM-30756 showed that proximal ADAM converged faster, to a better solution, using chi^2 initialization while FISTA was faster and more accurate with wavelet initialization.

            Isolated sources
            total runtimes:

            • adam: 1207.5s
            • fista: 898.7s
            • short: 853.2s
            • hsc: 1448.1s

            failures:

            • adam: 0
            • fista: 0
            • hsc: 0

            All blends
            total runtimes:

            • adam: 1506.7s
            • fista: 826.7s
            • short: 796.8s
            • hsc: 3037.2s

            failures:

            • adam: 1
            • fista: 0
            • hsc: 0

            Blends with more than 5 sources

            total runtimes:

            • adam: 752.7s
            • fista: 313.9s
            • short: 289.5s
            • hsc: 2183.6s

            failures:

            • adam: 1
            • fista: 0
            • hsc: 0

            fred3m Fred Moolekamp added a comment - I tested the scarlet lite implementation on a random HSC patch (tract 9697, patch 40) to fix bugs in both the scarlet lite codebase and implementation in the stack. The runtime plots below show the runtime per blend for isolated sources, all blended sources, and all blends with more than 5 sources to show how the improvements in speed are highly non-linear and increase significantly for larger blends. The logL plots show the log likelihood histograms for each blends in all three datasets, illustrating that while the current scarlet and the lite branch and the scarlet lite with ADAM achieve slightly better results, the overall fits are not that different for the much faster FISTA implementations. On DC2 data it is expected that the results using redistributed flux will show even smaller differences from the truth between FISTA and ADAM, making it likely that we can use the faster (and slightly less memory intensive) FISTA algorithm. In all of the plots there are 4 different runs: ADAM: scarlet lite with the proximal ADAM optimizer and chi^2 coadd for initialization FISTA: scarlet lite with the FISTA proximal gradient method optimizer and wavelet initialization short: The same as FISTA but using 10^-2 relative error for convergence instead of the currrent 10^-4 main: The current main branch using full scarlet The reason ADAM and FISTA have different initialization is that testing in DM-30756 showed that proximal ADAM converged faster, to a better solution, using chi^2 initialization while FISTA was faster and more accurate with wavelet initialization. Isolated sources total runtimes: adam: 1207.5s fista: 898.7s short: 853.2s hsc: 1448.1s failures: adam: 0 fista: 0 hsc: 0 All blends total runtimes: adam: 1506.7s fista: 826.7s short: 796.8s hsc: 3037.2s failures: adam: 1 fista: 0 hsc: 0 Blends with more than 5 sources total runtimes: adam: 752.7s fista: 313.9s short: 289.5s hsc: 2183.6s failures: adam: 1 fista: 0 hsc: 0
            fred3m Fred Moolekamp made changes -
            Link This issue blocks DM-32760 [ DM-32760 ]
            fred3m Fred Moolekamp made changes -
            Story Points 12 20
            yusra Yusra AlSayyad made changes -
            Sprint DRP S21b [ 1094 ] DRP S21b, DRP S22A [ 1094, 1137 ]
            fred3m Fred Moolekamp added a comment - - edited

            Extra bonus. In ci_hsc (gen2) there is a test that a minimal number of PSF stars are correctly identified. This used to be set at 90%, but in DM-28294 it was identified that this test was broken in scarlet because of a bad PSF model, so the ratio was lowered to 70% and has been useful identifying catastrophic bugs in scarlet updates. Implementing scarlet lite with re-weighting improves this number to just over 90%, so if we wish we can raise the threshold to 90% (or perhaps 89%) again.

            fred3m Fred Moolekamp added a comment - - edited Extra bonus. In ci_hsc (gen2) there is a test that a minimal number of PSF stars are correctly identified. This used to be set at 90%, but in DM-28294 it was identified that this test was broken in scarlet because of a bad PSF model, so the ratio was lowered to 70% and has been useful identifying catastrophic bugs in scarlet updates. Implementing scarlet lite with re-weighting improves this number to just over 90%, so if we wish we can raise the threshold to 90% (or perhaps 89%) again.
            fred3m Fred Moolekamp made changes -
            Link This issue is blocked by RFC-823 [ RFC-823 ]
            fred3m Fred Moolekamp made changes -
            Story Points 20 22
            fred3m Fred Moolekamp added a comment - Successful Jenkins build: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35761/pipeline/

            Hi Dan,
            Would you mind reviewing this scarlet lite implementation ticket?

            Thanks,
            -Fred

            fred3m Fred Moolekamp added a comment - Hi Dan, Would you mind reviewing this scarlet lite implementation ticket? Thanks, -Fred
            fred3m Fred Moolekamp made changes -
            Reviewers Dan Taranu [ dtaranu ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            dtaranu Dan Taranu added a comment -

            Can you comment more on the bad PSF model issue and what re-weighting/flux-weighted catalogs means, exactly? Or put another way, where would a potential user find docs on what's in a deepCoadd_deblendedFlux catalog and whether it's suited to their needs?

            dtaranu Dan Taranu added a comment - Can you comment more on the bad PSF model issue and what re-weighting/flux-weighted catalogs means, exactly? Or put another way, where would a potential user find docs on what's in a deepCoadd_deblendedFlux catalog and whether it's suited to their needs?

            Sure. There is an issue with the I-band PSF in ci_hsc where masked pixels make it into the PSF model. Since the scarlet models are convolved with the PSF to fit the data, this causes all of the models to have strange artifacts. One of the ci_hsc tests is to make sure that stars are being modeled correctly , which of course will be off with a bad PSF model, so we had to lower the threshold for the number of stars that need to pass the cut.

            But when the flux is redistributed the problem mostly goes away because the input data is distributed to all of the sources according to the ratio of the models in each pixel. Since most sources have the majority of their footprints unblended, redistributing the image flux recovers the majority of the signal for each source correctly, so now nearly all of the stars are passing the test again.

            fred3m Fred Moolekamp added a comment - Sure. There is an issue with the I-band PSF in ci_hsc where masked pixels make it into the PSF model. Since the scarlet models are convolved with the PSF to fit the data, this causes all of the models to have strange artifacts. One of the ci_hsc tests is to make sure that stars are being modeled correctly , which of course will be off with a bad PSF model, so we had to lower the threshold for the number of stars that need to pass the cut. But when the flux is redistributed the problem mostly goes away because the input data is distributed to all of the sources according to the ratio of the models in each pixel. Since most sources have the majority of their footprints unblended, redistributing the image flux recovers the majority of the signal for each source correctly, so now nearly all of the stars are passing the test again.
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30541 [ 511199 ] DM-30548 [ 511214 ]
            fred3m Fred Moolekamp made changes -
            Link This issue is triggering RFC-825 [ RFC-825 ]
            dtaranu Dan Taranu added a comment -

            Ok, so the PSF issue is DM-12058 and is not fixed. I left some comments about (avoiding) potential duplicate footprint storage in RFC-825.

            dtaranu Dan Taranu added a comment - Ok, so the PSF issue is DM-12058 and is not fixed. I left some comments about (avoiding) potential duplicate footprint storage in RFC-825 .
            fred3m Fred Moolekamp added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35864/pipeline/
            dtaranu Dan Taranu added a comment - - edited

            The meas_extensions_scarlet PR looks fine now. Regardless of the outcome of RFC-825, I'd like documentation or an easy-to-find note somewhere explaining which kind of Scarlet model you can expect to find in which datasetType (and if it's config-dependent, which config parameters change the behaviour).

            dtaranu Dan Taranu added a comment - - edited The meas_extensions_scarlet PR looks fine now. Regardless of the outcome of RFC-825 , I'd like documentation or an easy-to-find note somewhere explaining which kind of Scarlet model you can expect to find in which datasetType (and if it's config-dependent, which config parameters change the behaviour).
            dtaranu Dan Taranu made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            dtaranu Dan Taranu made changes -
            Status Reviewed [ 10101 ] In Review [ 10004 ]
            dtaranu Dan Taranu added a comment - - edited

            The pipe_tasks PR doesn't look like it was rebased properly, or at least is out of date with main; please re-rebase it (e: looks like this was a main/master mix-up).

            dtaranu Dan Taranu added a comment - - edited The pipe_tasks PR doesn't look like it was rebased properly, or at least is out of date with main; please re-rebase it (e: looks like this was a main/master mix-up).
            dtaranu Dan Taranu added a comment -

            Looks good, though I'd suggest re-running Jenkins just to be sure about pipe_tasks.

            dtaranu Dan Taranu added a comment - Looks good, though I'd suggest re-running Jenkins just to be sure about pipe_tasks .
            dtaranu Dan Taranu made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            fred3m Fred Moolekamp added a comment - Successful Jenkins build: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35879/pipeline/47
            fred3m Fred Moolekamp made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-41986 [ DM-41986 ]

            People

              fred3m Fred Moolekamp
              fred3m Fred Moolekamp
              Dan Taranu
              Dan Taranu, Fred Moolekamp
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.