Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31713

Investigate ArithmeticError in scarlet deblender

    XMLWordPrintable

Details

    • Data Release Production
    • No

    Description

      As noted in DM-31067, a thus far unidentified/acknowledged error has been observed in the DC2 reprocessing runs.  The traceback for a gen2 run log is:

      /datasets/DC2/repoRun2.2i/rerun/w_2021_32/DM-31351/logs/04_multiBandDriver/mt3829-05.o54859:
       
      Traceback (most recent call last):
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_scarlet/21.0.0-14-g574b836+2eb1d5918e/python/lsst/meas/extensions/scarlet/scarletDeblendTask.py", line 755, in deblend
       blend, skipped, spectrumInit = deblend(mExposure, foot, self.config)
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_scarlet/21.0.0-14-g574b836+2eb1d5918e/python/lsst/meas/extensions/scarlet/scarletDeblendTask.py", line 300, in deblend
       blend = Blend(sources, observation)
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf+f31336177f/lib/python/scarlet/blend.py", line 79, in __init__
       super().__init__(self.sources)
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf+f31336177f/lib/python/scarlet/component.py", line 237, in __init__
       super().__init__(frame, children=components, bbox=box)
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\
      +f31336177f/lib/python/scarlet/component.py", line 38, in __init__
       super().__init__(*parameters, children=children)
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf+f31336177f/lib/python/scarlet/model.py", line 47, in __init__
       self.check_parameters()
       File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf+f31336177f/lib/python/scarlet/model.py", line 122, in check_parameters
       raise ArithmeticError(msg)
      ArithmeticError: Model Blend, Parameter 'spectrum' is not finite:
      [3.3891191e+04 3.0141157e-01 9.9999997e-21 nan 9.9999997e-21
       9.9999997e-21]
      

      And a similar error is seen in the Gen3 run (DM-31044) in this log file for patch 35:

      /scratch/brendal4/bps-gen3-dc2/submit/2.2i/runs/test-med-1/w_2021_28/DM-31044/20210716T012652Z/jobs/deblend/1362_deblend_3829_35_.3117668.err
      

      The earliest occurrence appears to date back to w_2021_24. It doesn't appear in the w_2021_16 and prior logs (but log persistence has been inconsistent...). This error has not been seen in the HSC RC2 processing, so it does seem this is an edge case, but is still worth investigating.

      Attachments

        Issue Links

          Activity

            This is no longer an issue now that scarlet_lite is set to replace scarlet.

            fred3m Fred Moolekamp added a comment - This is no longer an issue now that scarlet_lite is set to replace scarlet.

            People

              fred3m Fred Moolekamp
              lauren Lauren MacArthur
              Fred Moolekamp, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.