[DM-31067] DC2 Reprocessing with w_2021_28 (gen2) - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
- SciencePipelines

Story Points:
8
Epic Link:
f21a-drp-qa
Sprint:
DRP S21b
Team:
Data Release Production
Urgent?:
No

Description

Process the DC2 dataset with the w_2021_28 stack. Will follow same procedure as in ~~DM-30730~~.

Attachments

Issue Links

relates to

DM-31351 DC2 Reprocessing with w_2021_32 (gen2)

Done

DM-31070 Compare the data products of the gen2 vs. gen3 w_2021_28 DC2 runs up to Single Frame Processing

Won't Fix

DM-31713 Investigate ArithmeticError in scarlet deblender

Won't Fix

DM-30730 DC2 Reprocessing with w_2021_24 (gen2)

Done

DM-30812 Compare the data products of the gen2 vs. gen3 w_2021_24 DC2 runs up to Single Frame Processing

Done

DM-31044 Gen3 DC2 reprocessing with bps and w_2021_28

Done

(1 relates to)

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Lauren MacArthur added a comment - 12/Jul/21 1:09 AM - edited

To overcome the GAaP-related failures noted on ~~DM-31044~~ and fixed on ~~DM-31062~~, I have kicked off this run having setup w_2021_28 + master(@26e77d7) of meas_extensions_gaap.

Show

Lauren MacArthur added a comment - 12/Jul/21 1:09 AM - edited To overcome the GAaP-related failures noted on DM-31044 and fixed on DM-31062 , I have kicked off this run having setup w_2021_28 + master(@26e77d7) of meas_extensions_gaap .

Hide

Permalink

Lauren MacArthur added a comment - 08/Sep/21 8:52 PM

I just noticed the following in the mutibandDriver log for patch 0,5 from this run:

/datasets/DC2/repoRun2.2i/rerun/w_2021_32/DM-31351/logs/04_multiBandDriver/mt3829-05.o54859:

Traceback (most recent call last):

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_scarlet/21.0\

.0-14-g574b836+2eb1d5918e/python/lsst/meas/extensions/scarlet/scarletDeblendTask.py", line 755, in deblend

    blend, skipped, spectrumInit = deblend(mExposure, foot, self.config)

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_scarlet/21.0\

.0-14-g574b836+2eb1d5918e/python/lsst/meas/extensions/scarlet/scarletDeblendTask.py", line 300, in deblend

    blend = Blend(sources, observation)

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\

+f31336177f/lib/python/scarlet/blend.py", line 79, in __init__

    super().__init__(self.sources)

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\

+f31336177f/lib/python/scarlet/component.py", line 237, in __init__

    super().__init__(frame, children=components, bbox=box)

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\

+f31336177f/lib/python/scarlet/component.py", line 38, in __init__

    super().__init__(*parameters, children=children)

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\

+f31336177f/lib/python/scarlet/model.py", line 47, in __init__

    self.check_parameters()

  File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/scarlet/lsst-dev-g965bb5fbbf\

+f31336177f/lib/python/scarlet/model.py", line 122, in check_parameters

    raise ArithmeticError(msg)

ArithmeticError: Model Blend, Parameter 'spectrum' is not finite:

[3.3891191e+04 3.0141157e-01 9.9999997e-21           nan 9.9999997e-21

 9.9999997e-21]

I also see a similar error in the Gen3 run (~~DM-31044~~) in this log file for patch 35:

/scratch/brendal4/bps-gen3-dc2/submit/2.2i/runs/test-med-1/w_2021_28/DM-31044/20210716T012652Z/jobs/deblend/1362_deblend_3829_35_.3117668.err

As far as I can tell, the earliest occurrence dates back to w_2021_24. It doesn't appear in the w_2021_16 and prior logs (but log persistence has been inconsistent...) I also don't see any occurrences in the HSC RC2 processing, so it does seem this is an edge case. Any thoughts Fred Moolekamp?

Show

Lauren MacArthur added a comment - 08/Sep/21 8:52 PM I just noticed the following in the mutibandDriver log for patch 0,5 from this run: / datasets / DC2 / repoRun2. 2i / rerun / w_2021_32 / DM - 31351 / logs / 04_multiBandDriver / mt3829 - 05.o54859 : Traceback (most recent call last): File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / meas_extensions_scarlet / 21.0 \ . 0 - 14 - g574b836 + 2eb1d5918e / python / lsst / meas / extensions / scarlet / scarletDeblendTask.py", line 755 , in deblend blend, skipped, spectrumInit = deblend(mExposure, foot, self .config) File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / meas_extensions_scarlet / 21.0 \ . 0 - 14 - g574b836 + 2eb1d5918e / python / lsst / meas / extensions / scarlet / scarletDeblendTask.py", line 300 , in deblend blend = Blend(sources, observation) File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / scarlet / lsst - dev - g965bb5fbbf\ + f31336177f / lib / python / scarlet / blend.py", line 79 , in __init__ super ().__init__( self .sources) File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / scarlet / lsst - dev - g965bb5fbbf\ + f31336177f / lib / python / scarlet / component.py", line 237 , in __init__ super ().__init__(frame, children = components, bbox = box) File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / scarlet / lsst - dev - g965bb5fbbf\ + f31336177f / lib / python / scarlet / component.py", line 38 , in __init__ super ().__init__( * parameters, children = children) File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / scarlet / lsst - dev - g965bb5fbbf\ + f31336177f / lib / python / scarlet / model.py", line 47 , in __init__ self .check_parameters() File " / software / lsstsw / stack_20210520 / stack / miniconda3 - py38_4. 9.2 - 0.6 . 0 / Linux64 / scarlet / lsst - dev - g965bb5fbbf\ + f31336177f / lib / python / scarlet / model.py", line 122 , in check_parameters raise ArithmeticError(msg) ArithmeticError: Model Blend, Parameter 'spectrum' is not finite: [ 3.3891191e + 04 3.0141157e - 01 9.9999997e - 21 nan 9.9999997e - 21 9.9999997e - 21 ] I also see a similar error in the Gen3 run ( DM-31044 ) in this log file for patch 35: / scratch / brendal4 / bps - gen3 - dc2 / submit / 2.2i / runs / test - med - 1 / w_2021_28 / DM - 31044 / 20210716T012652Z / jobs / deblend / 1362_deblend_3829_35_ . 3117668.err As far as I can tell, the earliest occurrence dates back to w_2021_24 . It doesn't appear in the w_2021_16 and prior logs (but log persistence has been inconsistent...) I also don't see any occurrences in the HSC RC2 processing, so it does seem this is an edge case. Any thoughts Fred Moolekamp ?

Hide

Permalink

Fred Moolekamp added a comment - 09/Sep/21 4:31 PM

That must be an edge case that we haven't seen yet, especially since it ran all of HSC RC2 without any instances. Lauren MacArthur would you mind opening a ticket so that I don't forget to look into this when I get back? Even if it is an edge case, that should (hopefully) make it easy to diagnose and fix.

Show

Fred Moolekamp added a comment - 09/Sep/21 4:31 PM That must be an edge case that we haven't seen yet, especially since it ran all of HSC RC2 without any instances. Lauren MacArthur would you mind opening a ticket so that I don't forget to look into this when I get back? Even if it is an edge case, that should (hopefully) make it easy to diagnose and fix.

Hide

Permalink

Lauren MacArthur added a comment - 10/Sep/21 2:53 AM

Ticket created for scarlet ArithmeticError: ~~DM-31713~~.

Show

Lauren MacArthur added a comment - 10/Sep/21 2:53 AM Ticket created for scarlet ArithmeticError: DM-31713 .

Hide

Permalink

Lauren MacArthur added a comment - 14/Sep/21 8:26 PM - edited

All plots and logs are linked here. They can also be found at /datasets/DC2/repoRun2.2i/rerun/w_2021_28/DM-31067/ (along with all the processing run output). Metrics were dispatched and ingested. They can be seen on the dashboard titled "DRP metrics monthly for DC2 (Gen2)" at https://chronograf-demo.lsst.codes/sources/2/dashboards.

Show

Lauren MacArthur added a comment - 14/Sep/21 8:26 PM - edited All plots and logs are linked here . They can also be found at /datasets/DC2/repoRun2.2i/rerun/w_2021_28/ DM-31067 / (along with all the processing run output). Metrics were dispatched and ingested. They can be seen on the dashboard titled "DRP metrics monthly for DC2 (Gen2)" at https://chronograf-demo.lsst.codes/sources/2/dashboards .

Hide

Permalink

Lauren MacArthur added a comment - 14/Sep/21 8:26 PM

When you've got a chance, can you confirm whether all is complete for this run?

Show

Lauren MacArthur added a comment - 14/Sep/21 8:26 PM When you've got a chance, can you confirm whether all is complete for this run?

Hide

Permalink

Yusra AlSayyad added a comment - 16/Sep/21 2:33 PM

needs story points

Show

Yusra AlSayyad added a comment - 16/Sep/21 2:33 PM needs story points

People

Assignee:: Lauren MacArthur

Reporter:: Lauren MacArthur

Reviewers:: Yusra AlSayyad

Watchers:: Brock Brendal [X] (Inactive), Fred Moolekamp, Jim Bosch, Lauren MacArthur, Monika Adamow, Yusra AlSayyad

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 09/Jul/21 8:05 PM

Updated:: 16/Sep/21 6:28 PM

Resolved:: 16/Sep/21 6:28 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins