[DM-32391] Gen3 DC2 reprocessing with bps and w_2021_44 - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
None

Epic Link:
Commissioning: Reprocessing data in Gen2 or Gen3
Team:
Data Facility
Urgent?:
No

Description

Submit path: /scratch/brendal4/bps-gen3-dc2/w_2021_44/submit/2.2i/runs/test-med-1/w_2021_44/~~DM-32391~~/20211028T181741Z

Attachments

Issue Links

relates to

DM-31777 Determine and use correct position for PSF shape determination in ImageDifferenceTask

Done

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(14 mentioned in)

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Brock Brendal [X] (Inactive) added a comment - 29/Oct/21 7:57 PM - edited

Step 1 completed with 3301 errors in characerizeImage and 18 in calibrate, all of which were DB connection errors which, as always, should be resolved upon resubmission. 2 errors were in isr which looked like:

174534/47250_isr_174534_47.6427736.err:lsst.pex.exceptions.wrappers.FitsError:

174534/47250_isr_174534_47.6427736.err:lsst::afw::fits::FitsError: 'cfitsio error: couldn't create the named file (105) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits' with mode 'w'

174534/47250_isr_174534_47.6427736.err:    raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} "

174534/47250_isr_174534_47.6427736.err:RuntimeError: Failed to serialize dataset postISRCCD@{instrument: 'LSSTCam-imSim', detector: 47, exposure: 174534, ...}, sc=Exposure] (id=23f8f2ff-bd96-4298-b154-62a5e1c44583) of type <class 'lsst.afw.image.exposure.ExposureF'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits

174534/47250_isr_174534_47.6427736.err:lsst.pex.exceptions.wrappers.FitsError:

174534/47250_isr_174534_47.6427736.err:lsst::afw::fits::FitsError: 'cfitsio error: couldn't create the named file (105) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits' with mode 'w'

174534/47250_isr_174534_47.6427736.err:    raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} "

174534/47250_isr_174534_47.6427736.err:RuntimeError: Failed to serialize dataset postISRCCD@{instrument: 'LSSTCam-imSim', detector: 47, exposure: 174534, ...}, sc=Exposure] (id=23f8f2ff-bd96-4298-b154-62a5e1c44583) of type <class 'lsst.afw.image.exposure.ExposureF'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits

My hope is that these 2 errors will also be resolved upon resubmission.

There was also 1 error in calibrate which was seen in the last DC2 weekly, w40:

265317/174188_calibrate_265317_8.6517954.err:    raise pipeBase.TaskError(

265317/174188_calibrate_265317_8.6517954.err:lsst.pipe.base.task.TaskError: Fit failed: median scatter on sky = 13.437 arcsec > 10.000 config.maxScatterArcsec

Show

Brock Brendal [X] (Inactive) added a comment - 29/Oct/21 7:57 PM - edited Step 1 completed with 3301 errors in characerizeImage and 18 in calibrate, all of which were DB connection errors which, as always, should be resolved upon resubmission. 2 errors were in isr which looked like: 174534 /47250_isr_174534_47. 6427736 .err:lsst.pex.exceptions.wrappers.FitsError: 174534 /47250_isr_174534_47. 6427736 .err:lsst::afw::fits::FitsError: 'cfitsio error: couldn' t create the named file ( 105 ) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits' with mode 'w' 174534 /47250_isr_174534_47. 6427736 .err: raise RuntimeError(f "Failed to serialize dataset {ref} of type {type(inMemoryDataset)} " 174534 /47250_isr_174534_47. 6427736 .err:RuntimeError: Failed to serialize dataset postISRCCD@{instrument: 'LSSTCam-imSim' , detector: 47 , exposure: 174534 , ...}, sc=Exposure] (id=23f8f2ff-bd96- 4298 -b154-62a5e1c44583) of type < class 'lsst.afw.image.exposure.ExposureF' > to location file: ///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits 174534 /47250_isr_174534_47. 6427736 .err:lsst.pex.exceptions.wrappers.FitsError: 174534 /47250_isr_174534_47. 6427736 .err:lsst::afw::fits::FitsError: 'cfitsio error: couldn' t create the named file ( 105 ) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits' with mode 'w' 174534 /47250_isr_174534_47. 6427736 .err: raise RuntimeError(f "Failed to serialize dataset {ref} of type {type(inMemoryDataset)} " 174534 /47250_isr_174534_47. 6427736 .err:RuntimeError: Failed to serialize dataset postISRCCD@{instrument: 'LSSTCam-imSim' , detector: 47 , exposure: 174534 , ...}, sc=Exposure] (id=23f8f2ff-bd96- 4298 -b154-62a5e1c44583) of type < class 'lsst.afw.image.exposure.ExposureF' > to location file: ///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211028T181741Z/postISRCCD/20220818/174534/postISRCCD_LSSTCam-imSim_i_sim_1_4_174534_R12_S02_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211028T181741Z.fits My hope is that these 2 errors will also be resolved upon resubmission. There was also 1 error in calibrate which was seen in the last DC2 weekly, w40 : 265317 /174188_calibrate_265317_8. 6517954 .err: raise pipeBase.TaskError( 265317 /174188_calibrate_265317_8. 6517954 .err:lsst.pipe.base.task.TaskError: Fit failed: median scatter on sky = 13.437 arcsec > 10.000 config.maxScatterArcsec

Hide

Permalink

Brock Brendal [X] (Inactive) added a comment - 01/Nov/21 9:10 PM

All connection issues were resolved with 2 resubmissions. Proceeding to step2

Show

Brock Brendal [X] (Inactive) added a comment - 01/Nov/21 9:10 PM All connection issues were resolved with 2 resubmissions. Proceeding to step2

Hide

Permalink

Brock Brendal [X] (Inactive) added a comment - 03/Nov/21 6:42 PM

Steps 2 and 3 completed with no errors. Step 4 had just 3 errors which are from an ongoing issue: ~~DM-31777~~:

183772/29359_imageDifference_183772_58.6755844.err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (20667.8, 18700.3); no input images at that point.'

408941/34597_imageDifference_408941_1.6750685.err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (17254.6, 24950.9); no input images at that point.'

466756/31226_imageDifference_466756_10.6753177.err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (10432.2, 13566.7); no input images at that point.'

Show

Brock Brendal [X] (Inactive) added a comment - 03/Nov/21 6:42 PM Steps 2 and 3 completed with no errors. Step 4 had just 3 errors which are from an ongoing issue: DM-31777 : 183772 /29359_imageDifference_183772_58. 6755844 .err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (20667.8, 18700.3); no input images at that point.' 408941 /34597_imageDifference_408941_1. 6750685 .err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (17254.6, 24950.9); no input images at that point.' 466756 /31226_imageDifference_466756_10. 6753177 .err:lsst::pex::exceptions::InvalidParameterError: 'Cannot compute CoaddPsf at point (10432.2, 13566.7); no input images at that point.'

Hide

Permalink

Brock Brendal [X] (Inactive) added a comment - 04/Nov/21 3:11 PM

Step 5 completed with 1 failure in forcedPhotCcdOnDiaObjects:

jobs/forcedPhotCcdOnDiaObjects/3829/248970/4425_forcedPhotCcdOnDiaObjects_3829_248970_54.6796390.err:lsst.pex.exceptions.wrappers.FitsError:

jobs/forcedPhotCcdOnDiaObjects/3829/248970/4425_forcedPhotCcdOnDiaObjects_3829_248970_54.6796390.err:lsst::afw::fits::FitsError: 'cfitsio error: couldn't create the named file (105) : Opening file '/repo/dc2/2.

2i/runs/test-med-1/w_2021_44/DM-32391/20211103T184424Z/forced_src_diaObject/3829/20221209/i/i_sim_1.4/248970/forced_src_diaObject_LSSTCam-imSim_3829_i_i_sim_1_4_248970_R13_S00_DC2_2_2i_runs_test-med-1_w_2021_44

_DM-32391_20211103T184424Z.fits' with mode 'w'

jobs/forcedPhotCcdOnDiaObjects/3829/248970/4425_forcedPhotCcdOnDiaObjects_3829_248970_54.6796390.err:    raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} "

jobs/forcedPhotCcdOnDiaObjects/3829/248970/4425_forcedPhotCcdOnDiaObjects_3829_248970_54.6796390.err:RuntimeError: Failed to serialize dataset forced_src_diaObject@{instrument: 'LSSTCam-imSim', skymap: 'DC2', d

etector: 54, tract: 3829, visit: 248970, ...}, sc=SourceCatalog] (id=409e4917-fedc-49ca-929e-569e4b8296fb) of type <class 'lsst.afw.table.SourceCatalog'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021

_44/DM-32391/20211103T184424Z/forced_src_diaObject/3829/20221209/i/i_sim_1.4/248970/forced_src_diaObject_LSSTCam-imSim_3829_i_i_sim_1_4_248970_R13_S00_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424

Z.fits

and 1, similar failure in forcedPhotDiffOnDiaObjects:

jobs/forcedPhotDiffOnDiaObjects/3829/12471/14141_forcedPhotDiffOnDiaObjects_3829_12471_77.6796397.err:lsst.pex.exceptions.wrappers.FitsError:

jobs/forcedPhotDiffOnDiaObjects/3829/12471/14141_forcedPhotDiffOnDiaObjects_3829_12471_77.6796397.err:lsst::afw::fits::FitsError: 'cfitsio error: couldn't create the named file (105) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211103T184424Z/forced_diff_diaObject/3829/20220117/y/y_sim_1.4/12471/forced_diff_diaObject_LSSTCam-imSim_3829_y_y_sim_1_4_12471_R20_S12_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424Z.fits' with mode 'w'

jobs/forcedPhotDiffOnDiaObjects/3829/12471/14141_forcedPhotDiffOnDiaObjects_3829_12471_77.6796397.err:    raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} "

jobs/forcedPhotDiffOnDiaObjects/3829/12471/14141_forcedPhotDiffOnDiaObjects_3829_12471_77.6796397.err:RuntimeError: Failed to serialize dataset forced_diff_diaObject@{instrument: 'LSSTCam-imSim', skymap: 'DC2', detector: 77, tract: 3829, visit: 12471, ...}, sc=SourceCatalog] (id=f7c0dfaf-5c22-4b96-b2fc-3ff1437bc125) of type <class 'lsst.afw.table.SourceCatalog'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211103T184424Z/forced_diff_diaObject/3829/20220117/y/y_sim_1.4/12471/forced_diff_diaObject_LSSTCam-imSim_3829_y_y_sim_1_4_12471_R20_S12_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424Z.fits

Both errors were fixed by resubmitting the dag file.

Show

Brock Brendal [X] (Inactive) added a comment - 04/Nov/21 3:11 PM Step 5 completed with 1 failure in forcedPhotCcdOnDiaObjects: jobs/forcedPhotCcdOnDiaObjects/ 3829 / 248970 /4425_forcedPhotCcdOnDiaObjects_3829_248970_54. 6796390 .err:lsst.pex.exceptions.wrappers.FitsError: jobs/forcedPhotCcdOnDiaObjects/ 3829 / 248970 /4425_forcedPhotCcdOnDiaObjects_3829_248970_54. 6796390 .err:lsst::afw::fits::FitsError: 'cfitsio error: couldn' t create the named file ( 105 ) : Opening file '/repo/dc2/ 2 . 2i/runs/test-med- 1 /w_2021_44/DM- 32391 /20211103T184424Z/forced_src_diaObject/ 3829 / 20221209 /i/i_sim_1. 4 / 248970 /forced_src_diaObject_LSSTCam-imSim_3829_i_i_sim_1_4_248970_R13_S00_DC2_2_2i_runs_test-med-1_w_2021_44 _DM-32391_20211103T184424Z.fits ' with mode ' w' jobs/forcedPhotCcdOnDiaObjects/ 3829 / 248970 /4425_forcedPhotCcdOnDiaObjects_3829_248970_54. 6796390 .err: raise RuntimeError(f "Failed to serialize dataset {ref} of type {type(inMemoryDataset)} " jobs/forcedPhotCcdOnDiaObjects/ 3829 / 248970 /4425_forcedPhotCcdOnDiaObjects_3829_248970_54. 6796390 .err:RuntimeError: Failed to serialize dataset forced_src_diaObject@{instrument: 'LSSTCam-imSim' , skymap: 'DC2' , d etector: 54 , tract: 3829 , visit: 248970 , ...}, sc=SourceCatalog] (id=409e4917-fedc-49ca-929e-569e4b8296fb) of type < class 'lsst.afw.table.SourceCatalog' > to location file: ///repo/dc2/2.2i/runs/test-med-1/w_2021 _44/DM- 32391 /20211103T184424Z/forced_src_diaObject/ 3829 / 20221209 /i/i_sim_1. 4 / 248970 /forced_src_diaObject_LSSTCam-imSim_3829_i_i_sim_1_4_248970_R13_S00_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424 Z.fits and 1, similar failure in forcedPhotDiffOnDiaObjects: jobs/forcedPhotDiffOnDiaObjects/ 3829 / 12471 /14141_forcedPhotDiffOnDiaObjects_3829_12471_77. 6796397 .err:lsst.pex.exceptions.wrappers.FitsError: jobs/forcedPhotDiffOnDiaObjects/ 3829 / 12471 /14141_forcedPhotDiffOnDiaObjects_3829_12471_77. 6796397 .err:lsst::afw::fits::FitsError: 'cfitsio error: couldn' t create the named file ( 105 ) : Opening file '/repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211103T184424Z/forced_diff_diaObject/3829/20220117/y/y_sim_1.4/12471/forced_diff_diaObject_LSSTCam-imSim_3829_y_y_sim_1_4_12471_R20_S12_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424Z.fits' with mode 'w' jobs/forcedPhotDiffOnDiaObjects/ 3829 / 12471 /14141_forcedPhotDiffOnDiaObjects_3829_12471_77. 6796397 .err: raise RuntimeError(f "Failed to serialize dataset {ref} of type {type(inMemoryDataset)} " jobs/forcedPhotDiffOnDiaObjects/ 3829 / 12471 /14141_forcedPhotDiffOnDiaObjects_3829_12471_77. 6796397 .err:RuntimeError: Failed to serialize dataset forced_diff_diaObject@{instrument: 'LSSTCam-imSim' , skymap: 'DC2' , detector: 77 , tract: 3829 , visit: 12471 , ...}, sc=SourceCatalog] (id=f7c0dfaf-5c22-4b96-b2fc-3ff1437bc125) of type < class 'lsst.afw.table.SourceCatalog' > to location file: ///repo/dc2/2.2i/runs/test-med-1/w_2021_44/DM-32391/20211103T184424Z/forced_diff_diaObject/3829/20220117/y/y_sim_1.4/12471/forced_diff_diaObject_LSSTCam-imSim_3829_y_y_sim_1_4_12471_R20_S12_DC2_2_2i_runs_test-med-1_w_2021_44_DM-32391_20211103T184424Z.fits Both errors were fixed by resubmitting the dag file.

Hide

Permalink

Brock Brendal [X] (Inactive) added a comment - 04/Nov/21 3:13 PM

One thing to note: I accidentally ran the majority of this weekly without mergeExecutionButler. I'm not sure if this will cause any data problems, but I thought it was worth mentioning/recording.

Show

Brock Brendal [X] (Inactive) added a comment - 04/Nov/21 3:13 PM One thing to note: I accidentally ran the majority of this weekly without mergeExecutionButler. I'm not sure if this will cause any data problems, but I thought it was worth mentioning/recording.

People

Assignee:: Brock Brendal [X] (Inactive)

Reporter:: Brock Brendal [X] (Inactive)

Reviewers:: Yusra AlSayyad

Watchers:: Brock Brendal [X] (Inactive), Yusra AlSayyad

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 28/Oct/21 5:03 PM

Updated:: 03/May/22 3:04 AM

Resolved:: 24/Mar/22 12:52 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins