Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Sprint:AP S21-2 (January)
-
Team:Alert Production
-
Urgent?:No
Description
The ap_verify_ci_hits2015 repository was apparently not updated on DM-27857, only its configs were. Perform the update now.
Attachments
Issue Links
Activity
(lsst-scipipe) [mrawls@lsst-devl01 hits2015-3]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "visit in (412518, 412568) and band='g'" --output DM-28438/ |
datasetIngest-gen3 INFO: Ingesting raw images...
|
datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures |
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully
|
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully
|
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully
|
datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest. |
datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets |
datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015-3/DM-28438/repo |
datasetIngest-gen3 INFO: Defining visits...
|
datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs.
|
datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0: one-to-one. |
datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits. |
datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s). |
datasetIngest-gen3 INFO: Storing data-specific configs...
|
datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015-3/DM-28438/config |
ap.verify.ingestion.ingestDataset INFO: Data ingested
|
ap.verify.ap_verify.main INFO: Running pipeline...
|
py.warnings WARN: /software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/qgraph.py:132: UserWarning: QuantumGraph is empty |
qgraph = f.makeGraph(pipelineObj, args)
|
|
|
|
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py", line 329, in invoke |
cli.main(args=args or (), prog_name=prog_name, **extra)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main |
rv = self.invoke(ctx)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke |
return _process_result(sub_ctx.command.invoke(sub_ctx)) |
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke |
return ctx.invoke(self.callback, **ctx.params) |
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke |
return callback(*args, **kwargs) |
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func |
return f(get_current_context(), *args, **kwargs) |
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run |
script.run(qgraphObj=qgraph, **kwargs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run |
f.runPipeline(qgraphObj, taskFactory, args)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 577, in runPipeline |
preExecInit.initialize(graph,
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py", line 84, in initialize |
self.initializeDatasetTypes(graph, registerDatasetTypes)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py", line 117, in initializeDatasetTypes |
pipeline = graph.taskGraph
|
AttributeError: 'NoneType' object has no attribute 'taskGraph' |
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py", line 29, in <module> |
result = runApVerify()
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py", line 184, in runApVerify |
return runApPipeGen3(workspace, args, processes=args.processes) |
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py", line 179, in runApPipeGen3 |
raise RuntimeError("Pipeline failed.") from results.exception |
RuntimeError: Pipeline failed.
|
And then, another try without specifying visits
(lsst-scipipe) [mrawls@lsst-devl01 hits2015-3]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "band='g'" --output DM-28438/
|
datasetIngest-gen3 INFO: Ingesting raw images...
|
datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures
|
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully
|
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully
|
datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully
|
datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest.
|
datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets
|
datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015-3/DM-28438/repo
|
datasetIngest-gen3 INFO: Defining visits...
|
datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs.
|
datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0: one-to-one.
|
datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits.
|
datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s).
|
datasetIngest-gen3 INFO: Storing data-specific configs...
|
datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015-3/DM-28438/config
|
ap.verify.ingestion.ingestDataset INFO: Data ingested
|
ap.verify.ap_verify.main INFO: Running pipeline...
|
ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 85 quanta for 29 tasks, graph ID: '1611196598.954383-1540694'
|
conda.common.io INFO: overtaking stderr and stdout
|
conda.common.io INFO: stderr and stdout yielding back
|
py.warnings WARN: /software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/astropy/table/row.py:76: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future.
|
return self.as_void() == other
|
|
|
isr INFO: Converting exposure to floating point values.
|
isr INFO: Applying crosstalk correction.
|
isr.crosstalk INFO: Applying crosstalk correction.
|
isr.crosstalk WARN: Crosstalk contains interChip coefficients, but no sources found!
|
isr INFO: Assembling CCD from amplifiers.
|
isr INFO: Applying bias correction.
|
isr INFO: Applying linearizer.
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py", line 329, in invoke
|
cli.main(args=args or (), prog_name=prog_name, **extra)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main
|
rv = self.invoke(ctx)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
|
return _process_result(sub_ctx.command.invoke(sub_ctx))
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
|
return ctx.invoke(self.callback, **ctx.params)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke
|
return callback(*args, **kwargs)
|
File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
|
return f(get_current_context(), *args, **kwargs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run
|
script.run(qgraphObj=qgraph, **kwargs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run
|
f.runPipeline(qgraphObj, taskFactory, args)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 595, in runPipeline
|
executor.execute(graph, butler)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 278, in execute
|
self._executeQuantaInProcess(graph, butler)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 328, in _executeQuantaInProcess
|
self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute
|
self.runQuantum(task, quantum, taskDef, butler)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum
|
task.runQuantum(butlerQC, inputRefs, outputRefs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 996, in runQuantum
|
outputs = self.run(**inputs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+2605bebd58/python/lsst/pipe/base/timer.py", line 157, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 1443, in run
|
linearizer.applyLinearity(image=ccdExposure.getMaskedImage().getImage(),
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 459, in applyLinearity
|
success, outOfRange = linearizer()(ampView, **{'coeffs': self.linearityCoeffs[ampName],
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 577, in __call__
|
numTableRows = table.shape[0]
|
AttributeError: 'NoneType' object has no attribute 'shape'
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py", line 29, in <module>
|
result = runApVerify()
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py", line 184, in runApVerify
|
return runApPipeGen3(workspace, args, processes=args.processes)
|
File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py", line 179, in runApPipeGen3
|
raise RuntimeError("Pipeline failed.") from results.exception
|
RuntimeError: Pipeline failed.
|
For the first case, have visits been properly defined from exposures? For the second case, perhaps the linearizer needs to be turned off? It's not complaining about skymaps not existing anymore, so at least that's progress!
Visits should be defined as part of ap_verify's ingestion, yes. For the linearizer, see DM-23985; I assumed you knew about it and had a workaround.
Krzysztof Findeisen
added a comment - Visits should be defined as part of ap_verify 's ingestion, yes. For the linearizer, see DM-23985 ; I assumed you knew about it and had a workaround. In the first case, the only visits covered by the CI dataset are 411371, 411420, and 419802. 
Krzysztof Findeisen
added a comment - In the first case, the only visits covered by the CI dataset are 411371, 411420, and 419802. I knew about the lack of gen3 linearization (and crosstalk), but assumed CI was somehow running in gen3. My bad (user error) for telling it the wrong visits, though thank you for making a note to update the docs example to visits that exist. I also learned that we're currently only running CI in gen3 with the HSC ap_verify dataset and still using gen2 for the DECam one.
Lacking a way to run ap_verify on DECam in gen3 until linearization and crosstalk work, it is difficult to review or test this. Here's what I did:
- Successfully ran ap_verify on the CI HiTS dataset with gen2 on this ticket branch (you didn't break it!)
- Successfully launched ap_verify on the CI HiTS dataset with gen3 on this ticket branch, which ran until it fell over at the linearization step of ISR (ap_verify is designed to not be configurable from the command line). This is farther than it got before this ticket.
And now I also managed to run ap_verify on the CI HSC (cosmos_pdr2) dataset in both gen2 and gen3. All set!
Meredith Rawls, can you sign off on this? Thanks!