Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28438

Update CI-HiTS2015 repository

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • ap_verify
    • None
    • 1
    • AP S21-2 (January)
    • Alert Production
    • No

    Description

      The ap_verify_ci_hits2015 repository was apparently not updated on DM-27857, only its configs were. Perform the update now.

      Attachments

        Issue Links

          Activity

            mrawls, can you sign off on this? Thanks!

            krzys Krzysztof Findeisen added a comment - mrawls , can you sign off on this? Thanks!

            (lsst-scipipe) [mrawls@lsst-devl01 hits2015-3]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "visit in (412518, 412568) and band='g'" --output DM-28438/
            datasetIngest-gen3 INFO: Ingesting raw images...
            datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully
            datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest.
            datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets
            datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015-3/DM-28438/repo
            datasetIngest-gen3 INFO: Defining visits...
            datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs.
            datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0: one-to-one.
            datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits.
            datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s).
            datasetIngest-gen3 INFO: Storing data-specific configs...
            datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015-3/DM-28438/config
            ap.verify.ingestion.ingestDataset INFO: Data ingested
            ap.verify.ap_verify.main INFO: Running pipeline...
            py.warnings WARN: /software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/qgraph.py:132: UserWarning: QuantumGraph is empty
              qgraph = f.makeGraph(pipelineObj, args)
             
             
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py", line 329, in invoke
                cli.main(args=args or (), prog_name=prog_name, **extra)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main
                rv = self.invoke(ctx)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
                return _process_result(sub_ctx.command.invoke(sub_ctx))
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
                return ctx.invoke(self.callback, **ctx.params)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke
                return callback(*args, **kwargs)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
                return f(get_current_context(), *args, **kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run
                script.run(qgraphObj=qgraph, **kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run
                f.runPipeline(qgraphObj, taskFactory, args)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 577, in runPipeline
                preExecInit.initialize(graph,
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py", line 84, in initialize
                self.initializeDatasetTypes(graph, registerDatasetTypes)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py", line 117, in initializeDatasetTypes
                pipeline = graph.taskGraph
            AttributeError: 'NoneType' object has no attribute 'taskGraph'
             
             
            The above exception was the direct cause of the following exception:
             
             
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py", line 29, in <module>
                result = runApVerify()
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py", line 184, in runApVerify
                return runApPipeGen3(workspace, args, processes=args.processes)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py", line 179, in runApPipeGen3
                raise RuntimeError("Pipeline failed.") from results.exception
            RuntimeError: Pipeline failed. 

            And then, another try without specifying visits

            (lsst-scipipe) [mrawls@lsst-devl01 hits2015-3]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "band='g'" --output DM-28438/
            datasetIngest-gen3 INFO: Ingesting raw images...
            datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully
            datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully
            datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest.
            datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets
            datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015-3/DM-28438/repo
            datasetIngest-gen3 INFO: Defining visits...
            datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs.
            datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0: one-to-one.
            datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits.
            datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s).
            datasetIngest-gen3 INFO: Storing data-specific configs...
            datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015-3/DM-28438/config
            ap.verify.ingestion.ingestDataset INFO: Data ingested
            ap.verify.ap_verify.main INFO: Running pipeline...
            ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 85 quanta for 29 tasks, graph ID: '1611196598.954383-1540694'
            conda.common.io INFO: overtaking stderr and stdout
            conda.common.io INFO: stderr and stdout yielding back
            py.warnings WARN: /software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/astropy/table/row.py:76: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future.
              return self.as_void() == other
             
            isr INFO: Converting exposure to floating point values.
            isr INFO: Applying crosstalk correction.
            isr.crosstalk INFO: Applying crosstalk correction.
            isr.crosstalk WARN: Crosstalk contains interChip coefficients, but no sources found!
            isr INFO: Assembling CCD from amplifiers.
            isr INFO: Applying bias correction.
            isr INFO: Applying linearizer.
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py", line 329, in invoke
                cli.main(args=args or (), prog_name=prog_name, **extra)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main
                rv = self.invoke(ctx)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
                return _process_result(sub_ctx.command.invoke(sub_ctx))
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
                return ctx.invoke(self.callback, **ctx.params)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke
                return callback(*args, **kwargs)
              File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
                return f(get_current_context(), *args, **kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run
                script.run(qgraphObj=qgraph, **kwargs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run
                f.runPipeline(qgraphObj, taskFactory, args)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 595, in runPipeline
                executor.execute(graph, butler)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 278, in execute
                self._executeQuantaInProcess(graph, butler)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 328, in _executeQuantaInProcess
                self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute
                self.runQuantum(task, quantum, taskDef, butler)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum
                task.runQuantum(butlerQC, inputRefs, outputRefs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 996, in runQuantum
                outputs = self.run(**inputs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+2605bebd58/python/lsst/pipe/base/timer.py", line 157, in wrapper
                res = func(self, *args, **keyArgs)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 1443, in run
                linearizer.applyLinearity(image=ccdExposure.getMaskedImage().getImage(),
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 459, in applyLinearity
                success, outOfRange = linearizer()(ampView, **{'coeffs': self.linearityCoeffs[ampName],
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 577, in __call__
                numTableRows = table.shape[0]
            AttributeError: 'NoneType' object has no attribute 'shape'
             
            The above exception was the direct cause of the following exception:
             
            Traceback (most recent call last):
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py", line 29, in <module>
                result = runApVerify()
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py", line 184, in runApVerify
                return runApPipeGen3(workspace, args, processes=args.processes)
              File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py", line 179, in runApPipeGen3
                raise RuntimeError("Pipeline failed.") from results.exception
            RuntimeError: Pipeline failed.

            For the first case, have visits been properly defined from exposures? For the second case, perhaps the linearizer needs to be turned off? It's not complaining about skymaps not existing anymore, so at least that's progress!

            mrawls Meredith Rawls added a comment - (lsst-scipipe) [mrawls @lsst -devl01 hits2015- 3 ]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "visit in (412518, 412568) and band='g'" --output DM- 28438 / datasetIngest-gen3 INFO: Ingesting raw images... datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest. datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015- 3 /DM- 28438 /repo datasetIngest-gen3 INFO: Defining visits... datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs. datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0 : one-to-one. datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits. datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s). datasetIngest-gen3 INFO: Storing data-specific configs... datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015- 3 /DM- 28438 /config ap.verify.ingestion.ingestDataset INFO: Data ingested ap.verify.ap_verify.main INFO: Running pipeline... py.warnings WARN: /software/lsstsw/stack_20210114/stack/miniconda3-py38_4. 9.2 - 0.1 . 5 /Linux64/ctrl_mpexec/ 21.0 . 0 - 15 -g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/qgraph.py: 132 : UserWarning: QuantumGraph is empty   qgraph = f.makeGraph(pipelineObj, args)     Traceback (most recent call last):   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py" , line 329 , in invoke     cli.main(args=args or (), prog_name=prog_name, **extra)   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py" , line 782 , in main     rv = self.invoke(ctx)   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py" , line 1259 , in invoke     return _process_result(sub_ctx.command.invoke(sub_ctx))   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py" , line 1066 , in invoke     return ctx.invoke(self.callback, **ctx.params)   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py" , line 610 , in invoke     return callback(*args, **kwargs)   File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py" , line 21 , in new_func     return f(get_current_context(), *args, **kwargs)   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py" , line 103 , in run     script.run(qgraphObj=qgraph, **kwargs)   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py" , line 167 , in run     f.runPipeline(qgraphObj, taskFactory, args)   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py" , line 577 , in runPipeline     preExecInit.initialize(graph,   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py" , line 84 , in initialize     self.initializeDatasetTypes(graph, registerDatasetTypes)   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/preExecInit.py" , line 117 , in initializeDatasetTypes     pipeline = graph.taskGraph AttributeError: 'NoneType' object has no attribute 'taskGraph'     The above exception was the direct cause of the following exception:     Traceback (most recent call last):   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py" , line 29 , in <module>     result = runApVerify()   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py" , line 184 , in runApVerify     return runApPipeGen3(workspace, args, processes=args.processes)   File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py" , line 179 , in runApPipeGen3     raise RuntimeError( "Pipeline failed." ) from results.exception RuntimeError: Pipeline failed. And then, another try without specifying visits (lsst-scipipe) [mrawls@lsst-devl01 hits2015-3]$ ap_verify.py --dataset CI-HiTS2015 --gen3 --id "band='g'" --output DM-28438/ datasetIngest-gen3 INFO: Ingesting raw images... datasetIngest-gen3.ingester INFO: Successfully extracted metadata from 3 files with 0 failures datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t052637 ingested successfully datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150309t055516 ingested successfully datasetIngest-gen3.ingester INFO: Exposure DECam:ct4m20150218t070355 ingested successfully datasetIngest-gen3.ingester INFO: Successfully processed data from 3 exposures with 0 failures from exposure registration and 0 failures from file ingest. datasetIngest-gen3.ingester INFO: Ingested 3 distinct Butler datasets datasetIngest-gen3 INFO: Images are now ingested in /project/mrawls/hits2015-3/DM-28438/repo datasetIngest-gen3 INFO: Defining visits... datasetIngest-gen3.visitDefiner INFO: Preprocessing data IDs. datasetIngest-gen3.visitDefiner INFO: Registering visit_system 0: one-to-one. datasetIngest-gen3.visitDefiner INFO: Grouping 3 exposure(s) into visits. datasetIngest-gen3.visitDefiner INFO: Computing regions and other metadata for 3 visit(s). datasetIngest-gen3 INFO: Storing data-specific configs... datasetIngest-gen3 INFO: Configs are now stored in /project/mrawls/hits2015-3/DM-28438/config ap.verify.ingestion.ingestDataset INFO: Data ingested ap.verify.ap_verify.main INFO: Running pipeline... ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 85 quanta for 29 tasks, graph ID: '1611196598.954383-1540694' conda.common.io INFO: overtaking stderr and stdout conda.common.io INFO: stderr and stdout yielding back py.warnings WARN: /software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/astropy/table/row.py:76: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future. return self.as_void() == other   isr INFO: Converting exposure to floating point values. isr INFO: Applying crosstalk correction. isr.crosstalk INFO: Applying crosstalk correction. isr.crosstalk WARN: Crosstalk contains interChip coefficients, but no sources found! isr INFO: Assembling CCD from amplifiers. isr INFO: Applying bias correction. isr INFO: Applying linearizer. Traceback (most recent call last): File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/testing.py", line 329, in invoke cli.main(args=args or (), prog_name=prog_name, **extra) File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/software/lsstsw/stack_20210114/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run script.run(qgraphObj=qgraph, **kwargs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run f.runPipeline(qgraphObj, taskFactory, args) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 595, in runPipeline executor.execute(graph, butler) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 278, in execute self._executeQuantaInProcess(graph, butler) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 328, in _executeQuantaInProcess self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute self.runQuantum(task, quantum, taskDef, butler) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ctrl_mpexec/21.0.0-15-g7f4067b+b22f4079e0/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 274, in runQuantum task.runQuantum(butlerQC, inputRefs, outputRefs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 996, in runQuantum outputs = self.run(**inputs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/pipe_base/21.0.0-6-gd3283ba+2605bebd58/python/lsst/pipe/base/timer.py", line 157, in wrapper res = func(self, *args, **keyArgs) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/isrTask.py", line 1443, in run linearizer.applyLinearity(image=ccdExposure.getMaskedImage().getImage(), File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 459, in applyLinearity success, outOfRange = linearizer()(ampView, **{'coeffs': self.linearityCoeffs[ampName], File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ip_isr/21.0.0-3-gd5de2f2+905c13b855/python/lsst/ip/isr/linearize.py", line 577, in __call__ numTableRows = table.shape[0] AttributeError: 'NoneType' object has no attribute 'shape'   The above exception was the direct cause of the following exception:   Traceback (most recent call last): File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/bin/ap_verify.py", line 29, in <module> result = runApVerify() File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/ap_verify.py", line 184, in runApVerify return runApPipeGen3(workspace, args, processes=args.processes) File "/software/lsstsw/stack_20210114/stack/miniconda3-py38_4.9.2-0.1.5/Linux64/ap_verify/21.0.0-8-ge08b8c0+9475aa14e8/python/lsst/ap/verify/pipeline_driver.py", line 179, in runApPipeGen3 raise RuntimeError("Pipeline failed.") from results.exception RuntimeError: Pipeline failed. For the first case, have visits been properly defined from exposures? For the second case, perhaps the linearizer needs to be turned off? It's not complaining about skymaps not existing anymore, so at least that's progress!

            Visits should be defined as part of ap_verify's ingestion, yes. For the linearizer, see DM-23985; I assumed you knew about it and had a workaround.

            krzys Krzysztof Findeisen added a comment - Visits should be defined as part of ap_verify 's ingestion, yes. For the linearizer, see DM-23985 ; I assumed you knew about it and had a workaround.

            In the first case, the only visits covered by the CI dataset are 411371, 411420, and 419802.

            krzys Krzysztof Findeisen added a comment - In the first case, the only visits covered by the CI dataset are 411371, 411420, and 419802.

            I knew about the lack of gen3 linearization (and crosstalk), but assumed CI was somehow running in gen3. My bad (user error) for telling it the wrong visits, though thank you for making a note to update the docs example to visits that exist. I also learned that we're currently only running CI in gen3 with the HSC ap_verify dataset and still using gen2 for the DECam one.

            Lacking a way to run ap_verify on DECam in gen3 until linearization and crosstalk work, it is difficult to review or test this. Here's what I did:

            • Successfully ran ap_verify on the CI HiTS dataset with gen2 on this ticket branch (you didn't break it!)
            • Successfully launched ap_verify on the CI HiTS dataset with gen3 on this ticket branch, which ran until it fell over at the linearization step of ISR (ap_verify is designed to not be configurable from the command line). This is farther than it got before this ticket.

             

            mrawls Meredith Rawls added a comment - I knew about the lack of gen3 linearization (and crosstalk), but assumed CI was somehow running in gen3. My bad (user error) for telling it the wrong visits, though thank you for making a note to update the docs example to visits that exist. I also learned that we're currently only running CI in gen3 with the HSC ap_verify dataset and still using gen2 for the DECam one. Lacking a way to run ap_verify on DECam in gen3 until linearization and crosstalk work, it is difficult to review or test this. Here's what I did: Successfully ran ap_verify on the CI HiTS dataset with gen2 on this ticket branch (you didn't break it!) Successfully launched ap_verify on the CI HiTS dataset with gen3 on this ticket branch, which ran until it fell over at the linearization step of ISR (ap_verify is designed to not be configurable from the command line). This is farther than it got before this ticket.  

            And now I also managed to run ap_verify on the CI HSC (cosmos_pdr2) dataset in both gen2 and gen3. All set!

            mrawls Meredith Rawls added a comment - And now I also managed to run ap_verify on the CI HSC (cosmos_pdr2) dataset in both gen2 and gen3. All set!

            People

              krzys Krzysztof Findeisen
              krzys Krzysztof Findeisen
              Meredith Rawls
              John Parejko, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.