Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_drivers, pipe_tasks
-
Labels:
-
Story Points:2
-
Epic Link:
-
Team:Data Facility
Description
Error running multiBandDriver if refObjLoader is retargeted to LoadIndexedReferenceObjectsTask in measureCoaddSources, e.g. with this config override:
from lsst.meas.algorithms.loadIndexedReferenceObjects import LoadIndexedReferenceObjectsTask |
config.match.refObjLoader.retarget(LoadIndexedReferenceObjectsTask)
|
Traceback (most recent call last):
|
File "/software/lsstsw/stack/Linux64/pipe_drivers/12.1-7-ga5bc178+1/bin/multiBandDriver.py", line 3, in <module> |
MultiBandDriverTask.parseAndSubmit()
|
File "/software/lsstsw/stack/Linux64/ctrl_pool/12.1-1-g3e1834e/python/lsst/ctrl/pool/parallel.py", line 410, in parseAndSubmit |
if not cls.RunnerClass(cls, batchArgs.parent).precall(batchArgs.parent): # Write config, schema |
File "/software/lsstsw/stack/Linux64/pipe_base/12.1-1-g06158e9+2/python/lsst/pipe/base/cmdLineTask.py", line 300, in precall |
task = self.makeTask(parsedCmd=parsedCmd)
|
File "/software/lsstsw/stack/Linux64/pipe_drivers/12.1-7-ga5bc178+1/python/lsst/pipe/drivers/multiBandDriver.py", line 111, in makeTask |
return self.TaskClass(config=self.config, log=self.log, butler=butler) |
File "/software/lsstsw/stack/Linux64/pipe_drivers/12.1-7-ga5bc178+1/python/lsst/pipe/drivers/multiBandDriver.py", line 131, in __init__ |
peakSchema=afwTable.Schema(self.mergeCoaddDetections.merged.getPeakSchema()))
|
File "/software/lsstsw/stack/Linux64/pipe_base/12.1-1-g06158e9+2/python/lsst/pipe/base/task.py", line 237, in makeSubtask |
subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
|
File "/software/lsstsw/stack/Linux64/pex_config/12.1+6/python/lsst/pex/config/configurableField.py", line 83, in apply |
return self.target(*args, config=self.value, **kw) |
File "/software/lsstsw/stack/Linux64/pipe_tasks/12.1-3-g35418c8/python/lsst/pipe/tasks/multiBand.py", line 1039, in __init__ |
self.makeSubtask("match", butler=butler) |
File "/software/lsstsw/stack/Linux64/pipe_base/12.1-1-g06158e9+2/python/lsst/pipe/base/task.py", line 237, in makeSubtask |
subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
|
File "/software/lsstsw/stack/Linux64/pex_config/12.1+6/python/lsst/pex/config/configurableField.py", line 83, in apply |
return self.target(*args, config=self.value, **kw) |
File "/software/lsstsw/stack/Linux64/meas_astrom/12.1-2-gf2a177e+2/python/lsst/meas/astrom/directMatch.py", line 78, in __init__ |
self.makeSubtask("refObjLoader", butler=butler) |
File "/software/lsstsw/stack/Linux64/pipe_base/12.1-1-g06158e9+2/python/lsst/pipe/base/task.py", line 237, in makeSubtask |
subtask = taskField.apply(name=name, parentTask=self, **keyArgs)
|
File "/software/lsstsw/stack/Linux64/pex_config/12.1+6/python/lsst/pex/config/configurableField.py", line 83, in apply |
return self.target(*args, config=self.value, **kw) |
File "/software/lsstsw/stack/Linux64/meas_algorithms/12.1-6-g1f798ce+1/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py", line 48, in __init__ |
|
|
ingest_config = butler.get(self.config.ingest_config_name, immediate=True)
|
|
|
AttributeError: 'NoneType' object has no attribute 'get' |
I included a way to reproduce this without actual data in obs_decam branch u/hfc/DM-8000. There I added an empty Butler repo and the measureCoaddSources config override.
With that branch this command reproduces the error:
multiBandDriver.py $OBS_DECAM_DIR/repo/ --rerun test --cores 1
Attachments
Issue Links
Activity
Sorry I thought I found the solution and didn't explain well. It can be reproduced using the testdata_decam repo. I added a few things to branch u/hfc/DM-8000.
The schema schema/deepCoadd_det.fits from previous pipeline is needed.
I put my tentative fix to the ticket branch in pipe_tasks and pipe_drivers. After it passes the error of the ticket, config/IngestIndexedReferenceTask.py is needed. In actual scenario this config lives with the GAIA catalog repo. Then, it would report "Not running the task because there is no data to process" as there is no data to be processed in testdata_decam.
(I will double check again, but not sure if I can do it today)
Hsin-Fang Chiang
added a comment - Sorry I thought I found the solution and didn't explain well. It can be reproduced using the testdata_decam repo. I added a few things to branch u/hfc/ DM-8000 .
The schema schema/deepCoadd_det.fits from previous pipeline is needed.
I put my tentative fix to the ticket branch in pipe_tasks and pipe_drivers . After it passes the error of the ticket, config/IngestIndexedReferenceTask.py is needed. In actual scenario this config lives with the GAIA catalog repo. Then, it would report "Not running the task because there is no data to process" as there is no data to be processed in testdata_decam.
(I will double check again, but not sure if I can do it today) On second thought, the only thing I used from testdata_decam was its "_mapper" file; I don't really need testdata_decam.
So I rewrote the how-to-reproduce and included everything needed in obs_decam branch u/hfc/DM-8000. With that branch this command reproduces the error:
multiBandDriver.py $OBS_DECAM_DIR/repo/ --rerun test --cores 1
Hsin-Fang Chiang
added a comment - On second thought, the only thing I used from testdata_decam was its "_mapper" file; I don't really need testdata_decam.
So I rewrote the how-to-reproduce and included everything needed in obs_decam branch u/hfc/ DM-8000 . With that branch this command reproduces the error:
multiBandDriver.py $OBS_DECAM_DIR/repo/ --rerun test --cores 1 I think this happens because DirectMatchTask, the default of measureCoaddSources.match in multiBandDriver, requires either a butler or refObjLoader, but in MultiBandDriverTask's ctor the subtask measureCoaddSources is made without either. The minimum fix is to supply a butler to measureCoaddSources in pipe_drivers multiBandDriver.
I followed the lead of DM-6631 and added an alternative route of using refObjLoader from input. Adding refObjLoader and the pipe_tasks patch is not required to fix this ticket, but I think that would make it more correct or at least report a more useful error message.
Hsin-Fang Chiang
added a comment - I think this happens because DirectMatchTask , the default of measureCoaddSources.match in multiBandDriver, requires either a butler or refObjLoader , but in MultiBandDriverTask's ctor the subtask measureCoaddSources is made without either. The minimum fix is to supply a butler to measureCoaddSources in pipe_drivers multiBandDriver.
I followed the lead of DM-6631 and added an alternative route of using refObjLoader from input. Adding refObjLoader and the pipe_tasks patch is not required to fix this ticket, but I think that would make it more correct or at least report a more useful error message. Paul Price would you mind reviewing this ticket? I'm opening two PRs: in pipe_drivers and pipe_tasks.
Hsin-Fang Chiang
added a comment - Paul Price would you mind reviewing this ticket? I'm opening two PRs: in pipe_drivers and pipe_tasks . Looks reasonable to me. I have not reviewed the changes in a personal branch of obs_decam.
You may want to start using numpydoc for new docstrings, but don't feel obliged to do so in this case.
Paul Price
added a comment - Looks reasonable to me. I have not reviewed the changes in a personal branch of obs_decam.
You may want to start using numpydoc for new docstrings, but don't feel obliged to do so in this case. Thanks a lot Paul Price for your quick review.
The personal branch of obs_decam is only meant for reproducing the error in a clean way, and not for merging.
I think I'll pass about switching to numpydoc, as the two files I touch still use the old style throughout.
Although I'm not sure if these changes get tested at all, I'll hold on merging until after a successful Jenkins run.
Hsin-Fang Chiang
added a comment - Thanks a lot Paul Price for your quick review.
The personal branch of obs_decam is only meant for reproducing the error in a clean way, and not for merging.
I think I'll pass about switching to numpydoc, as the two files I touch still use the old style throughout.
Although I'm not sure if these changes get tested at all, I'll hold on merging until after a successful Jenkins run. Merged after running Jenkins.
Hsin-Fang Chiang
added a comment - Merged after running Jenkins . 
Can you please provide a how-to-reproduce?