# Fix reportPatches

XMLWordPrintable

## Details

• Type: Improvement
• Status: Invalid
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:

## Description

reportPatches.py is supposed to report which patches a visit (or set of visits) overlaps, but it currently doesn't work. The main error line is:

sqlite3.OperationalError: no such column: tract

## Activity

Hide
Russell Owen added a comment -

Please verify that a skymap exists for the type of coadd you are using.

If so, please provide an example, preferably that can be run on lsst-dev (or a self-contained repo that I can run on my own machine, but that may be more difficult).

Show
Russell Owen added a comment - Please verify that a skymap exists for the type of coadd you are using. If so, please provide an example, preferably that can be run on lsst-dev (or a self-contained repo that I can run on my own machine, but that may be more difficult).
Hide
David Nidever [X] (Inactive) added a comment -

Okay, here's what I do on lsst-dev. However, someone told me that this was already a known problem. Does reportPatches.py every work for you?

[dnidever@lsst-dev ~]$setup anaconda -t b1691 [dnidever@lsst-dev ~]$ setup pipe_tasks -t b1691
[dnidever@lsst-dev ~]$setup -j -r /home/dnidever/stack/obs_decam/ -t b1691 [dnidever@lsst-dev ~]$ eups declare -r /lsst8/yusra/index/cosmos_2mass/ astrometry_net_data 2mass-cosmos
[dnidever@lsst-dev ~]$setup astrometry_net_data 2mass-cosmos [dnidever@lsst-dev ~]$
[dnidever@lsst-dev ~]$reportPatches.py /Users/nidever/lsst/data/cosmos/ --id tract=0 patch=14,14 usage: reportPatches.py input [options] reportPatches.py: error: Error: input='/Users/nidever/lsst/data/cosmos' not found [dnidever@lsst-dev ~]$ reportPatches.py /lsst8/decam/redux/cp/cosmos/ --id tract=0 patch=14,14
: Config override file does not exist: '/home/dnidever/stack/obs_decam/config/reportPatches.py'
: Config override file does not exist: '/home/dnidever/stack/obs_decam/config/decam/reportPatches.py'
: input=/lsst8/decam/redux/cp/cosmos
: calib=None
: output=None
CameraMapper WARNING: Unable to locate calibRegistry registry in root: /lsst8/decam/redux/cp/cosmos/calibRegistry.sqlite3
CameraMapper WARNING: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
CameraMapper WARNING: No registry loaded; proceeding without one
Traceback (most recent call last):
File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 141, in <module>
File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/cmdLineTask.py", line 433, in parseAndRun
parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 399, in parse_args
self._processDataIds(namespace)
File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 453, in _processDataIds
dataIdContainer.makeDataRefList(namespace)
File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 135, in makeDataRefList
dataId = expandedDataId,
File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 368, in dataRef
subset = self.subset(datasetType, level, dataId, **rest)
File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 352, in subset
return ButlerSubset(self, datasetType, level, dataId)
File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butlerSubset.py", line 90, in _init_
level, fmt, self.dataId):
File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 172, in queryMetadata
tuples = self.mapper.queryMetadata(datasetType, key, format, dataId)
File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/mapper.py", line 101, in queryMetadata
return func(key, format, self.validate(dataId))
File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/cameraMapper.py", line 290, in queryClosure
return mapping.lookup(format, dataId)
File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup
where, self.range, values)
File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery
c = self.conn.execute(cmd, values)
sqlite3.OperationalError: no such column: tract

Show
David Nidever [X] (Inactive) added a comment - Okay, here's what I do on lsst-dev. However, someone told me that this was already a known problem. Does reportPatches.py every work for you? [dnidever@lsst-dev ~] $setup anaconda -t b1691 [dnidever@lsst-dev ~]$ setup pipe_tasks -t b1691 [dnidever@lsst-dev ~] $setup -j -r /home/dnidever/stack/obs_decam/ -t b1691 [dnidever@lsst-dev ~]$ eups declare -r /lsst8/yusra/index/cosmos_2mass/ astrometry_net_data 2mass-cosmos [dnidever@lsst-dev ~] $setup astrometry_net_data 2mass-cosmos [dnidever@lsst-dev ~]$ [dnidever@lsst-dev ~] $reportPatches.py /Users/nidever/lsst/data/cosmos/ --id tract=0 patch=14,14 usage: reportPatches.py input [options] reportPatches.py: error: Error: input='/Users/nidever/lsst/data/cosmos' not found [dnidever@lsst-dev ~]$ reportPatches.py /lsst8/decam/redux/cp/cosmos/ --id tract=0 patch=14,14 : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/reportPatches.py' : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/decam/reportPatches.py' : input=/lsst8/decam/redux/cp/cosmos : calib=None : output=None CameraMapper: Loading registry registry from /lsst8/decam/redux/cp/cosmos/registry.sqlite3 CameraMapper WARNING: Unable to locate calibRegistry registry in root: /lsst8/decam/redux/cp/cosmos/calibRegistry.sqlite3 CameraMapper WARNING: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3 CameraMapper WARNING: No registry loaded; proceeding without one Traceback (most recent call last): File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 141, in <module> ReportPatchesTask.parseAndRun() File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/cmdLineTask.py", line 433, in parseAndRun parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 399, in parse_args self._processDataIds(namespace) File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 453, in _processDataIds dataIdContainer.makeDataRefList(namespace) File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 135, in makeDataRefList dataId = expandedDataId, File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 368, in dataRef subset = self.subset(datasetType, level, dataId, **rest) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 352, in subset return ButlerSubset(self, datasetType, level, dataId) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butlerSubset.py", line 90, in _ init _ level, fmt, self.dataId): File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 172, in queryMetadata tuples = self.mapper.queryMetadata(datasetType, key, format, dataId) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/mapper.py", line 101, in queryMetadata return func(key, format, self.validate(dataId)) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/cameraMapper.py", line 290, in queryClosure return mapping.lookup(format, dataId) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup where, self.range, values) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery c = self.conn.execute(cmd, values) sqlite3.OperationalError: no such column: tract
Hide
Russell Owen added a comment -

It turns out the problem is a misleading error message from the butler that is triggered by your failure to specify a filter.

You can see the same error by doing this:

 butler.get("deepCoadd", {tract=0, patch="14,14"}) 

To see this you can put a debugging break such as import pdb; pdb.set_trace() in reportPatches.py at line 131 in makeDataRefList right after for dataId in self.idList: and then issue the command above, but replacing butler with namespace.butler.

If you add a filter to the data ID then the command will work or fail in a sensible way, depending on whether the coadd exists.

I will close this ticket for now, but feel free to reopen it as a ticket against the butler to give a more reasonable error message (such as "filter is not specified").

Note that you also have to specify an RA/Dec range to use this task, but if you fail to do that you get a reasonable (or at least semi-reasonable) error message:

 FieldValidationError: ListField 'raDecRange' failed validation: Required value cannot be None For more information read the Field definition at:  File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 51, in ReportPatchesConfig  length = 4, And the Config definition at:  File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 40, in   class ReportPatchesConfig(pexConfig.Config): 

Show
Hide
Russell Owen added a comment -

The problem is a misleading error message from the butler, rather than an error in the task

Show
Russell Owen added a comment - The problem is a misleading error message from the butler, rather than an error in the task
Hide
David Nidever [X] (Inactive) added a comment -

Thanks for this.

What is the format to specify the RA and DEC range? Also, why is an RA/DEC range needed? Doesn't the patch already set a RA/DEC range?

Show
David Nidever [X] (Inactive) added a comment - Thanks for this. What is the format to specify the RA and DEC range? Also, why is an RA/DEC range needed? Doesn't the patch already set a RA/DEC range?
Hide
David Nidever [X] (Inactive) added a comment -

Okay, I'm able to get reportPatches.py run, but now I'm confused about what it actually is doing. It appears that it requires some --id information but it doesn't matter what is given (you can give completely fake information). And you also can't input --dataId such as a visit number. I thought that this code would tell me which patches are overlapped by a certain visit/ccdnum. But maybe that is not the case. If so, maybe reportPatches.py is doing exactly what it is meant to do and I need something else.

Show
David Nidever [X] (Inactive) added a comment - Okay, I'm able to get reportPatches.py run, but now I'm confused about what it actually is doing. It appears that it requires some --id information but it doesn't matter what is given (you can give completely fake information). And you also can't input --dataId such as a visit number. I thought that this code would tell me which patches are overlapped by a certain visit/ccdnum. But maybe that is not the case. If so, maybe reportPatches.py is doing exactly what it is meant to do and I need something else.

## People

• Assignee:
Russell Owen
Reporter:
David Nidever [X] (Inactive)
Watchers:
David Nidever [X] (Inactive), Kian-Tat Lim, Nate Lust, Russell Owen