Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4627

Fix reportPatches

    Details

    • Type: Improvement
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Team:
      Alert Production

      Description

      reportPatches.py is supposed to report which patches a visit (or set of visits) overlaps, but it currently doesn't work. The main error line is:

      sqlite3.OperationalError: no such column: tract

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            Please verify that a skymap exists for the type of coadd you are using.

            If so, please provide an example, preferably that can be run on lsst-dev (or a self-contained repo that I can run on my own machine, but that may be more difficult).

            Show
            rowen Russell Owen added a comment - Please verify that a skymap exists for the type of coadd you are using. If so, please provide an example, preferably that can be run on lsst-dev (or a self-contained repo that I can run on my own machine, but that may be more difficult).
            Hide
            nidever David Nidever [X] (Inactive) added a comment -

            Okay, here's what I do on lsst-dev. However, someone told me that this was already a known problem. Does reportPatches.py every work for you?

            [dnidever@lsst-dev ~]$ setup anaconda -t b1691
            [dnidever@lsst-dev ~]$ setup pipe_tasks -t b1691
            [dnidever@lsst-dev ~]$ setup -j -r /home/dnidever/stack/obs_decam/ -t b1691
            [dnidever@lsst-dev ~]$ eups declare -r /lsst8/yusra/index/cosmos_2mass/ astrometry_net_data 2mass-cosmos
            [dnidever@lsst-dev ~]$ setup astrometry_net_data 2mass-cosmos
            [dnidever@lsst-dev ~]$
            [dnidever@lsst-dev ~]$ reportPatches.py /Users/nidever/lsst/data/cosmos/ --id tract=0 patch=14,14
            usage: reportPatches.py input [options]
            reportPatches.py: error: Error: input='/Users/nidever/lsst/data/cosmos' not found
            [dnidever@lsst-dev ~]$ reportPatches.py /lsst8/decam/redux/cp/cosmos/ --id tract=0 patch=14,14
            : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/reportPatches.py'
            : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/decam/reportPatches.py'
            : input=/lsst8/decam/redux/cp/cosmos
            : calib=None
            : output=None
            CameraMapper: Loading registry registry from /lsst8/decam/redux/cp/cosmos/registry.sqlite3
            CameraMapper WARNING: Unable to locate calibRegistry registry in root: /lsst8/decam/redux/cp/cosmos/calibRegistry.sqlite3
            CameraMapper WARNING: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
            CameraMapper WARNING: No registry loaded; proceeding without one
            Traceback (most recent call last):
            File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 141, in <module>
            ReportPatchesTask.parseAndRun()
            File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/cmdLineTask.py", line 433, in parseAndRun
            parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides)
            File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 399, in parse_args
            self._processDataIds(namespace)
            File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 453, in _processDataIds
            dataIdContainer.makeDataRefList(namespace)
            File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 135, in makeDataRefList
            dataId = expandedDataId,
            File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 368, in dataRef
            subset = self.subset(datasetType, level, dataId, **rest)
            File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 352, in subset
            return ButlerSubset(self, datasetType, level, dataId)
            File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butlerSubset.py", line 90, in _init_
            level, fmt, self.dataId):
            File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 172, in queryMetadata
            tuples = self.mapper.queryMetadata(datasetType, key, format, dataId)
            File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/mapper.py", line 101, in queryMetadata
            return func(key, format, self.validate(dataId))
            File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/cameraMapper.py", line 290, in queryClosure
            return mapping.lookup(format, dataId)
            File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup
            where, self.range, values)
            File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery
            c = self.conn.execute(cmd, values)
            sqlite3.OperationalError: no such column: tract

            Show
            nidever David Nidever [X] (Inactive) added a comment - Okay, here's what I do on lsst-dev. However, someone told me that this was already a known problem. Does reportPatches.py every work for you? [dnidever@lsst-dev ~] $ setup anaconda -t b1691 [dnidever@lsst-dev ~] $ setup pipe_tasks -t b1691 [dnidever@lsst-dev ~] $ setup -j -r /home/dnidever/stack/obs_decam/ -t b1691 [dnidever@lsst-dev ~] $ eups declare -r /lsst8/yusra/index/cosmos_2mass/ astrometry_net_data 2mass-cosmos [dnidever@lsst-dev ~] $ setup astrometry_net_data 2mass-cosmos [dnidever@lsst-dev ~] $ [dnidever@lsst-dev ~] $ reportPatches.py /Users/nidever/lsst/data/cosmos/ --id tract=0 patch=14,14 usage: reportPatches.py input [options] reportPatches.py: error: Error: input='/Users/nidever/lsst/data/cosmos' not found [dnidever@lsst-dev ~] $ reportPatches.py /lsst8/decam/redux/cp/cosmos/ --id tract=0 patch=14,14 : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/reportPatches.py' : Config override file does not exist: '/home/dnidever/stack/obs_decam/config/decam/reportPatches.py' : input=/lsst8/decam/redux/cp/cosmos : calib=None : output=None CameraMapper: Loading registry registry from /lsst8/decam/redux/cp/cosmos/registry.sqlite3 CameraMapper WARNING: Unable to locate calibRegistry registry in root: /lsst8/decam/redux/cp/cosmos/calibRegistry.sqlite3 CameraMapper WARNING: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3 CameraMapper WARNING: No registry loaded; proceeding without one Traceback (most recent call last): File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 141, in <module> ReportPatchesTask.parseAndRun() File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/cmdLineTask.py", line 433, in parseAndRun parsedCmd = argumentParser.parse_args(config=config, args=args, log=log, override=cls.applyOverrides) File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 399, in parse_args self._processDataIds(namespace) File "/home/lsstsw/stack/Linux64/pipe_base/11.0.rc2+5/python/lsst/pipe/base/argumentParser.py", line 453, in _processDataIds dataIdContainer.makeDataRefList(namespace) File "/home/lsstsw/stack/Linux64/pipe_tasks/11.0.rc2-1-gebd59be+4/bin/reportPatches.py", line 135, in makeDataRefList dataId = expandedDataId, File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 368, in dataRef subset = self.subset(datasetType, level, dataId, **rest) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 352, in subset return ButlerSubset(self, datasetType, level, dataId) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butlerSubset.py", line 90, in _ init _ level, fmt, self.dataId): File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/butler.py", line 172, in queryMetadata tuples = self.mapper.queryMetadata(datasetType, key, format, dataId) File "/home/lsstsw/stack/Linux64/daf_persistence/11.0.rc2+4/python/lsst/daf/persistence/mapper.py", line 101, in queryMetadata return func(key, format, self.validate(dataId)) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/cameraMapper.py", line 290, in queryClosure return mapping.lookup(format, dataId) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/mapping.py", line 168, in lookup where, self.range, values) File "/home/lsstsw/stack/Linux64/daf_butlerUtils/11.0.rc2+5/python/lsst/daf/butlerUtils/registries.py", line 120, in executeQuery c = self.conn.execute(cmd, values) sqlite3.OperationalError: no such column: tract
            Hide
            rowen Russell Owen added a comment -

            It turns out the problem is a misleading error message from the butler that is triggered by your failure to specify a filter.

            You can see the same error by doing this:

            butler.get("deepCoadd", {tract=0, patch="14,14"})
            

            To see this you can put a debugging break such as import pdb; pdb.set_trace() in reportPatches.py at line 131 in makeDataRefList right after for dataId in self.idList: and then issue the command above, but replacing butler with namespace.butler.

            If you add a filter to the data ID then the command will work or fail in a sensible way, depending on whether the coadd exists.

            I will close this ticket for now, but feel free to reopen it as a ticket against the butler to give a more reasonable error message (such as "filter is not specified").

            Note that you also have to specify an RA/Dec range to use this task, but if you fail to do that you get a reasonable (or at least semi-reasonable) error message:

            FieldValidationError: ListField 'raDecRange' failed validation: Required value cannot be None
            For more information read the Field definition at:
              File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 51, in ReportPatchesConfig
                length = 4,
            And the Config definition at:
              File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 40, in <module>
                class ReportPatchesConfig(pexConfig.Config):
            

            Show
            rowen Russell Owen added a comment - It turns out the problem is a misleading error message from the butler that is triggered by your failure to specify a filter. You can see the same error by doing this: butler.get("deepCoadd", {tract=0, patch="14,14"}) To see this you can put a debugging break such as import pdb; pdb.set_trace() in reportPatches.py at line 131 in makeDataRefList right after for dataId in self.idList: and then issue the command above, but replacing butler with namespace.butler . If you add a filter to the data ID then the command will work or fail in a sensible way, depending on whether the coadd exists. I will close this ticket for now, but feel free to reopen it as a ticket against the butler to give a more reasonable error message (such as "filter is not specified"). Note that you also have to specify an RA/Dec range to use this task, but if you fail to do that you get a reasonable (or at least semi-reasonable) error message: FieldValidationError: ListField 'raDecRange' failed validation: Required value cannot be None For more information read the Field definition at: File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 51, in ReportPatchesConfig length = 4, And the Config definition at: File "/ssd/rowen/lsstsw/stack/Linux64/pipe_tasks/11.0-5-g2e9610c/bin/reportPatches.py", line 40, in <module> class ReportPatchesConfig(pexConfig.Config):
            Hide
            rowen Russell Owen added a comment -

            The problem is a misleading error message from the butler, rather than an error in the task

            Show
            rowen Russell Owen added a comment - The problem is a misleading error message from the butler, rather than an error in the task
            Hide
            nidever David Nidever [X] (Inactive) added a comment -

            Thanks for this.

            What is the format to specify the RA and DEC range? Also, why is an RA/DEC range needed? Doesn't the patch already set a RA/DEC range?

            Show
            nidever David Nidever [X] (Inactive) added a comment - Thanks for this. What is the format to specify the RA and DEC range? Also, why is an RA/DEC range needed? Doesn't the patch already set a RA/DEC range?
            Hide
            nidever David Nidever [X] (Inactive) added a comment -

            Okay, I'm able to get reportPatches.py run, but now I'm confused about what it actually is doing. It appears that it requires some --id information but it doesn't matter what is given (you can give completely fake information). And you also can't input --dataId such as a visit number. I thought that this code would tell me which patches are overlapped by a certain visit/ccdnum. But maybe that is not the case. If so, maybe reportPatches.py is doing exactly what it is meant to do and I need something else.

            Show
            nidever David Nidever [X] (Inactive) added a comment - Okay, I'm able to get reportPatches.py run, but now I'm confused about what it actually is doing. It appears that it requires some --id information but it doesn't matter what is given (you can give completely fake information). And you also can't input --dataId such as a visit number. I thought that this code would tell me which patches are overlapped by a certain visit/ccdnum. But maybe that is not the case. If so, maybe reportPatches.py is doing exactly what it is meant to do and I need something else.

              People

              • Assignee:
                rowen Russell Owen
                Reporter:
                nidever David Nidever [X] (Inactive)
                Watchers:
                David Nidever [X] (Inactive), Kian-Tat Lim, Nate Lust, Russell Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel