Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:2
-
Epic Link:
-
Sprint:AP S18-6, DB_F18_06
-
Team:Data Access and Database
Description
When attempting to combined HSC PDR1 (DM-13666) data for WIDE and DEEP, the following Butler construction does not work correctly:
butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/DM-13666/WIDE/'}, {'root': '/datasets/hsc/repo/rerun/DM-13666/DEEP/'}])
|
One clue is that this prints the following INFO:
CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
|
CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
|
CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/DM-13666/DEEP
|
CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
|
CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
|
CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
|
CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
|
And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry)
Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following).
I believe that on the first call to _setParentRegistry() this argument is `None` (not sure why), but on subsequent calls (after repoData.repo = Repository(repoData) is called) then it's okay.
I checked what would happen if instead of kicking out instead it calls parentRepoData.repo = Repository(parentRepoData) and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests).
Attachments
Issue Links
- blocks
-
DM-14669 Run fgcmcal on HSC PDR1
- Done
Activity
Field | Original Value | New Value |
---|---|---|
Epic Link |
|
Risk Score | 0 |
Description |
When attempting to combined HSC PDR1 ( {{ butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/ }} One clue is that this prints the following INFO: {{ CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/ CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 }} And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry) Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following). I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay. I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests). |
When attempting to combined HSC PDR1 ( {code: python} butler = dafPersistence.Butler(inputs=[{'root': '/datasets/hsc/repo/rerun/ {code} One clue is that this prints the following INFO: {code: python} CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading Posix exposure registry from /datasets/hsc/repo/rerun/ CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 {code} And there should not be anything with Posix registries here. (note that if you reverse the order of the inputs, then the WIDE gets loaded as a Posix registry) Upon deeper inspection, the problem is logged to debug in getParentRegsitry (sic): https://github.com/lsst/daf_persistence/blob/1e7c341a1a884cf06fc74a946a126c1f72f56360/python/lsst/daf/persistence/butler.py#L570 (and following). I believe that on the first call to {{_setParentRegistry()}} this argument is `None` (not sure why), but on subsequent calls (after {{repoData.repo = Repository(repoData)}} is called) then it's okay. I checked what would happen if instead of kicking out instead it calls {{parentRepoData.repo = Repository(parentRepoData)}} and that appears to work, though I'm afraid of the side-effects. (Although it does pass all local scons tests). |
Assignee | Nate Pease [ npease ] |
Story Points | 1 | 2 |
Team | Data Release Production [ 10301 ] | Data Access and Database [ 10204 ] |
Sprint | AP S18-6 [ 686 ] | AP S18-6, DB_F18_06 [ 686, 744 ] |
Status | To Do [ 10001 ] | In Progress [ 3 ] |
Reviewers | Kian-Tat Lim [ ktl ] | |
Status | In Progress [ 3 ] | In Review [ 10004 ] |
Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
Resolution | Done [ 10000 ] | |
Status | Reviewed [ 10101 ] | Done [ 10002 ] |