Details
-
Type:
Story
-
Status: In Progress
-
Resolution: Unresolved
-
Fix Version/s: None
-
Component/s: Design Documents
-
Labels:None
-
Team:Architecture
Description
In order to facilitate the independent development of both:
- ObsCore-compliant image metadata services (ObsTAP and SIAv2); and
- a SODA service
we need a near-term-usable definition of how to construct ObsCore obs_publisher_did values for data in a Gen3 repository.
"Near-term" is meant to mean at a minimum "expected to work for Gen3-based HSC datasets" and usable for them for the DP0 data release and the Spring 2020 LSP testing round.
It would probably be good to have an idea for how this would also work for LSST simulated images (e.g., from DESC DC2), should these end up being part of DP0 but this is not a requirement for this ticket.
Nor is it a requirement for this ticket that either
- we devise a solution that is relevant to the LSST operations era or
- the proposed solution continue to work without change throughout the rest of construction and commissioning.
It is OK if a redesign turns out to be needed somewhere along the way. It is more important that we get this off the ground.
The deliverable from this ticket is a written spec, not code.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
In order to facilitate the independent development of both:
# ObsCore-compliant image metadata services (ObsTAP and SIAv2); and # a SODA service we need a near-term-usable definition of how to construct ObsCore {{obs_publisher_did}} values for data in a Gen3 repository. "Near-term" is meant to mean at a minimum "expected to work for Gen3-based HSC datasets" and usable for them for the DP0 data release and the Spring 2020 LSP testing round. It would probably be good to have an idea for how this would also work for LSST simulated images (e.g., from DESC DC2), should these end up being part of DP0 *but this is not a requirement for this ticket*. Nor is it a requirement for this ticket that either # we devise a solution that will work in the LSST operations era *or* # the proposed solution continue to work without change throughout the rest of construction and commissioning. It is OK if a redesign turns out to be needed somewhere along the way. It is more important that we get this off the ground. The deliverable from this ticket is a written spec, not code. |
In order to facilitate the independent development of both:
# ObsCore-compliant image metadata services (ObsTAP and SIAv2); and # a SODA service we need a near-term-usable definition of how to construct ObsCore {{obs_publisher_did}} values for data in a Gen3 repository. "Near-term" is meant to mean at a minimum "expected to work for Gen3-based HSC datasets" and usable for them for the DP0 data release and the Spring 2020 LSP testing round. It would probably be good to have an idea for how this would also work for LSST simulated images (e.g., from DESC DC2), should these end up being part of DP0 *but this is not a requirement for this ticket*. Nor is it a requirement for this ticket that either # we devise a solution that is relevant to the LSST operations era *or* # the proposed solution continue to work without change throughout the rest of construction and commissioning. It is OK if a redesign turns out to be needed somewhere along the way. It is more important that we get this off the ground. The deliverable from this ticket is a written spec, not code. |
| Watchers | Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Tim Jenness [ Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Tim Jenness ] | Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness [ Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness ] |
That's what I was thinking as well. Instead of treating the 3 separately, as in legacy Imgserv, the SODA for Butler Gen3 must start treating the triples as the new Image ID, in LSST context.
| Remote Link | This issue links to "Page (Confluence)" [ 23798 ] |
| Watchers | Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness [ Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness ] | Andy Salnikov, Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness [ Andy Salnikov, Colin Slater, Fritz Mueller, Frossie Economou, Gregory Dubois-Felsmann, Kenny Lo, Kian-Tat Lim, Tim Jenness ] |
I believe we decided in the SODA discussion that the best way to reference a dataset is to use the butler registry UUID for it and not try to store a tuple of dataId, DatasetType, and run collection. The UUID should in theory be portable to other registries so does not need a butler repository root URI either.
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
I will post some more about this by tomorrow to clarify what the issues are.
At a very crude level, though, I believe what is needed is a way to represent the triple of ( BG3 repo, DataId, DatasetType ).