Get all useful dataId keys

XMLWordPrintable

Details

• Type: Story
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Data Access and Database

Description

We usually want the full range of available dataId keys to be available when given a dataId, so much so that this hack, which started in obs_subaru, is slowly finding its way to other mappers:

  # Ensure each dataset type of interest knows about the full range of keys available from the registry  keys = {'field': str,  'visit': int,  'filter': str,  'ccd': int,  'dateObs': str,  'taiObs': str,  'expTime': float,  'pointing': int,  }  for name in ("raw",  # processCcd outputs  "postISRCCD", "calexp", "postISRCCD", "src", "icSrc", "icMatch", "icMatchFull",  "srcMatch", "srcMatchFull",  # forcedPhot outputs  "forced_src",  # Warp  "coaddTempExp",  ):  self.mappings[name].keyDict.update(keys) 

Please make this a standard feature.

Activity

Hide
Kian-Tat Lim added a comment -

Why should there be a need for all keys to be provided for all dataset types? And it actually can't be all dataset types or all keys – it has to be just those that are aligned with raw/calexp images. (Coadds, for example, shouldn't have all of those keys). Making this generic will require identifying which dataset types are raw/calexp-derived and which keys apply to those dataset types.

I worry that the need for this means that some unusual use is being made of dataId keys and that this is thus just a workaround for a more fundamental problem.

Show
Kian-Tat Lim added a comment - Why should there be a need for all keys to be provided for all dataset types? And it actually can't be all dataset types or all keys – it has to be just those that are aligned with raw/calexp images. (Coadds, for example, shouldn't have all of those keys). Making this generic will require identifying which dataset types are raw/calexp-derived and which keys apply to those dataset types. I worry that the need for this means that some unusual use is being made of dataId keys and that this is thus just a workaround for a more fundamental problem.
Hide
Robert Lupton added a comment -

I agree that we don't need all keys for e.g. coadds, but we do need this functionality for CCD-like datasets.

The mapper entries have a field tables: raw – was this supposed to provide this functionality?

Show
Robert Lupton added a comment - I agree that we don't need all keys for e.g. coadds, but we do need this functionality for CCD-like datasets. The mapper entries have a field tables: raw – was this supposed to provide this functionality?
Hide
Hsin-Fang Chiang added a comment -

Notes from last week:

One way to reproduce and demostrate what was wanted originally in this ticket is to processCcd any HSC data with the id proposal. proposal is a column in the HSC sqlite3 registry but is not part of the raw template nor added to the keys by this hack. processCcd.py hsc/repo --id proposal=o15167 --rerun demo gives:

 processCcd.py: error: "Unrecognized ID key 'proposal'; valid keys are: [u'ccd', u'dateObs', 'expTime', u'field', u'filter', u'pointing', 'taiObs', u'visit']" 

The error comes from pipe/base/argumentParser.py

Robert Lupton got a fix for that; I pushed it to the branch u/hfc/DM-5902 for now so it doesn't disappear.

Show
Hsin-Fang Chiang added a comment - Notes from last week: One way to reproduce and demostrate what was wanted originally in this ticket is to processCcd any HSC data with the id proposal . proposal is a column in the HSC sqlite3 registry but is not part of the raw template nor added to the keys by this hack. processCcd.py hsc/repo --id proposal=o15167 --rerun demo gives: processCcd.py: error: "Unrecognized ID key 'proposal'; valid keys are: [u'ccd', u'dateObs', 'expTime', u'field', u'filter', u'pointing', 'taiObs', u'visit']" The error comes from pipe/base/argumentParser.py Robert Lupton got a fix for that; I pushed it to the branch u/hfc/ DM-5902 for now so it doesn't disappear.
Hide
Hsin-Fang Chiang added a comment -

Robert's patch in my last comment has been merged to master in DM-10221. (so I'm deleting the u/hfc/DM-5902 branch.) Using w_2017_16 to run the same command no loger give the above error.

I'm not sure if there are other use cases in need of the "hack" in the descriptions, so I don't dare to propose removing the hack in all obs packages yet.

Show
Hsin-Fang Chiang added a comment - Robert's patch in my last comment has been merged to master in DM-10221 . (so I'm deleting the u/hfc/ DM-5902 branch.) Using w_2017_16 to run the same command no loger give the above error. I'm not sure if there are other use cases in need of the "hack" in the descriptions, so I don't dare to propose removing the hack in all obs packages yet.

People

• Assignee:
Kian-Tat Lim
Reporter:
Paul Price
Watchers:
Colin Slater, Hsin-Fang Chiang, John Parejko, John Swinbank, Kian-Tat Lim, Merlin Fisher-Levine, Paul Price, Robert Lupton