Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5902

Get all useful dataId keys

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: daf_persistence, obs_base
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      We usually want the full range of available dataId keys to be available when given a dataId, so much so that this hack, which started in obs_subaru, is slowly finding its way to other mappers:

              # Ensure each dataset type of interest knows about the full range of keys available from the registry
              keys = {'field': str,
                      'visit': int,
                      'filter': str,
                      'ccd': int,
                      'dateObs': str,
                      'taiObs': str,
                      'expTime': float,
                      'pointing': int,
                      }
              for name in ("raw",
                           # processCcd outputs
                           "postISRCCD", "calexp", "postISRCCD", "src", "icSrc", "icMatch", "icMatchFull",
                           "srcMatch", "srcMatchFull",
                           # forcedPhot outputs
                           "forced_src",
                           # Warp
                           "coaddTempExp",
                           ):
                  self.mappings[name].keyDict.update(keys)
      

      Please make this a standard feature.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            Why should there be a need for all keys to be provided for all dataset types? And it actually can't be all dataset types or all keys – it has to be just those that are aligned with raw/calexp images. (Coadds, for example, shouldn't have all of those keys). Making this generic will require identifying which dataset types are raw/calexp-derived and which keys apply to those dataset types.

            I worry that the need for this means that some unusual use is being made of dataId keys and that this is thus just a workaround for a more fundamental problem.

            Show
            ktl Kian-Tat Lim added a comment - Why should there be a need for all keys to be provided for all dataset types? And it actually can't be all dataset types or all keys – it has to be just those that are aligned with raw/calexp images. (Coadds, for example, shouldn't have all of those keys). Making this generic will require identifying which dataset types are raw/calexp-derived and which keys apply to those dataset types. I worry that the need for this means that some unusual use is being made of dataId keys and that this is thus just a workaround for a more fundamental problem.
            Hide
            rhl Robert Lupton added a comment -

            I agree that we don't need all keys for e.g. coadds, but we do need this functionality for CCD-like datasets.

            The mapper entries have a field tables: raw – was this supposed to provide this functionality?

            Show
            rhl Robert Lupton added a comment - I agree that we don't need all keys for e.g. coadds, but we do need this functionality for CCD-like datasets. The mapper entries have a field tables: raw – was this supposed to provide this functionality?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Notes from last week:

            One way to reproduce and demostrate what was wanted originally in this ticket is to processCcd any HSC data with the id proposal. proposal is a column in the HSC sqlite3 registry but is not part of the raw template nor added to the keys by this hack. processCcd.py hsc/repo --id proposal=o15167 --rerun demo gives:

            processCcd.py: error: "Unrecognized ID key 'proposal'; valid keys are: [u'ccd', u'dateObs', 'expTime', u'field', u'filter', u'pointing', 'taiObs', u'visit']"
            

            The error comes from pipe/base/argumentParser.py

            Robert Lupton got a fix for that; I pushed it to the branch u/hfc/DM-5902 for now so it doesn't disappear.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Notes from last week: One way to reproduce and demostrate what was wanted originally in this ticket is to processCcd any HSC data with the id proposal . proposal is a column in the HSC sqlite3 registry but is not part of the raw template nor added to the keys by this hack. processCcd.py hsc/repo --id proposal=o15167 --rerun demo gives: processCcd.py: error: "Unrecognized ID key 'proposal'; valid keys are: [u'ccd', u'dateObs', 'expTime', u'field', u'filter', u'pointing', 'taiObs', u'visit']" The error comes from pipe/base/argumentParser.py Robert Lupton got a fix for that; I pushed it to the branch u/hfc/ DM-5902 for now so it doesn't disappear.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Robert's patch in my last comment has been merged to master in DM-10221. (so I'm deleting the u/hfc/DM-5902 branch.) Using w_2017_16 to run the same command no loger give the above error.

            I'm not sure if there are other use cases in need of the "hack" in the descriptions, so I don't dare to propose removing the hack in all obs packages yet.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Robert's patch in my last comment has been merged to master in DM-10221 . (so I'm deleting the u/hfc/ DM-5902 branch.) Using w_2017_16 to run the same command no loger give the above error. I'm not sure if there are other use cases in need of the "hack" in the descriptions, so I don't dare to propose removing the hack in all obs packages yet.

              People

              • Assignee:
                ktl Kian-Tat Lim
                Reporter:
                price Paul Price
                Watchers:
                Colin Slater, Hsin-Fang Chiang, John Parejko, John Swinbank, Kian-Tat Lim, Merlin Fisher-Levine, Paul Price, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel