Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3765

For registry-free butler, look up information in related data type.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Story Points:
      12
    • Sprint:
      DB_W16_11
    • Team:
      Data Access and Database

      Description

      Butler reading information (in particular the observation time and length) out of an input dataset's file representation in order to provide rendezvous with calibration data in another repository (that does have a registry). Today, that read is handled by genInputRegistry.py so that the butler doesn't need to look into the dataset itself. If there's no registry, such a read will be necessary.

      The ultimate test is to run processCcd.py (a CmdLineTask that runs Instrument Signature Removal) on a repository that contains raw frames that require more than one set of calibrations and have it pick up the correct ones.  That would have to be condensed down into a unit test.

        Attachments

          Issue Links

            Activity

            No builds found.
            npease Nate Pease [X] (Inactive) created issue -
            npease Nate Pease [X] (Inactive) made changes -
            Field Original Value New Value
            Epic Link DM-2404 [ 16718 ]
            npease Nate Pease [X] (Inactive) made changes -
            Assignee Nate Pease [ npease ]
            npease Nate Pease [X] (Inactive) made changes -
            Team Data Access and Database [ 10204 ]
            jbecla Jacek Becla made changes -
            Sprint DB_W16_10 [ 170 ] DB_W16_11 [ 171 ]
            Hide
            npease Nate Pease [X] (Inactive) added a comment -

            WRT DecamMapper.paf,
            the call: butler.get(flat, dataid(visit=n, ccdnum=m)) will look up data in the
            'flat' dataset type. In the case where the information is not available, need
            to consult the 'reference' dataset type, and look it up there, like so:
            1. dataid is used to generate a location using the raw dataset type. (because it
            doesn't exist in flat, and we wish policy:reference was "raw", but actually it's
            raw_visit which is a subset of the raw table that is optimized to go faster but
            is different.
            2. generate the location, it should be a fits file. in this case it has a bracket
            extension. call lsst.afw.image.readmetadata (or similar in astroypy) on the
            location (with brackets or not) and it will get the right fits header.
            3. depending on the camera, the header may need to be converted (numeric to string
            or vice versa). so we need a hook for the mapper subclass to be able to do
            something (transform) the header value.
            4. then it gets fed into the existing calibrationMapping subclass that does the
            query.
            CalibrationMapping gets the data via the registry (that I need to get via
            metadata) in lookup, at line 348:
            lookups = self.refRegistry.executeQuery(columns, self.reference,
            where, None, values)
            it passes it to Mapping.lookup and THAT uses the validity lookup - using
            self.obsTimeName and self.range which is set in the CalibrationMapping ctor,
            based on the policy.
            to create the test:
            1. modify the policy to look more like the obs_decam policy:
            subpolicy needs template, validTime, etc
            policy would ask for the MJD-OBS - the julian date since (a very long time ago),
            it's modified because it's had a (fixed, known) number subtracted from it.
            or, try using https://github.com/lsst/obs_cfht

            Show
            npease Nate Pease [X] (Inactive) added a comment - WRT DecamMapper.paf, the call: butler.get(flat, dataid(visit=n, ccdnum=m)) will look up data in the 'flat' dataset type. In the case where the information is not available, need to consult the 'reference' dataset type, and look it up there, like so: 1. dataid is used to generate a location using the raw dataset type. (because it doesn't exist in flat, and we wish policy:reference was "raw", but actually it's raw_visit which is a subset of the raw table that is optimized to go faster but is different. 2. generate the location, it should be a fits file. in this case it has a bracket extension. call lsst.afw.image.readmetadata (or similar in astroypy) on the location (with brackets or not) and it will get the right fits header. 3. depending on the camera, the header may need to be converted (numeric to string or vice versa). so we need a hook for the mapper subclass to be able to do something (transform) the header value. 4. then it gets fed into the existing calibrationMapping subclass that does the query. CalibrationMapping gets the data via the registry (that I need to get via metadata) in lookup, at line 348: lookups = self.refRegistry.executeQuery(columns, self.reference, where, None, values) it passes it to Mapping.lookup and THAT uses the validity lookup - using self.obsTimeName and self.range which is set in the CalibrationMapping ctor, based on the policy. to create the test: 1. modify the policy to look more like the obs_decam policy: subpolicy needs template, validTime, etc policy would ask for the MJD-OBS - the julian date since (a very long time ago), it's modified because it's had a (fixed, known) number subtracted from it. or, try using https://github.com/lsst/obs_cfht
            npease Nate Pease [X] (Inactive) made changes -
            Link This issue is duplicated by DM-4140 [ DM-4140 ]
            npease Nate Pease [X] (Inactive) made changes -
            Summary Make butler read information from an input dataset's file representation in order to provide rendezvous with calibration data in another repository. For registry-free butler, look up information in related data type.
            jbecla Jacek Becla made changes -
            Link This issue relates to DM-3591 [ DM-3591 ]
            npease Nate Pease [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            ktl Kian-Tat Lim added a comment - - edited

            Robert Lupton says in HipChat that we should start with just the "standard" TIME-OBS header (which is actually a combination of DATE-OBS and TIME-OBS only if present) to do calibration rendezvous, adding capabilities for additional field names later.

            See http://heasarc.gsfc.nasa.gov/docs/fcg/common_dict.html

            Show
            ktl Kian-Tat Lim added a comment - - edited Robert Lupton says in HipChat that we should start with just the "standard" TIME-OBS header (which is actually a combination of DATE-OBS and TIME-OBS only if present) to do calibration rendezvous, adding capabilities for additional field names later. See http://heasarc.gsfc.nasa.gov/docs/fcg/common_dict.html
            Hide
            tjenness Tim Jenness added a comment -

            There is a standard (without quotes) for time headers: http://dx.doi.org/10.1051/0004-6361/201424653

            That suggests DATE-BEG and DATE-END (or MJD equivalents) and many other keywords. I think what we really need is an internal metadata standard and a header translation infrastructure (see e.g. https://github.com/Starlink/perl-Astro-FITS-HdrTrans)

            Show
            tjenness Tim Jenness added a comment - There is a standard (without quotes) for time headers: http://dx.doi.org/10.1051/0004-6361/201424653 That suggests DATE-BEG and DATE-END (or MJD equivalents) and many other keywords. I think what we really need is an internal metadata standard and a header translation infrastructure (see e.g. https://github.com/Starlink/perl-Astro-FITS-HdrTrans )
            jbecla Jacek Becla made changes -
            Sprint DB_W16_11 [ 171 ] DB_W16_11, DB_W16_12 [ 171, 172 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            jbecla Jacek Becla made changes -
            Sprint DB_W16_11, DB_W16_12 [ 171, 172 ] DB_W16_11 [ 171 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Sprint DB_W16_11 [ 171 ] DB_W16_11, DB_W16_02 [ 171, 179 ]
            npease Nate Pease [X] (Inactive) made changes -
            Sprint DB_W16_11, DB_W16_02 [ 171, 179 ] DB_W16_11 [ 171 ]
            npease Nate Pease [X] (Inactive) made changes -
            Rank Ranked lower
            jbecla Jacek Becla made changes -
            Epic Link DM-2404 [ 16718 ] DM-4341 [ 21422 ]
            npease Nate Pease [X] (Inactive) made changes -
            Epic Link DM-4341 [ 21422 ] DM-4340 [ 21421 ]
            fritzm Fritz Mueller made changes -
            Epic Link DM-4340 [ 21421 ] DM-10678 [ 32630 ]
            fritzm Fritz Mueller made changes -
            Rank Ranked higher
            fritzm Fritz Mueller made changes -
            Rank Ranked higher
            fritzm Fritz Mueller made changes -
            Epic Link DM-10678 [ 32630 ] DM-12759 [ 36358 ]
            fritzm Fritz Mueller made changes -
            Assignee Nate Pease [ npease ]
            Hide
            Parejkoj John Parejko added a comment -

            Can this be closed `won't fix`?

            Show
            Parejkoj John Parejko added a comment - Can this be closed `won't fix`?
            Hide
            tjenness Tim Jenness added a comment -

            I'm going to go through all the butler/daf_persistence tickets when daf_butler takes over.

            Show
            tjenness Tim Jenness added a comment - I'm going to go through all the butler/daf_persistence tickets when daf_butler takes over.
            Hide
            tjenness Tim Jenness added a comment -

            Gen2 is dead.

            Show
            tjenness Tim Jenness added a comment - Gen2 is dead.
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              npease Nate Pease [X] (Inactive)
              Watchers:
              John Parejko, Kian-Tat Lim, Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.