Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38688

Implement more compressed data ID packing for Rubin instruments

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      Ops Pipelines 2023
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The packed integer forms of visit+detector data IDs for Rubin instruments do not leave enough bits for per-data ID source counters. Implement a custom packer for all Rubin instruments that solves this problem at the expense of making it impossible for humans to read off the day_obs and seq_num from (at least) the decimal form of the packed integer.

      This builds on DM-31924, which provides the hooks for obs packages to override the data ID packing algorithm that this ticket will make use of.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31924 Design and implement configurable data ID packing system

          • Done
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. DM-38871 characterizeImage cannot deal with one-to-one visit ids

          • Invalid
          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. DM-38871 characterizeImage cannot deal with one-to-one visit ids

          • Invalid

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-917 Data ID -> integer packing for Rubin instruments and related code removals

          • Implemented

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            For my future reference, here's

            Show
            jbosch Jim Bosch added a comment - For my future reference, here's an accounting of the ranges for the components of exposure.id values: https://github.com/LSSTDESC/imSim/issues/329 how we make a visit.id from an exposure.id for the one-to-one visit that corresponds to the first snap in a multi-snap sequence: https://github.com/lsst/Tbs_base/blob/24faba0429a517bc7f734165b8ec1b7adc5a5859/python/lsst/obs/base/defineVisits.py#L940 how we compute exposure.id values from their components: https://github.com/lsst/obs_lsst/blob/b89ecca122f47bdf3ad78862f5e34321d5cc500c/python/lsst/obs/lsst/translators/lsst.py#L359-L411 . This is changed for UCDCam, TS3, and TS8, and I as a result I am inclined to leave those as they are unless someone says we need more bits for those too.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            James Chiang, can I ask you to review this?

            Only PR is https://github.com/lsst/obs_lsst/pull/446

             

            Show
            jbosch Jim Bosch added a comment - James Chiang , can I ask you to review this? Only PR is https://github.com/lsst/obs_lsst/pull/446  
            Hide
            Permalink
            jchiang James Chiang added a comment -

            Yes, I will have a look.

            Show
            jchiang James Chiang added a comment - Yes, I will have a look.
            Hide
            Permalink
            jchiang James Chiang added a comment -

            I approved at the PR, and ran SFP on imSim 2.0 data that produces the "Next ID '1024' is too large" errors in the calibrate task using w_2023_17.   Everything ran fine using the tickets/DM-38688 branch.

            Show
            jchiang James Chiang added a comment - I approved at the PR, and ran SFP on imSim 2.0 data that produces the "Next ID '1024' is too large" errors in the calibrate task using w_2023_17.   Everything ran fine using the tickets/ DM-38688 branch.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              James Chiang
              Watchers:
              Eric Bellm, James Chiang, Jim Bosch, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.