Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15503

Improve raw data handling in gen2convert

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      DM-15189 improves support for raw data by adding new Formatters that provide the adaptation logic provided by std_ hooks in Gen2.  But those Formatters aren't used in gen2convert yet, because gen2convert doesn't see raw data as special.  As a result, loading raw with a Gen3 Butler pointed at a converted repo just gets you a DecoratedImage, not an Exposure.

      The simplest fix is probably to have gen2convert just run raw ingest instead of transferring raws via the usual mechanism.  The tricky part of that is to provide a way for gen2convert to identify the RawIngestTask specialization for a particular camera.  We need to solve that problem anyway, but I'd hoped to get some CameraGeom and package organization problems in better shape before tackling it.  An alternative may be to just force the user to provide the task name in the gen2convert configuration for now.

      Tentatively assigning this to me since I have a pretty good idea in my head of how to approach this, but Christopher Waters is also a possibility.

       

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue relates to DM-15189 [ DM-15189 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Epic Link DM-16675 [ 235235 ]
            jbosch Jim Bosch made changes -
            Story Points 4 2
            Description DM-15189 improves support for raw data by adding new Formatters that provide the adaptation logic provided by {{std_}} hooks in Gen2.  But those Formatters aren't used in {{gen2convert}} yet, because {{gen2convert}} doesn't see raw data as special.  As a result, loading {{raw}} with a Gen3 Butler pointed at a converted repo just gets you a {{DecoratedImage}}, not an {{Exposure}}.

            The simplest fix is probably to have {{gen2convert}} just run raw ingest instead of transferring raws via the usual mechanism.  The tricky part of that is to provide a way for {{gen2convert}} to identify the {{RawIngestTask}} specialization for a particular camera.  We need to solve that problem anyway, but I'd hoped to get some CameraGeom and package organization problems in better shape before tackling it.  An alternative may be to just force the user to provide the task name in the {{gen2convert}} configuration for now.

            Tentatively assigning this to me since I have a pretty good idea in my head of how to approach this, but [~czw] is also a possibility.

            [~mbutler], this ticket may interest you as it's the reason you couldn't use {{raw}} as the input for your dummy {{PipelineTask}} last week.
            DM-15189 improves support for raw data by adding new Formatters that provide the adaptation logic provided by {{std_}} hooks in Gen2.  But those Formatters aren't used in {{gen2convert}} yet, because {{gen2convert}} doesn't see raw data as special.  As a result, loading {{raw}} with a Gen3 Butler pointed at a converted repo just gets you a {{DecoratedImage}}, not an {{Exposure}}.

            The simplest fix is probably to have {{gen2convert}} just run raw ingest instead of transferring raws via the usual mechanism.  The tricky part of that is to provide a way for {{gen2convert}} to identify the {{RawIngestTask}} specialization for a particular camera.  We need to solve that problem anyway, but I'd hoped to get some CameraGeom and package organization problems in better shape before tackling it.  An alternative may be to just force the user to provide the task name in the {{gen2convert}} configuration for now.

            Tentatively assigning this to me since I have a pretty good idea in my head of how to approach this, but [~czw] is also a possibility.

             
            yusra Yusra AlSayyad made changes -
            Link This issue blocks DM-16467 [ DM-16467 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-16803 [ DM-16803 ]
            jbosch Jim Bosch made changes -
            Reviewers Christopher Waters [ cwaters ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Link This issue is triggering DM-16829 [ DM-16829 ]
            czw Christopher Waters made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Christopher Waters
                Watchers:
                Christopher Waters, Jim Bosch
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel