Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15843

Clean up details of pipeline task conversion process

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP F18-4, DRP F18-5, DRP F18-6, DRP S19-1, DRP S19-2
    • Team:
      Data Release Production

      Description

      The pipeline task conversion processes involves separating all code which needs access to io from algorithm code. In some cases this is not yet possible (such as with IdFactories) as the middleware necessary for doing this does not yet exist.

      However, to make process on the bulk of the conversion process it is possible to use stand in code to allow conversion and testing in place of a final solution.

      This ticket will cover the work of going back to all the places where this occurs and fixing it up to a final solution once the required middleware exists.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17689 Expose maximum number of bits used by IdFactory

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            nlust Nate Lust created issue -
            nlust Nate Lust made changes -
            Field Original Value New Value
            Epic Link DM-14448 [ 80390 ]
            nlust Nate Lust made changes -
            Risk Score 0
            yusra Yusra AlSayyad made changes -
            Sprint DRP F18-4 [ 774 ] DRP F18-4, DRP F18-5 [ 774, 790 ]
            yusra Yusra AlSayyad made changes -
            Story Points 4
            yusra Yusra AlSayyad made changes -
            Sprint DRP F18-4, DRP F18-5 [ 774, 790 ] DRP F18-4, DRP F18-5, DRP F18-6 [ 774, 790, 811 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F18-4, DRP F18-5, DRP F18-6 [ 774, 790, 811 ] DRP F18-4, DRP F18-5, DRP F18-6, DRP S19-1 [ 774, 790, 811, 822 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-14448 [ 80390 ] DM-16675 [ 235235 ]
            yusra Yusra AlSayyad made changes -
            Sprint DRP F18-4, DRP F18-5, DRP F18-6, DRP S19-1 [ 774, 790, 811, 822 ] DRP F18-4, DRP F18-5, DRP F18-6, DRP S19-1, DRP S19-2 [ 774, 790, 811, 822, 840 ]
            nlust Nate Lust made changes -
            Assignee Nate Lust [ nlust ] Jim Bosch [ jbosch ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I'm starting this now just to fix the IdFactory issue, for which support landed in daf_butler on DM-14821.  There may be some miscellaneous cleanups if I spot anything we've since decided to do differently.

            Show
            jbosch Jim Bosch added a comment - I'm starting this now just to fix the IdFactory issue, for which support landed in daf_butler on DM-14821 .  There may be some miscellaneous cleanups if I spot anything we've since decided to do differently.
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Nate Lust, after all of the excitement over the weekend I forgot that I had basically finished this ticket on Friday (actually, I was waiting for MergeMeasurements to land first, and today realized it didn't actually need any fixing).

            In any case, it is now ready for review, and I'll let you decide how you want to stage merging and testing it.

            Changes are all in pipe_tasks, and just involve calling packDataId in a few different places: https://github.com/lsst/pipe_tasks/pull/267.

            Show
            jbosch Jim Bosch added a comment - Nate Lust , after all of the excitement over the weekend I forgot that I had basically finished this ticket on Friday (actually, I was waiting for MergeMeasurements to land first, and today realized it didn't actually need any fixing). In any case, it is now ready for review, and I'll let you decide how you want to stage merging and testing it. Changes are all in pipe_tasks, and just involve calling packDataId in a few different places: https://github.com/lsst/pipe_tasks/pull/267 .
            jbosch Jim Bosch made changes -
            Reviewers Nate Lust [ nlust ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            "Only" took three more runs, but I've finally finished testing this with the full suite of PipelineTasks; it all now runs to completion.  I've also moved Nate Lust's changes to meas_base from tickets/DM-17496 to this ticket:

            https://github.com/lsst/meas_base/pull/136

            I've reviewed those, and my only comment is that the commits should be squashed.

             

            Show
            jbosch Jim Bosch added a comment - "Only" took three more runs, but I've finally finished testing this with the full suite of PipelineTasks; it all now runs to completion.  I've also moved Nate Lust 's changes to meas_base from tickets/ DM-17496 to this ticket: https://github.com/lsst/meas_base/pull/136 I've reviewed those, and my only comment is that the commits should be squashed.  
            Hide
            Permalink
            nlust Nate Lust added a comment -

            Consider changing the magic number 64 in pipe tasks

            Show
            nlust Nate Lust added a comment - Consider changing the magic number 64 in pipe tasks
            nlust Nate Lust made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-17689 [ DM-17689 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-16675 [ 235235 ] DM-16771 [ 235477 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              nlust Nate Lust
              Reviewers:
              Nate Lust
              Watchers:
              Jim Bosch, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.