Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15843

Clean up details of pipeline task conversion process

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      DRP F18-4, DRP F18-5, DRP F18-6, DRP S19-1, DRP S19-2
    • Team:
      Data Release Production

      Description

      The pipeline task conversion processes involves separating all code which needs access to io from algorithm code. In some cases this is not yet possible (such as with IdFactories) as the middleware necessary for doing this does not yet exist.

      However, to make process on the bulk of the conversion process it is possible to use stand in code to allow conversion and testing in place of a final solution.

      This ticket will cover the work of going back to all the places where this occurs and fixing it up to a final solution once the required middleware exists.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            I'm starting this now just to fix the IdFactory issue, for which support landed in daf_butler on DM-14821.  There may be some miscellaneous cleanups if I spot anything we've since decided to do differently.

            Show
            jbosch Jim Bosch added a comment - I'm starting this now just to fix the IdFactory issue, for which support landed in daf_butler on DM-14821 .  There may be some miscellaneous cleanups if I spot anything we've since decided to do differently.
            Hide
            jbosch Jim Bosch added a comment -

            Nate Lust, after all of the excitement over the weekend I forgot that I had basically finished this ticket on Friday (actually, I was waiting for MergeMeasurements to land first, and today realized it didn't actually need any fixing).

            In any case, it is now ready for review, and I'll let you decide how you want to stage merging and testing it.

            Changes are all in pipe_tasks, and just involve calling packDataId in a few different places: https://github.com/lsst/pipe_tasks/pull/267.

            Show
            jbosch Jim Bosch added a comment - Nate Lust , after all of the excitement over the weekend I forgot that I had basically finished this ticket on Friday (actually, I was waiting for MergeMeasurements to land first, and today realized it didn't actually need any fixing). In any case, it is now ready for review, and I'll let you decide how you want to stage merging and testing it. Changes are all in pipe_tasks, and just involve calling packDataId in a few different places: https://github.com/lsst/pipe_tasks/pull/267 .
            Hide
            jbosch Jim Bosch added a comment -

            "Only" took three more runs, but I've finally finished testing this with the full suite of PipelineTasks; it all now runs to completion.  I've also moved Nate Lust's changes to meas_base from tickets/DM-17496 to this ticket:

            https://github.com/lsst/meas_base/pull/136

            I've reviewed those, and my only comment is that the commits should be squashed.

             

            Show
            jbosch Jim Bosch added a comment - "Only" took three more runs, but I've finally finished testing this with the full suite of PipelineTasks; it all now runs to completion.  I've also moved Nate Lust 's changes to meas_base from tickets/ DM-17496 to this ticket: https://github.com/lsst/meas_base/pull/136 I've reviewed those, and my only comment is that the commits should be squashed.  
            Hide
            nlust Nate Lust added a comment -

            Consider changing the magic number 64 in pipe tasks

            Show
            nlust Nate Lust added a comment - Consider changing the magic number 64 in pipe tasks

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                nlust Nate Lust
                Reviewers:
                Nate Lust
                Watchers:
                Jim Bosch, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel