Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2191

Define command line tasks for pre-ingest transformation

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      DM-1903 provided a command line task which would transform a src catalogue into calibrated form. Here, we build on that to provide command line tasks for all source catalogues which will need to be ingested; will include at least deepCoadd_src, goodSeeingCoadd_src, chiSquaredCoadd_src.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Jim Bosch – I fixed things up to be more like your recommendations on my flight yesterday. Unfortunately I won't get chance to finish polishing this up for the next few days, so I'm not putting it out for review, but if you have chance to comment as to whether tickets/DM-2191a on pipe_tasks would be more to your taste that would be very useful.

            On a related note: I would like to produce some tests which actually demonstrate the operation of this code on coadds. However, the contents of obs_test don't seem up to the task, and pipe_task's own testCoadds.py does a lot of magic with its own custom mapper classes – if it's worthwhile investing the time to learn what's going on there I will, but you've warned me off sinking too much time into understanding Butler-related issues in the past. I wondered if you could suggest any alternative routes to quickly put together the equivalent of RunTransformTestCase for coadds.

            Show
            swinbank John Swinbank added a comment - Jim Bosch – I fixed things up to be more like your recommendations on my flight yesterday. Unfortunately I won't get chance to finish polishing this up for the next few days, so I'm not putting it out for review, but if you have chance to comment as to whether tickets/DM-2191a on pipe_tasks would be more to your taste that would be very useful. On a related note: I would like to produce some tests which actually demonstrate the operation of this code on coadds. However, the contents of obs_test don't seem up to the task, and pipe_task 's own testCoadds.py does a lot of magic with its own custom mapper classes – if it's worthwhile investing the time to learn what's going on there I will, but you've warned me off sinking too much time into understanding Butler-related issues in the past. I wondered if you could suggest any alternative routes to quickly put together the equivalent of RunTransformTestCase for coadds.
            Hide
            swinbank John Swinbank added a comment -

            With apologies for the delay, a reworked version of this is now available on tickets/DM-2191 in obs_test and pipe_tasks. Are you happier with this approach?

            Show
            swinbank John Swinbank added a comment - With apologies for the delay, a reworked version of this is now available on tickets/ DM-2191 in obs_test and pipe_tasks . Are you happier with this approach?
            Hide
            jbosch Jim Bosch added a comment -

            This approach looks great, and modulo the minor comments on the new GitHub PR, I think it's ready to merge.

            There are a couple of remaining concerns I don't think should delay this issue, but are worth bringing up now as we may want to spawn some new issues now:

            • We might need a top-level transform task for the "icSrc" dataset, which is produced by CalibrateTask. I'm worried that the tasks here aren't sufficiently general to handle that, but I don't think we should generalize them further to handle it - if that would be necessary, it'd probably be better to create a new CmdLineTask from scratch, as "icSrc" is a very weird bird that involves a SchemaMapper between measurement and actually writing the files to disk. I'm also anticipating a CalibrateTask rewrite at some point, and we might just want to wait to write a transform CmdLineTask until that happens.
            • The multiband stuff Lauren MacArthur is transferring from the HSC side will need some new transform {{CmdLineTask}}s. I'm pretty confident this system will work for those, too, so it's just a matter of writing a few more specializations. We should create issues in the HSC deblender epic for these.
            Show
            jbosch Jim Bosch added a comment - This approach looks great, and modulo the minor comments on the new GitHub PR, I think it's ready to merge. There are a couple of remaining concerns I don't think should delay this issue, but are worth bringing up now as we may want to spawn some new issues now: We might need a top-level transform task for the "icSrc" dataset, which is produced by CalibrateTask . I'm worried that the tasks here aren't sufficiently general to handle that, but I don't think we should generalize them further to handle it - if that would be necessary, it'd probably be better to create a new CmdLineTask from scratch, as "icSrc" is a very weird bird that involves a SchemaMapper between measurement and actually writing the files to disk. I'm also anticipating a CalibrateTask rewrite at some point, and we might just want to wait to write a transform CmdLineTask until that happens. The multiband stuff Lauren MacArthur is transferring from the HSC side will need some new transform {{CmdLineTask}}s. I'm pretty confident this system will work for those, too, so it's just a matter of writing a few more specializations. We should create issues in the HSC deblender epic for these.
            Hide
            swinbank John Swinbank added a comment -

            Thanks for the review. I've addressed your comments and merged to master.

            I've also created the issues DM-2879 and DM-2880 to cover the points you raise above.

            Show
            swinbank John Swinbank added a comment - Thanks for the review. I've addressed your comments and merged to master. I've also created the issues DM-2879 and DM-2880 to cover the points you raise above.
            Hide
            swinbank John Swinbank added a comment -

            Also bumping SPs to reflect the time spent on this issue.

            Show
            swinbank John Swinbank added a comment - Also bumping SPs to reflect the time spent on this issue.

              People

              • Assignee:
                swinbank John Swinbank
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel