Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2880

Create transformation task for CalibrateTask outputs

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      The tasks defined in DM-2191 are probably not sufficiently general to handle the icSrc dataset generated by CalibrateTask. An icSrc specific task will need to be defined.

      Jim Bosch is anticipating a CalibrateTask rewrite at some point; we should keep this issue on hold until the plan there is more concrete.

        Attachments

          Issue Links

            Activity

            Hide
            krughoff Simon Krughoff added a comment - - edited

            John Swinbank and Jim Bosch the CalibrateTask rewrite is happening. More specifically, ProcessCcdTask and all its sub-tasks are being reworked to be more modular and to simplify conceptual flow. I don't understand exactly what is being asked for, but it sounds like this should be covered by the requirements and design on clo.

            Can I close this and link it back to the ticket with the refactor implementation work in it, DM-4692?

            Show
            krughoff Simon Krughoff added a comment - - edited John Swinbank and Jim Bosch the CalibrateTask rewrite is happening. More specifically, ProcessCcdTask and all its sub-tasks are being reworked to be more modular and to simplify conceptual flow. I don't understand exactly what is being asked for, but it sounds like this should be covered by the requirements and design on clo . Can I close this and link it back to the ticket with the refactor implementation work in it, DM-4692 ?
            Hide
            jbosch Jim Bosch added a comment -

            This ticket is requesting an afterburner CmdLineTask that transforms icSrc quantities to absolute units (ra/dec, magnitudes). We already have afterburners for src and some coadd datasets. These afterburners are needed before database ingest, since we no longer transform units during ingest.

            So what's required on this ticket in detail would certainly change with the CalibrateTask rewrite, but it still involves functionality that doesn't exist in the old CalibrateTask, and will still be needed after the CalibrateTask rewrite. I'll leave it to you whether to include that work in your other CalibrateTask issues or leave it on this issue.

            Show
            jbosch Jim Bosch added a comment - This ticket is requesting an afterburner CmdLineTask that transforms icSrc quantities to absolute units (ra/dec, magnitudes). We already have afterburners for src and some coadd datasets. These afterburners are needed before database ingest, since we no longer transform units during ingest. So what's required on this ticket in detail would certainly change with the CalibrateTask rewrite, but it still involves functionality that doesn't exist in the old CalibrateTask , and will still be needed after the CalibrateTask rewrite. I'll leave it to you whether to include that work in your other CalibrateTask issues or leave it on this issue.
            Hide
            krughoff Simon Krughoff added a comment -

            I see. I'll leave it hear for now, but I think we'll probably move it to a different epic when all is said and done. On a related, but side note, should we need afterburners for this, or is this something that should exist in the task? I guess I'm wondering if you think this is the final solution for this sort of thing, or should we schedule some work to do more design on this?

            Show
            krughoff Simon Krughoff added a comment - I see. I'll leave it hear for now, but I think we'll probably move it to a different epic when all is said and done. On a related, but side note, should we need afterburners for this, or is this something that should exist in the task? I guess I'm wondering if you think this is the final solution for this sort of thing, or should we schedule some work to do more design on this?
            Hide
            jbosch Jim Bosch added a comment -

            I do find the current system a bit messy. The challenge is that it isn't something we want to run at the same time we run the main task that generates the raw-unit outputs, but it is something that needs to know a bit about the details of the main task, so it's a parallel hiearchy, and it's always a pain to do that without a lot of boilerplate and code duplication. From a user standpoint, I think we'd want these afterburners to appear like different top-level commands that belong to the same task ("processCcd run" vs "processCcd transform", for instance), but I'm not sure from an implementation standpoint whether they should actually be methods of the same class, given that they don't need all the same instance variables.

            Show
            jbosch Jim Bosch added a comment - I do find the current system a bit messy. The challenge is that it isn't something we want to run at the same time we run the main task that generates the raw-unit outputs, but it is something that needs to know a bit about the details of the main task, so it's a parallel hiearchy, and it's always a pain to do that without a lot of boilerplate and code duplication. From a user standpoint, I think we'd want these afterburners to appear like different top-level commands that belong to the same task ("processCcd run" vs "processCcd transform", for instance), but I'm not sure from an implementation standpoint whether they should actually be methods of the same class, given that they don't need all the same instance variables.
            Hide
            krughoff Simon Krughoff added a comment -

            O.K. Thanks for the thoughts. I'll keep this in mind.

            Show
            krughoff Simon Krughoff added a comment - O.K. Thanks for the thoughts. I'll keep this in mind.
            Hide
            yusra Yusra AlSayyad added a comment -

            Superceded by DM-14568

            Show
            yusra Yusra AlSayyad added a comment - Superceded by  DM-14568

              People

              • Assignee:
                Unassigned
                Reporter:
                swinbank John Swinbank
                Watchers:
                Jim Bosch, John Swinbank, Simon Krughoff, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel