Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21888

Stand up Gen3 version of ap_verify HiTS (2015) CI test

    XMLWordPrintable

    Details

      Description

      This is an umbrella ticket for getting all current Gen2 ap_verify CI functionality working in Gen3.  It includes (at least):

      • defining a Pipeline that includes (and reconfigures as necessary) (DM-21919):
        • the former ApPipeTask, which will be a pipeline rather than a Task in Gen 3 (DM-21939)
        • Many concrete MetricTasks (DM-21885)
      • creating scripts that ingest HiTS test data into a Gen3 repository (DM-21862).
      • creating scripts that upload metric values from a Gen 3 repository (DM-21916).
      • validation that the Gen3 version of the CI package produces results that are either scientifically equivalent or both understandably and acceptably different from the Gen2 version (no separate ticket, yet).

        Attachments

          Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. DM-23983 Cannot apply crosstalk in Gen 3 DECam processing

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-23985 Cannot do linearity corrections in Gen 3 DECam processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21885 Convert concrete MetricTasks to PipelineTasks

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21916 SQuaSH upload of Gen 3 Measurements

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21919 Run ap_verify end-to-end in Gen 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21939 Create Gen 3 AP Pipeline

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29265 Create DECam override file for ap.verify.Gen3DatasetIngestTask

          • Invalid
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29186 Gen3 refloader incorrectly adds filter_flux fields to refcat schema

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29241 Create Gen 3 AP pipeline for DECam

          • Invalid
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            I think I've enumerated everything we need to run ap_verify on the existing datasets in Gen 3. Jim Bosch, am I correct in assuming that the 4 SP on this ticket are reserved for the "validation" ticket you mentioned?

            Show
            krzys Krzysztof Findeisen added a comment - I think I've enumerated everything we need to run ap_verify on the existing datasets in Gen 3. Jim Bosch , am I correct in assuming that the 4 SP on this ticket are reserved for the "validation" ticket you mentioned?
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Yes, that and the integration work ("write and commit a pipeline YAML file") to put it all together.

            Show
            jbosch Jim Bosch added a comment - Yes, that and the integration work ("write and commit a pipeline YAML file") to put it all together.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            Hmm... well, DM-21919 naturally covers that for metrics.

            Should we split off another umbrella ticket for "make ap_pipe runnable from the command line as a Gen 3 pipeline"? That would block DM-21919 and be itself blocked by DM-21862, DM-21874, DM-21886, and possibly DM-21915 (ap_verify datasets are often used to create ap_pipe test repositories, even though it's not their intended use).

            Show
            krzys Krzysztof Findeisen added a comment - - edited Hmm... well, DM-21919 naturally covers that for metrics. Should we split off another umbrella ticket for "make ap_pipe runnable from the command line as a Gen 3 pipeline"? That would block DM-21919 and be itself blocked by DM-21862 , DM-21874 , DM-21886 , and possibly DM-21915 ( ap_verify datasets are often used to create ap_pipe test repositories, even though it's not their intended use).
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Should we split off another umbrella ticket for "make ap_pipe runnable from the command line as a Gen 3 pipeline"?

            .  I'm more than happy for you to organize this in whatever way works best for you; at this point I think your idea of what needs to be done here is pretty complete, in addition to just being better than mine.

            Show
            jbosch Jim Bosch added a comment - Should we split off another umbrella ticket for "make ap_pipe runnable from the command line as a Gen 3 pipeline"? .  I'm more than happy for you to organize this in whatever way works best for you; at this point I think your idea of what needs to be done here is pretty complete, in addition to just being better than mine.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment - - edited

            I've confirmed in tests that DM-29186 (and the conflict it has with our calibrate configs) is the only remaining blocker for this ticket. Crosstalk seems to run without DM-27833 (though it does issue a warning).

            Show
            krzys Krzysztof Findeisen added a comment - - edited I've confirmed in tests that DM-29186 (and the conflict it has with our calibrate configs) is the only remaining blocker for this ticket. Crosstalk seems to run without DM-27833 (though it does issue a warning).
            Hide
            Permalink
            ebellm Eric Bellm added a comment -

            Looks good!

            Show
            ebellm Eric Bellm added a comment - Looks good!

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, Ian Sullivan, Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.