Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32389

Create refcat management script for ap_verify datasets

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None

      Description

      Based on the results of DM-31281, create scripts for each ap_verify dataset that can regenerate the refcats to be used with that dataset's science processing. Consult with John Parejko on the most appropriate way to do this (e.g., generating new refcats like the existing ap_verify_hits2015/scripts/gaia_HiTS_2015.py does, or simply grabbing a subset of the /repo/main refcats). I am assuming that the solution to DM-31681 will not need special refcat support, as template generation will take place within a development repository.

      This issue covers making scripts for ap_verify_ci_hits2015 and ap_verify_hits2015, but it may also cover ap_verify_ci_cosmos_pdr2 if no dataset-specific knowledge is needed to do so. Otherwise, support for ap_verify_ci_cosmos_pdr2 can wait until DM-29857 finalizes the complete workflow.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29857 Create pure Gen 3 dataset management scripts for ap_verify datasets

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29543 Gen3 refcat converter

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Link This issue blocks DM-29857 [ DM-29857 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-25943 [ DM-25943 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-29543 [ DM-29543 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue is blocked by DM-25943 [ DM-25943 ]
            krzys Krzysztof Findeisen made changes -
            Description Based on the results of DM-31281, create scripts for each ap_verify dataset that can regenerate the refcats to be used with that dataset's science processing. Consult with [~Parejkoj] on the most appropriate way to do this (e.g., generating new refcats like the existing {{ap_verify_hits2015/scripts/gaia_HiTS_2015.py} does, or simply grabbing a subset of the {{/repo/main}} refcats). I am assuming that the solution to DM-31681 will not need special refcat support, as template generation will take place within a development repository.

            This issue covers making scripts for {{ap_verify_ci_hits2015}} and {{ap_verify_hits2015}}, but it may also cover {{ap_verify_ci_cosmos_pdr2}} if no dataset-specific knowledge is needed to do so. Otherwise, support for {{ap_verify_ci_cosmos_pdr2}} can wait until DM-29857 finalizes the complete workflow.             		Based on the results of DM-31281, create scripts for each ap_verify dataset that can regenerate the refcats to be used with that dataset's science processing. Consult with [~Parejkoj] on the most appropriate way to do this (e.g., generating new refcats like the existing {{ap_verify_hits2015/scripts/gaia_HiTS_2015.py}} does, or simply grabbing a subset of the {{/repo/main}} refcats). I am assuming that the solution to DM-31681 will not need special refcat support, as template generation will take place within a development repository.

            This issue covers making scripts for {{ap_verify_ci_hits2015}} and {{ap_verify_hits2015}}, but it may also cover {{ap_verify_ci_cosmos_pdr2}} if no dataset-specific knowledge is needed to do so. Otherwise, support for {{ap_verify_ci_cosmos_pdr2}} can wait until DM-29857 finalizes the complete workflow.
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Description Based on the results of DM-31281, create scripts for each ap_verify dataset that can regenerate the refcats to be used with that dataset's science processing. Consult with [~Parejkoj] on the most appropriate way to do this (e.g., generating new refcats like the existing {{ap_verify_hits2015/scripts/gaia_HiTS_2015.py}} does, or simply grabbing a subset of the {{/repo/main}} refcats). I am assuming that the solution to DM-31681 will not need special refcat support, as template generation will take place within a development repository.

            This issue covers making scripts for {{ap_verify_ci_hits2015}} and {{ap_verify_hits2015}}, but it may also cover {{ap_verify_ci_cosmos_pdr2}} if no dataset-specific knowledge is needed to do so. Otherwise, support for {{ap_verify_ci_cosmos_pdr2}} can wait until DM-29857 finalizes the complete workflow.             		Based on the results of DM-31281, create scripts for each ap_verify dataset that can regenerate the refcats to be used with that dataset's science processing. Consult with [~Parejkoj] on the most appropriate way to do this (e.g., generating new refcats like the existing [{{ap_verify_hits2015/scripts/gaia_HiTS_2015.py}}|https://github.com/lsst/ap_verify_hits2015/blob/master/scripts/gaia_HiTS_2015.py] does, or simply grabbing a subset of the {{/repo/main}} refcats). I am assuming that the solution to DM-31681 will not need special refcat support, as template generation will take place within a development repository.

            This issue covers making scripts for {{ap_verify_ci_hits2015}} and {{ap_verify_hits2015}}, but it may also cover {{ap_verify_ci_cosmos_pdr2}} if no dataset-specific knowledge is needed to do so. Otherwise, support for {{ap_verify_ci_cosmos_pdr2}} can wait until DM-29857 finalizes the complete workflow.
            krzys Krzysztof Findeisen made changes -
            Story Points 4 5
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Thanks for agreeing to review this, John Parejko! The main development took place in ap_verify_hits2015; the other two are basically copies with the repository names and input fields swapped out.

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for agreeing to review this, John Parejko ! The main development took place in ap_verify_hits2015 ; the other two are basically copies with the repository names and input fields swapped out.
            krzys Krzysztof Findeisen made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            sullivan Ian Sullivan made changes -
            Epic Link DM-30514 [ 510188 ] DM-30515 [ 510190 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            I reviewed one of the scripts; those comments apply to the others as well.

            Show
            Parejkoj John Parejko added a comment - I reviewed one of the scripts; those comments apply to the others as well.
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Thanks for the review!

            Show
            krzys Krzysztof Findeisen added a comment - Thanks for the review!

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              John Parejko
              Watchers:
              Ian Sullivan, John Parejko, Krzysztof Findeisen, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.