Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15142

Create CI-sized DECam Dataset

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP F18-2, AP F18-3, AP F18-4
    • Team:
      Alert Production

      Description

      Create a dataset (in the Reiss sense) that is a subset of ap_verify_hits2015 for use in CI. The dataset must contain multiple visits to the same location in order to test source association, and should have at most ~10 images. See discussion on #dm for more details.

      In addition to containing only a subset of raw images, the new dataset should contain only those templates that are necessary to process the subset. Unfortunately the only way I know of to figure out which templates get used is to run the pipeline and parse the log output.

        Attachments

          Issue Links

            Activity

            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-14431 [ 80156 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-13970 [ DM-13970 ]
            krzys Krzysztof Findeisen made changes -
            Risk Score 0
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            krzys Krzysztof Findeisen made changes -
            Rank Ranked higher
            krzys Krzysztof Findeisen made changes -
            Description Create a dataset (in the Reiss sense) that is a subset of {{ap_verify_hits2015}} for use in CI. The dataset must contain multiple visits to the same location in order to test source association, and by my estimates it should have at most 50-60 images. See discussion on [#dm|https://lsstc.slack.com/archives/C69NDF3BJ/p1531417963000276] for more details.

            In addition to containing only a subset of raw images, the new dataset should contain only those templates that are necessary to process the subset. Unfortunately the only way I know of to figure out which templates get used is to run the pipeline and parse the log output.
            Create a dataset (in the Reiss sense) that is a subset of {{ap_verify_hits2015}} for use in CI. The dataset must contain multiple visits to the same location in order to test source association, and by my estimates it should have at most 50-60 images. See [discussion on #dm|https://lsstc.slack.com/archives/C69NDF3BJ/p1531417963000276] for more details.

            In addition to containing only a subset of raw images, the new dataset should contain only those templates that are necessary to process the subset. Unfortunately the only way I know of to figure out which templates get used is to run the pipeline and parse the log output.
            krzys Krzysztof Findeisen made changes -
            Description Create a dataset (in the Reiss sense) that is a subset of {{ap_verify_hits2015}} for use in CI. The dataset must contain multiple visits to the same location in order to test source association, and by my estimates it should have at most 50-60 images. See [discussion on #dm|https://lsstc.slack.com/archives/C69NDF3BJ/p1531417963000276] for more details.

            In addition to containing only a subset of raw images, the new dataset should contain only those templates that are necessary to process the subset. Unfortunately the only way I know of to figure out which templates get used is to run the pipeline and parse the log output.
            Create a dataset (in the Reiss sense) that is a subset of {{ap_verify_hits2015}} for use in CI. The dataset must contain multiple visits to the same location in order to test source association, and should have at most ~10 images. See [discussion on #dm|https://lsstc.slack.com/archives/C69NDF3BJ/p1531417963000276] for more details.

            In addition to containing only a subset of raw images, the new dataset should contain only those templates that are necessary to process the subset. Unfortunately the only way I know of to figure out which templates get used is to run the pipeline and parse the log output.
            krzys Krzysztof Findeisen made changes -
            Comment [ I propose the following six-image dataset: HiTS visits 411420, 419802, and 412518 (all for HiTS field {{Blind15A_40}}) and CCDs 52 and 56 (these correspond to CCDs N21 and N25, which are adjacent but at different radii). That should let us cover as many bases as possible while sticking to a minutes processing timescale. ]
            krzys Krzysztof Findeisen made changes -
            Attachment overlap.png [ 33433 ]
            krzys Krzysztof Findeisen made changes -
            Risk Score 0 1
            krzys Krzysztof Findeisen made changes -
            Attachment overlap.png [ 33434 ]
            krzys Krzysztof Findeisen made changes -
            Attachment overlap.png [ 33433 ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Risk Score 1 0
            krzys Krzysztof Findeisen made changes -
            Component/s Continuous Integration [ 10711 ]
            krzys Krzysztof Findeisen made changes -
            Reviewers Simon Krughoff [ krughoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-15224 [ DM-15224 ]
            abh Andy Hanushevsky made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Done [ 10002 ] In Review [ 10004 ]
            swinbank John Swinbank made changes -
            Sprint AP F18-2 [ 747 ] AP F18-2, AP F18-3 [ 747, 748 ]
            swinbank John Swinbank made changes -
            Sprint AP F18-2, AP F18-3 [ 747, 748 ] AP F18-2, AP F18-3, AP F18-4 [ 747, 748, 749 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue has to be done before DM-15742 [ DM-15742 ]
            krzys Krzysztof Findeisen made changes -
            Status In Review [ 10004 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            krughoff Simon Krughoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Simon Krughoff
                Watchers:
                Eric Bellm, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel