Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21915

Support Gen 3 ingestion of ap_verify datasets

    Details

    • Story Points:
      6
    • Sprint:
      AP S20-5 (April), AP S20-6 (May), AP F20-1 (June), AP F20-2 (July)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      lsst.ap.verify.DatasetIngestTask is responsible for ingesting an ap_verify dataset into a Gen 2 repository. Create a second task that starts with the same inputs, but creates a Gen 3 repository. Provide a (command-line?) option in ingest_dataset.py that lets users choose between the two.

      Note that the current dataset format includes a stub Gen 2 repo (possibly redundant in Gen 3) and a Gen 2 templates repo (must be updated or replaced). Changes to this format are within scope of the ticket.

      Some overlap with DM-21862, which wants to use HiTS to run single-frame processing.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Following some discussion started by Meredith Rawls on #dm-naming-things, the Gen 3 repository for non-raws will be called preloaded/. Better to name the data products by what they are, not what they aren't.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Following some discussion started by Meredith Rawls on #dm-naming-things , the Gen 3 repository for non-raws will be called preloaded/ . Better to name the data products by what they are, not what they aren't.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows:

            • The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on DM-24260, along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest.
            • To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is DM-24844.
            • This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
            Show
            krzys Krzysztof Findeisen added a comment - - edited This ticket has suffered a bit from rabbitholeing (though not actual scope creep, I think). To mitigate this, I'm splitting it up as follows: The conversion script that implements the Gen 3 format described above (and an obs_base bug that blocked it) will now go on DM-24260 , along with the updated datasets. I'll put it in review now, even though there's a chance there are bugs that will only be detected on ingest. To add Gen 3 support to ap_verify code I need a Gen 3-compatible test dataset; this is DM-24844 . This ticket will consist only of code changes in ap_verify related to processing the Gen 3 components of ap_verify datasets, plus any bug fixes in the conversion script discovered while developing that code.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786.

            Show
            krzys Krzysztof Findeisen added a comment - Progress report: the key code has been written and tested, but final integration into ingest_dataset.py is blocked on DM-25786 .
            Hide
            krzys Krzysztof Findeisen added a comment -

            I've asked Jim Bosch to review this as an expert on `RawIngestTask`. Changes:

            • ap_verify has the main work
            • obs_base has a bug fix related to transfer configuration
            • ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
            Show
            krzys Krzysztof Findeisen added a comment - I've asked Jim Bosch to review this as an expert on `RawIngestTask`. Changes: ap_verify has the main work obs_base has a bug fix related to transfer configuration ap_verify_* are updates to the Gen 3 repositories in each dataset. This is essentially a re-conversion from Gen 2, and I don't think it requires review.
            Hide
            jbosch Jim Bosch added a comment -

            Looks good.  As suggested, I did not really look at the test data repos, but I left a few comments for ap_verify and did not have any for obs_base.

            Show
            jbosch Jim Bosch added a comment - Looks good.  As suggested, I did not really look at the test data repos, but I left a few comments for ap_verify and did not have any for obs_base.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel