Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22655

Genericize gen2to3.py to be useable with any gen2 repo

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP S20-1 (December), AP S20-2 (January), AP S20-3 (February), AP S20-4 (March)
    • Team:
      Alert Production

      Description

      To deal with the various obs_package gen3 support tickets, I'm going to try to make a more generic version of the gen2to3.py conversion script. This will serve as a useful stopgap for using test datasets, while the official "create a gen3 repo from scratch" code is sorted out.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21865 Extend Gen3 butler support for obs_cfht

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21862 Extend Gen3 butler support for obs_decam

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23616 Run converted ap_verify testdata through gen3 pipeline

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21866 Extend Gen3 butler support for obs_sdss

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21864 Extend Gen3 butler support for obs_lsst

          • Invalid
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22708 Fix decam gen3 ingest

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23630 fgcmcal failure Ubuntu

          • Done
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23728 Cleanup ci_hsc_gen2 to use new convert script instead of custom one

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23729 Generalize dataset to formatter mapping in obs packages

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23778 Write config and tests for obs_lsst gen2-gen3 convert

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23828 Create small test decam gen2 repo with calibs for test_convert2to3.py

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23640 Unify raw and calibration dataset types in gen2 mappers

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22676 Create gaia dr2 refcat for ap_verify_hits2015

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-17444 dark exposure does not contain getVisitInfo()/getDarkTime()

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            15 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            I have some major comments and questions on the obs_base changes. See GitHub.

            Show
            tjenness Tim Jenness added a comment - I have some major comments and questions on the obs_base changes. See GitHub.
            Hide
            Permalink
            Parejkoj John Parejko added a comment - - edited

            Jenkins run with ci_hsc: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31337/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited Jenkins run with ci_hsc: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31337/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            I believe I have dealt with most of your comments. I've fired off a new Jenkins run, but we should sort out a few of the remaining questions (in particular what to do about ci for DECam given file sizes: see my comment on that PR). It may be worth a short telecon to discuss our options for that.

            Show
            Parejkoj John Parejko added a comment - I believe I have dealt with most of your comments. I've fired off a new Jenkins run, but we should sort out a few of the remaining questions (in particular what to do about ci for DECam given file sizes: see my comment on that PR). It may be worth a short telecon to discuss our options for that.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Final rebase Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31343/pipeline

            Show
            Parejkoj John Parejko added a comment - Final rebase Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31343/pipeline
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Jenkins was clean. Huge thank you to Tim Jenness for helping me work through the final questions (several of which spawned extra tickets).

            Show
            Parejkoj John Parejko added a comment - Jenkins was clean. Huge thank you to Tim Jenness for helping me work through the final questions (several of which spawned extra tickets).

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Tim Jenness
              Watchers:
              Dino Bektesevic, Eric Bellm, Heather Kelly, James Chiang, Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Meredith Rawls, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.