Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22771

Resurrect HSC RC2 Gen3 repo bootstrap




      The refactoring of the Gen2 conversion tools on DM-17023 broke the scripts we had previously put together for creating a Gen3 data repository containing the HSC RC2 dataset. At the time this wasn't considered worth fixing, because we still hadn't addressed the issues that made RC2 ingest and preflight painfully slow. That has been partially done (i.e. ingest should now be faster) on DM-21768, so it's time to get that working again.

      This ticket is for getting the new Gen2 conversion tools working on RC2. After that's done, we may want to also make some (hopefully minor) improvements to the yaml import/export mechanism to provide a faster way of re-creating the Gen3 repo from scratch, given that a lot of the slowness in Gen2 conversion is due to limitations in how fast we can discover related datasets in the Gen2 repo.


        Issue Links


            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-22586 [ 427653 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21768 [ DM-21768 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Reviewers Hsin-Fang Chiang [ hchiang2 ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]


              jbosch Jim Bosch
              jbosch Jim Bosch
              Hsin-Fang Chiang
              Hsin-Fang Chiang, Jim Bosch
              0 Vote for this issue
              2 Start watching this issue




                  No builds found.