Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23778

Write config and tests for obs_lsst gen2-gen3 convert

    XMLWordPrintable

Details

    • 2
    • Architecture
    • No

    Description

      tjenness pointed out during the review of DM-22655 that we have gen2 repos to test obs_lsst gen2-3 conversion. We will also need to specify some configs specific to this package. Implementing `test_convert2to3.py` would be an easy start, once the right test data is identified.

      tjenness: In your comment you said "This repo even comes with a bunch of gen2 repos...": where are they, and what to they contain? We do have `testdata_lsst`, but that's not in lsst_distrib (how big is it? The readme isn't helpful.) and it doesn't contain premade gen2 repos.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.

            tjenness Tim Jenness added a comment - data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.
            tjenness Tim Jenness added a comment -

            jbosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.

            tjenness Tim Jenness added a comment - jbosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.
            tjenness Tim Jenness added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            tjenness Tim Jenness added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            Looks great. Thanks for fixing that tjenness.

            krughoff Simon Krughoff (Inactive) added a comment - Looks great. Thanks for fixing that tjenness .
            jbosch Jim Bosch added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.

            jbosch Jim Bosch added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment. I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.

            People

              tjenness Tim Jenness
              Parejkoj John Parejko
              Jim Bosch
              Jim Bosch, John Parejko, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.