Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23778

Write config and tests for obs_lsst gen2-gen3 convert

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Tim Jenness pointed out during the review of DM-22655 that we have gen2 repos to test obs_lsst gen2-3 conversion. We will also need to specify some configs specific to this package. Implementing `test_convert2to3.py` would be an easy start, once the right test data is identified.

      Tim Jenness: In your comment you said "This repo even comes with a bunch of gen2 repos...": where are they, and what to they contain? We do have `testdata_lsst`, but that's not in lsst_distrib (how big is it? The readme isn't helpful.) and it doesn't contain premade gen2 repos.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.

            Show
            tjenness Tim Jenness added a comment - data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.
            Hide
            tjenness Tim Jenness added a comment -

            Jim Bosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to Simon Krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. Simon Krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.

            Show
            tjenness Tim Jenness added a comment - Jim Bosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to Simon Krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. Simon Krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.
            Hide
            tjenness Tim Jenness added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            Show
            tjenness Tim Jenness added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.
            Hide
            krughoff Simon Krughoff added a comment -

            Looks great. Thanks for fixing that Tim Jenness.

            Show
            krughoff Simon Krughoff added a comment - Looks great. Thanks for fixing that Tim Jenness .
            Hide
            jbosch Jim Bosch added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.

            Show
            jbosch Jim Bosch added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment. I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, John Parejko, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.