Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23778

Write config and tests for obs_lsst gen2-gen3 convert

    XMLWordPrintable

Details

    • 2
    • Architecture
    • No

    Description

      tjenness pointed out during the review of DM-22655 that we have gen2 repos to test obs_lsst gen2-3 conversion. We will also need to specify some configs specific to this package. Implementing `test_convert2to3.py` would be an easy start, once the right test data is identified.

      tjenness: In your comment you said "This repo even comes with a bunch of gen2 repos...": where are they, and what to they contain? We do have `testdata_lsst`, but that's not in lsst_distrib (how big is it? The readme isn't helpful.) and it doesn't contain premade gen2 repos.

      Attachments

        Issue Links

          Activity

            No builds found.
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue is triggered by DM-22655 [ DM-22655 ]
            tjenness Tim Jenness added a comment -

            data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.

            tjenness Tim Jenness added a comment - data/input has 8 gen2 repositories with test files used by the gen2 tests. obs_lsst does not depend on an external testdata package.
            tjenness Tim Jenness made changes -
            Labels gen2-deprecation-blocker gen3-middleware
            tjenness Tim Jenness made changes -
            Assignee Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Team Alert Production [ 10300 ] Architecture [ 10304 ]
            tjenness Tim Jenness made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-21706 [ DM-21706 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-23931 [ DM-23931 ]
            tjenness Tim Jenness added a comment -

            jbosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.

            tjenness Tim Jenness added a comment - jbosch the changes are mostly minor in obs_base to make it a bit more generic and in obs_lsst to make writeCuratedCalibrations work properly for defects and qe_curve. Doing this meant that I had to fix qe_curve in obs_lsst_data so that it matched the dataset type name rather than being slightly different. I talked to krughoff before doing that and we agreed that it's easier if the directory in obs_*_data packages matches the dataset type name. krughoff would you mind looking at the obs_lsst_data and pipe_tasks pull requests? They are trivial renames so it's not critical that you look.
            tjenness Tim Jenness made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            tjenness Tim Jenness added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            Looks great. Thanks for fixing that tjenness.

            krughoff Simon Krughoff (Inactive) added a comment - Looks great. Thanks for fixing that tjenness .
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch added a comment -

            One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment.

            I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.

            jbosch Jim Bosch added a comment - One final point, I've changed obs_base so that you can convert multiple gen2 repos into a single gen3 repo. I can back that change out if you don't like it. The main problem is that I have to catch all exceptions from inserting of dimension data and registering instruments. This might not be a good idea. I have demonstrated that it works but the testing infrastructure in obs_base makes it hard to test it at the moment. I'm happy to leave this at however far you've gotten it.  I suspect it's going to be subject to race conditions, but not being able to convert multiple Gen2 repos into a single Gen3 was such a big limitation that it would have hindered adoption of the script form, which would be bad.  We'll fix any concurrency issues later.
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-23975 [ DM-23975 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-23976 [ DM-23976 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-21864 [ DM-21864 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-21864 [ DM-21864 ]
            tjenness Tim Jenness made changes -
            Link This issue duplicates DM-21864 [ DM-21864 ]

            People

              tjenness Tim Jenness
              Parejkoj John Parejko
              Jim Bosch
              Jim Bosch, John Parejko, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.