Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27113

Convert RC2 w_2020_38 to gen3 with w_2020_42 stack

    XMLWordPrintable

Details

    Description

      The conversion was originally attempted with w_2020_40, but it failed. A schema was renamed from rc2w38_ssw40 to rc2w38_ssw42, and another attempt was made with the w_2020_42 stack.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            I have a theory: there are equivalent deepCoadd_skyMap datasets in multiple Gen2 repos, we have no code dedicating to deduplicating them, and a

            config.runs["deepCoadd_skyMap"]

            override is telling the converter to put them all in one Gen3 collection. I think we want some of those to land in rerun-named collections instead, which means that override is active somewhere it shouldn't be in the converter code. But I'd like to poke around the Gen2 repos and conversion code to test that theory to be sure.

            jbosch Jim Bosch added a comment - I have a theory: there are equivalent deepCoadd_skyMap datasets in multiple Gen2 repos, we have no code dedicating to deduplicating them, and a config.runs ["deepCoadd_skyMap"] override is telling the converter to put them all in one Gen3 collection. I think we want some of those to land in rerun-named collections instead, which means that override is active somewhere it shouldn't be in the converter code. But I'd like to poke around the Gen2 repos and conversion code to test that theory to be sure.
            jbosch Jim Bosch added a comment -

            I have a potential fix on branch u/jbosch/DM-27113 of obs_base. madamow, could you test that? If it works, we should make sure it gets through Jenkins ci_hsc before merging it.

            jbosch Jim Bosch added a comment - I have a potential fix on branch u/jbosch/ DM-27113 of obs_base. madamow , could you test that? If it works, we should make sure it gets through Jenkins ci_hsc before merging it.

            jbosch one problem fixed, another popped out.

             

            INFO  2020-10-22T16:29:42.040-0500 convertRepo - Calibration validity gap closed from 2017-09-04 00:00:00.000 to 2017-09-05 00:00:00.000
            Traceback (most recent call last):
              File "./gen3-hsc-rc2/bootstrap.py", line 356, in <module>
                main()
              File "./gen3-hsc-rc2/bootstrap.py", line 352, in main
                continue_=options.continue_, reruns=reruns)
              File "./gen3-hsc-rc2/bootstrap.py", line 302, in run
                visits=makeVisitList(tracts, filters)
              File "/scratch/madamow/rc2w38_convert_w42/obs_base/python/lsst/obs/base/gen2to3/convertRepo.py", line 583, in run
                chain.append(spec.parent)
            AttributeError: 'Rerun' object has no attribute 'parent'
            

             

            madamow Monika Adamow (Inactive) added a comment - jbosch one problem fixed, another popped out.   INFO  2020-10-22T16:29:42.040-0500 convertRepo - Calibration validity gap closed from 2017-09-04 00:00:00.000 to 2017-09-05 00:00:00.000 Traceback (most recent call last):   File "./gen3-hsc-rc2/bootstrap.py", line 356, in <module>     main()   File "./gen3-hsc-rc2/bootstrap.py", line 352, in main     continue_=options.continue_, reruns=reruns)   File "./gen3-hsc-rc2/bootstrap.py", line 302, in run     visits=makeVisitList(tracts, filters)   File "/scratch/madamow/rc2w38_convert_w42/obs_base/python/lsst/obs/base/gen2to3/convertRepo.py", line 583, in run     chain.append(spec.parent) AttributeError: 'Rerun' object has no attribute 'parent'  
            jbosch Jim Bosch added a comment -

            I've pushed another commit to the u/jbosch/DM-27113 branch of obs_base that will hopefully fix that one.

            jbosch Jim Bosch added a comment - I've pushed another commit to the u/jbosch/ DM-27113 branch of obs_base that will hopefully fix that one.

            Thanks jbosch! The conversion is complete.

            madamow Monika Adamow (Inactive) added a comment - Thanks jbosch ! The conversion is complete.

            People

              madamow Monika Adamow (Inactive)
              madamow Monika Adamow (Inactive)
              Eli Rykoff, Jim Bosch, Michelle Gower, Monika Adamow (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.