Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27113

Convert RC2 w_2020_38 to gen3 with w_2020_42 stack

    XMLWordPrintable

    Details

      Description

      The conversion was originally attempted with w_2020_40, but it failed. A schema was renamed from rc2w38_ssw40 to rc2w38_ssw42, and another attempt was made with the w_2020_42 stack.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            I have a theory: there are equivalent deepCoadd_skyMap datasets in multiple Gen2 repos, we have no code dedicating to deduplicating them, and a

            config.runs["deepCoadd_skyMap"]

            override is telling the converter to put them all in one Gen3 collection. I think we want some of those to land in rerun-named collections instead, which means that override is active somewhere it shouldn't be in the converter code. But I'd like to poke around the Gen2 repos and conversion code to test that theory to be sure.

            Show
            jbosch Jim Bosch added a comment - I have a theory: there are equivalent deepCoadd_skyMap datasets in multiple Gen2 repos, we have no code dedicating to deduplicating them, and a config.runs ["deepCoadd_skyMap"] override is telling the converter to put them all in one Gen3 collection. I think we want some of those to land in rerun-named collections instead, which means that override is active somewhere it shouldn't be in the converter code. But I'd like to poke around the Gen2 repos and conversion code to test that theory to be sure.
            Hide
            jbosch Jim Bosch added a comment -

            I have a potential fix on branch u/jbosch/DM-27113 of obs_base. Monika Adamow, could you test that? If it works, we should make sure it gets through Jenkins ci_hsc before merging it.

            Show
            jbosch Jim Bosch added a comment - I have a potential fix on branch u/jbosch/ DM-27113 of obs_base. Monika Adamow , could you test that? If it works, we should make sure it gets through Jenkins ci_hsc before merging it.
            Hide
            madamow Monika Adamow added a comment -

            Jim Bosch one problem fixed, another popped out.

             

            INFO  2020-10-22T16:29:42.040-0500 convertRepo - Calibration validity gap closed from 2017-09-04 00:00:00.000 to 2017-09-05 00:00:00.000
            Traceback (most recent call last):
              File "./gen3-hsc-rc2/bootstrap.py", line 356, in <module>
                main()
              File "./gen3-hsc-rc2/bootstrap.py", line 352, in main
                continue_=options.continue_, reruns=reruns)
              File "./gen3-hsc-rc2/bootstrap.py", line 302, in run
                visits=makeVisitList(tracts, filters)
              File "/scratch/madamow/rc2w38_convert_w42/obs_base/python/lsst/obs/base/gen2to3/convertRepo.py", line 583, in run
                chain.append(spec.parent)
            AttributeError: 'Rerun' object has no attribute 'parent'
            

             

            Show
            madamow Monika Adamow added a comment - Jim Bosch one problem fixed, another popped out.   INFO  2020-10-22T16:29:42.040-0500 convertRepo - Calibration validity gap closed from 2017-09-04 00:00:00.000 to 2017-09-05 00:00:00.000 Traceback (most recent call last):   File "./gen3-hsc-rc2/bootstrap.py", line 356, in <module>     main()   File "./gen3-hsc-rc2/bootstrap.py", line 352, in main     continue_=options.continue_, reruns=reruns)   File "./gen3-hsc-rc2/bootstrap.py", line 302, in run     visits=makeVisitList(tracts, filters)   File "/scratch/madamow/rc2w38_convert_w42/obs_base/python/lsst/obs/base/gen2to3/convertRepo.py", line 583, in run     chain.append(spec.parent) AttributeError: 'Rerun' object has no attribute 'parent'  
            Hide
            jbosch Jim Bosch added a comment -

            I've pushed another commit to the u/jbosch/DM-27113 branch of obs_base that will hopefully fix that one.

            Show
            jbosch Jim Bosch added a comment - I've pushed another commit to the u/jbosch/ DM-27113 branch of obs_base that will hopefully fix that one.
            Hide
            madamow Monika Adamow added a comment -

            Thanks Jim Bosch! The conversion is complete.

            Show
            madamow Monika Adamow added a comment - Thanks Jim Bosch ! The conversion is complete.

              People

              Assignee:
              madamow Monika Adamow
              Reporter:
              madamow Monika Adamow
              Watchers:
              Eli Rykoff, Jim Bosch, Michelle Gower, Monika Adamow, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.