Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13760

pipe_analysis needs updates for the wcs dataset name changes

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None

      Description

      Running the visit-level scripts using the current master of pipe_analysis (6b5727d) with the w_2018_10 stack gave errors in getting the wcs dataset.

      Traceback (most recent call last):
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py", line 215, in run
          raise RuntimeError("No datasets found for datasetType = {:s}".format(repoInfo.dataset))
      RuntimeError: No datasets found for datasetType = wcs
      

      It needs updates for the dataset name changes of DM-11138.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Hsin-Fang, I've pushed a branch that I think should fix things to work with the new and the old wcs dataset naming conventions.  Could you double-check if this works on both your w_2018_10 and w_2018_08 RC2 runs?  I'm not entirely sure if the comparison script adaptation will work, so if you could check the comparison between those two runs as well, that would be great.

            Show
            lauren Lauren MacArthur added a comment - Hsin-Fang, I've pushed a branch that I think should fix things to work with the new and the old wcs dataset naming conventions.  Could you double-check if this works on both your w_2018_10 and w_2018_08 RC2 runs?  I'm not entirely sure if the comparison script adaptation will work, so if you could check the comparison between those two runs as well, that would be great.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Lauren MacArthur I ran into the following with compareVisitAnalysis:

            Traceback (most recent call last):
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
                result = task.run(dataRef, **kwargs)
              File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py", line 640, in run
                if self.config.doApplyUberCal2 and "wcs" in repoInfo2.dataset2:
            AttributeError: 'Struct' object has no attribute 'dataset2'
            

            Looks like codes around line 640 need updates too

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Lauren MacArthur  I ran into the following with compareVisitAnalysis: Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py" , line 408 , in __call__ result = task.run(dataRef, **kwargs) File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/visitAnalysis.py" , line 640 , in run if self.config.doApplyUberCal2 and "wcs" in repoInfo2.dataset2: AttributeError: 'Struct' object has no attribute 'dataset2' Looks like codes around line 640 need updates too
            Hide
            lauren Lauren MacArthur added a comment -

            Ok, I pushed a fix.  Let me know if it works now.

            Show
            lauren Lauren MacArthur added a comment - Ok, I pushed a fix.  Let me know if it works now.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The new fix works with compareVisitAnalysis with w_2018_10, and all 5 scripts for the RC2 w_2018_10 have been finished (DM-13761).

            I'll try to run a few examples with w_2018_08 and report back.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The new fix works with compareVisitAnalysis with  w_2018_10 , and all 5 scripts for the RC2 w_2018_10 have been finished ( DM-13761 ). I'll try to run a few examples with w_2018_08 and report back.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I ran visitAnalysis.py using the ticket branch with one visit in the w_2018_08 output repo, and it ran fine, with the expected message visitAnalysis INFO: Old meas_mosaic dataset naming: wcs (new name is jointcal_wcs) in the log.

            compareVisitAnalysis in DM-13761 was the w_2018_10 repo against the w_2018_08 repo, and it finished fine too. So I think the branch is good to merge.

            Not in this ticket, but may we add some unit tests to pipe_analysis?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I ran visitAnalysis.py using the ticket branch with one visit in the w_2018_08 output repo, and it ran fine, with the expected message visitAnalysis INFO: Old meas_mosaic dataset naming: wcs (new name is jointcal_wcs) in the log. compareVisitAnalysis in DM-13761 was the w_2018_10 repo against the w_2018_08 repo, and it finished fine too. So I think the branch is good to merge. Not in this ticket, but may we add some unit tests to pipe_analysis?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The ticket branch has been merged to master per Lauren MacArthur's request

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The ticket branch has been merged to master per Lauren MacArthur 's request

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel