Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9934

Recover from the recent CameraMapper changes and refcat changes

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Recent API changes in DM-8686 broke the Pegasus DAX-generating scripts. Recover from the changes and make it work again.

      First error:

        File "ciHsc/generateDax.py", line 179, in generateDax
          mapper = HscMapper(root=inputRepo, outputRoot=outPath)
        File "/home/hchiang2/lsstsw/stack/Linux64/obs_subaru/13.0-14-gb43a883+1/python/lsst/obs/hsc/hscMapper.py", line 31, in __init__
          super(HscMapper, self).__init__(policy, policyFile.getRepositoryPath(), **kwargs)
      TypeError: __init__() got an unexpected keyword argument 'outputRoot'
      

      Also, DM-9438 switched the reference catalog to use the LSST HTM format. Do the necessary changes to make the dax run again.

        Attachments

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          (1) Before DM-8686, mapper.map_{datasetType}.getLocations() gives the absolute paths, and if outputRoot is given, the path is through all _parent links. After DM-8686, mapper.map_{datasetType}.getLocations() gives the relative paths (~ the substituted templates). This actually simplified setting LFNs and PFNs for Pegasus.

          (2) DM-9438 means new file dependencies (the reference catalog) need to be added. Currently, if only the butler interface can be used and the science calculation in meas_algorithms cannot be used, there seems not a way to know which pixel_id (which FITS file) is needed to process a given data. For now, I assume all FITS in the ref_cats folder are needed for any task that uses the ref_cats.

          This ticket only fixes the broken scripts and make them run again. It does not fix the existing problems, such as having duplicate codes in two workflows. Some recent changes seem to have added outputs to some pipelines; I do not chase that down in this ticket either.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - (1) Before DM-8686 , mapper.map_{datasetType}.getLocations() gives the absolute paths, and if outputRoot is given, the path is through all _parent links. After DM-8686 , mapper.map_{datasetType}.getLocations() gives the relative paths (~ the substituted templates). This actually simplified setting LFNs and PFNs for Pegasus. (2) DM-9438 means new file dependencies (the reference catalog) need to be added. Currently, if only the butler interface can be used and the science calculation in meas_algorithms cannot be used, there seems not a way to know which pixel_id (which FITS file) is needed to process a given data. For now, I assume all FITS in the ref_cats folder are needed for any task that uses the ref_cats . This ticket only fixes the broken scripts and make them run again. It does not fix the existing problems, such as having duplicate codes in two workflows. Some recent changes seem to have added outputs to some pipelines; I do not chase that down in this ticket either.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          May you please review this?

          The changes are duplicate in the two workflows, as those codes are duplicate I think it needs refactoring with a better design (as well as adding in some of your Executor codes I guess?), but I do not intend to do so in the ticket.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - May you please review this? The changes are duplicate in the two workflows, as those codes are duplicate I think it needs refactoring with a better design (as well as adding in some of your Executor codes I guess?), but I do not intend to do so in the ticket.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          I have verified on lsst-dev that with this ticket patch, the scripts can run successfully again!

          Show
          hchiang2 Hsin-Fang Chiang added a comment - I have verified on lsst-dev that with this ticket patch, the scripts can run successfully again!
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thank you for your prompt review!

          Merged.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thank you for your prompt review! Merged.

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Mikolaj Kowalik
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: