# Recover from the recent CameraMapper changes and refcat changes

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
5
• Team:
Data Facility

#### Description

Recent API changes in DM-8686 broke the Pegasus DAX-generating scripts. Recover from the changes and make it work again.

First error:

  File "ciHsc/generateDax.py", line 179, in generateDax  mapper = HscMapper(root=inputRepo, outputRoot=outPath)  File "/home/hchiang2/lsstsw/stack/Linux64/obs_subaru/13.0-14-gb43a883+1/python/lsst/obs/hsc/hscMapper.py", line 31, in __init__  super(HscMapper, self).__init__(policy, policyFile.getRepositoryPath(), **kwargs) TypeError: __init__() got an unexpected keyword argument 'outputRoot' 

Also, DM-9438 switched the reference catalog to use the LSST HTM format. Do the necessary changes to make the dax run again.

#### Activity

Hide
Hsin-Fang Chiang added a comment -

(1) Before DM-8686, mapper.map_{datasetType}.getLocations() gives the absolute paths, and if outputRoot is given, the path is through all _parent links. After DM-8686, mapper.map_{datasetType}.getLocations() gives the relative paths (~ the substituted templates). This actually simplified setting LFNs and PFNs for Pegasus.

(2) DM-9438 means new file dependencies (the reference catalog) need to be added. Currently, if only the butler interface can be used and the science calculation in meas_algorithms cannot be used, there seems not a way to know which pixel_id (which FITS file) is needed to process a given data. For now, I assume all FITS in the ref_cats folder are needed for any task that uses the ref_cats.

This ticket only fixes the broken scripts and make them run again. It does not fix the existing problems, such as having duplicate codes in two workflows. Some recent changes seem to have added outputs to some pipelines; I do not chase that down in this ticket either.

Show
Hsin-Fang Chiang added a comment - (1) Before DM-8686 , mapper.map_{datasetType}.getLocations() gives the absolute paths, and if outputRoot is given, the path is through all _parent links. After DM-8686 , mapper.map_{datasetType}.getLocations() gives the relative paths (~ the substituted templates). This actually simplified setting LFNs and PFNs for Pegasus. (2) DM-9438 means new file dependencies (the reference catalog) need to be added. Currently, if only the butler interface can be used and the science calculation in meas_algorithms cannot be used, there seems not a way to know which pixel_id (which FITS file) is needed to process a given data. For now, I assume all FITS in the ref_cats folder are needed for any task that uses the ref_cats . This ticket only fixes the broken scripts and make them run again. It does not fix the existing problems, such as having duplicate codes in two workflows. Some recent changes seem to have added outputs to some pipelines; I do not chase that down in this ticket either.
Hide
Hsin-Fang Chiang added a comment -

The changes are duplicate in the two workflows, as those codes are duplicate I think it needs refactoring with a better design (as well as adding in some of your Executor codes I guess?), but I do not intend to do so in the ticket.

Show
Hsin-Fang Chiang added a comment - May you please review this? The changes are duplicate in the two workflows, as those codes are duplicate I think it needs refactoring with a better design (as well as adding in some of your Executor codes I guess?), but I do not intend to do so in the ticket.
Hide
Hsin-Fang Chiang added a comment -

I have verified on lsst-dev that with this ticket patch, the scripts can run successfully again!

Show
Hsin-Fang Chiang added a comment - I have verified on lsst-dev that with this ticket patch, the scripts can run successfully again!
Hide
Hsin-Fang Chiang added a comment -

Thank you for your prompt review!

Merged.

Show
Hsin-Fang Chiang added a comment - Thank you for your prompt review! Merged.

#### People

Assignee:
Hsin-Fang Chiang
Reporter:
Hsin-Fang Chiang
Reviewers:
Mikolaj Kowalik
Watchers:
Hsin-Fang Chiang, Mikolaj Kowalik