# Remove obs dependency from ap_verify_testdata and ap_pipe_testdata

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
4
• Sprint:
AP S21-1 (December)
• Team:

#### Description

The ap_verify dataset framework was originally designed on the principle that ap_verify itself should not depend on any obs packages or on any datasets, and that users should set up only those that they need. Datasets have a dependency on the obs package they need, so that the package is (almost) guaranteed to be available if the dataset is set up. As far as I know, the current CI system assumes this behavior.

However, this causes a problem with ap_verify_testdata, because the package is included in some Stack builds despite being only an optional dependency of ap_verify. This causes frequent downloads of ap_verify_testdata, which would be prevented if ap_verify_testdata had no dependencies and it was ap_verify that guaranteed the obs package. This change would make ap_verify_testdata unusable for general processing, but this is already the case because ap_verify_testdata is not registered with ap_verify as a known dataset (but we would no longer be able to remove this special-casing in the future).

ap_pipe_testdata also has a dependency (on obs_decam), but is a conventional test data repository rather than an ap_verify dataset. Its dependencies should also be reworked as part of this ticket.

I think we can keep ap_pipe and ap_verify from having hard dependencies on obs packages if we make them optional. However, this requires reworking the test code that checks if testdata are available; see jointcal for one way to do this.

Removing the obs_lsst dependency will also break ap_verify_testdata's Gen 3 maintenance script. However, as long as it's obvious to the script caller that the problem is a missing package, I don't think anything needs to be done.

#### Activity

Hide
Krzysztof Findeisen added a comment -

The error when ap_verify_testdata is rebuilt without obs_lsst is

 ModuleNotFoundError: No module named 'lsst.obs.lsst'

Can't get more obvious than that.

Show
Krzysztof Findeisen added a comment - The error when ap_verify_testdata is rebuilt without obs_lsst is ModuleNotFoundError: No module named 'lsst.obs.lsst' Can't get more obvious than that.
Hide
John Parejko added a comment -

Handful of suggestions/questions on the PRs.

Show
John Parejko added a comment - Handful of suggestions/questions on the PRs.

#### People

Assignee:
Krzysztof Findeisen
Reporter:
Krzysztof Findeisen
Reviewers:
John Parejko
Watchers:
John Parejko, Krzysztof Findeisen, Meredith Rawls, Tim Jenness