Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
-
Location:this ticket
Description
The discussion in DM-2527 demonstrated that test code uses EUPS to determine where things are located. This has two problems:
- Calling EUPS in a loop resulted in slow downs of the tests as the EUPS database was repeatedly read.
- This makes it impossible to run the tests standalone and burns-in a particular packaging implementation that is not relevant for the tests.
This RFC proposes that EUPS be removed from tests and an alternative scheme be used for locating supporting infrastructure. Some tests still need to locate other packages (in particular afwdata) so any replacement scheme must support some form of package location.
Some options for locating afwdata:
- Provide a test utility function for locating a package which can use an environment variable to find afwdata.
- Add afwdata as a git submodule in the test directory. This could result in multiple checkouts of the repository.
This RFC invites further discussion on whether EUPS should be removed from the tests. Comments on replacement functionality are welcomed to demonstrate feasibility.
Attachments
Issue Links
- blocks
-
- Adopted
- is blocked by
-
DM-2769 sconsUtils has a hard dependency on EUPS for both tests and installation
- Done
-
- Invalid
-
DM-2772
datarel tests depend on EUPS
- Done
-
DM-2773
testing_endtoend tests depend on EUPS
- Done
- relates to
-
- Won't Fix
-
DM-2635 Provide a function to return the path to a package, given its name
- Done
-
- Done
-
- Done
-
- Done
Activity
For this task I think an environment variable is fine, and in fact that's all eups.getProductDir() does under the covers. I think that tho means that the eups lookup could be made fast in this case but I agree that it's not a good idea; for now we could use eups to set the environment variable (a task it does well).
We'd wrap this in a function in e.g. lsst.utils so that we have the freedom to change implementations.
I don't like the multiple git checkout version. One reason to move test data to afwdata was to stop this data duplication.
The use of eups in meas_astrom is more of a problem than this (we use it to manage multiple data sets simultaneously). There can be one "current" afwdata, but there's no preferred astrometric catalogue. I suspect this will need a different design.
There were no objections to the concept of removing eups from the stack runtime and test python code. Implementation decisions will be handled in an explicit ticket.
Joshua Hoblitt has found some cases of SCons using eups when it doesn't really need to. eups itself is directly imported by code in datarel, daf_butlerUtils, obs_lsstSim, obs_sdss, obs_test, pipe_base and sconsUtils. More directly relevant to this ticket is that 44 tests (at least) import eups and a random pick of three determined (from pylint perspective) that two of them didn't require the import anyhow (presumably test boilerplate). We need to clean out the cruft so that we know what really needs eups and what is just pretending to need eups. It looks like the ones that require eups are using eups.productDir. I assume these should (mostly) be replaced with calls to the utils function. We missed them all the last time we discussed this.
Dumb question, but do all our repos depend on the butler (and I don't mean in actual fact, I mean for architectural reasons). If not, surely creating a dependence on the butler replaces one problem with another?