Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: lsstsw, obs_subaru
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Sprint:Science Pipelines DM-W16-6
-
Team:Data Release Production
Description
Thus:
$ eups distrib install -t w_2016_06 obs_subaru
|
...
|
[ 52/52 ] obs_subaru 5.0.0.1-60-ge4efae7+2 ...
|
|
|
***** error: from /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/build.log:
|
----------------------------------------------------------------------
|
Traceback (most recent call last):
|
File "tests/hscRepository.py", line 91, in setUp
|
self.repoPath = createDataRepository("lsst.obs.hsc.HscMapper", rawPath)
|
File "tests/hscRepository.py", line 63, in createDataRepository
|
check_call([ingest_cmd, repoPath] + glob(os.path.join(inputPath, "*.fits.gz")))
|
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
|
raise CalledProcessError(retcode, cmd)
|
CalledProcessError: Command '['/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/bin/hscIngestImages.py', '/var/folders/jp/lqz3n0m17nqft7bwtw3b8n380000gp/T/tmptUSKuf', '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/testdata_subaru/master-gf9ba9abdbe/hsc/raw/HSCA90402512.fits.gz']' returned non-zero exit status 1
|
|
|
----------------------------------------------------------------------
|
Ran 8 tests in 9.928s
|
|
|
FAILED (errors=7)
|
The following tests failed:
|
/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/tests/.tests/hscRepository.py.failed
|
1 tests failed
|
scons: *** [checkTestStatus] Error 1
|
scons: building terminated because of errors.
|
+ exit -4
|
Please fix it.
Attachments
Issue Links
Activity
By the way, I imagine this is breaking attempts to install lsst_distrib through eups distrib.
Should we create tests for the testdata_* packages that explicitly verify that the git-lfs pull worked?
This would prevent such packages from being setup.
Michael Wood-Vasey
added a comment - - edited Should we create tests for the testdata_* packages that explicitly verify that the git-lfs pull worked?
This would prevent such packages from being setup . EUPS doesn't know about git-lfs; we need to teach it about it, or you can also override the create() and/or fetch() verbs in eupspkg.cfg.sh in the short term.
To answer the original question, I think it's fine to blacklist this package in manifest.remap if you don't expect end users to want to install it using EUPS.
Thanks, Mario Juric. It's appropriate to treat testdata_subaru as analogous to afwdata, so blacklisting package creation seems like the way to go. The wider issue of eups distrib/git lfs integration isn't a current requirement.
Joshua Hoblitt, are you happy with the changes on lsstsw PR#83?
It looks like there were multiple issues going. Mario Juric identified the most severe with distrib packages being unaware of git-lfs. Another, minor, issue was that git tags were not being applied to the repo – resolved by Tim Jenness. I have merged John Swinbank PR to add testsdata_subaru to the remap list. TL;DR – the w_2016_06 weekly tag is broken for obs_subaru.
$ eups distrib install -t fe_test4 obs_subaru
|
[ 1/51 ] cfitsio 3360.lsst4 (already installed) done.
|
...
|
[ 51/51 ] obs_subaru 5.0.0.1-63-gfd8740e+1 done.
|
Thanks, everybody!
The obs_subaru table file declares testdata_subaru as setupOptional. However, eups distrib still attempts to install it:
$ eups distrib install -t w_2016_06 obs_subaru...[ 11/52 ] testdata_subaru master-gf9ba9abdbe (already installed) done....This produces an installed version which has not checked out files from git lfs:
$ cat ${TESTDATA_SUBARU_DIR}/hsc/calib/BIAS/2013-11-02/NONE/master/BIAS-050.fits.gzversion https://git-lfs.github.com/spec/v1oid sha256:d70512df8c1fd25f5c9bcc5b95f413c62f65da752f50a3da0a8331686682ad5asize 31345503obs_subaru will skip tests gracefully if testdata_subaru is not set up, however here it is set up but is populated with (from the point of view of the tests, at least) garbage data.
My suggested fix is that it should be treated in the same way as afwdata in lsstsw/etc/manifest.remap, which will suppress the creation of the package – I guess this will do the trick, as it's evidently possible to install afw through eups distrib. I'd appreciate input from eups distrib experts (Mario Juric?) on whether this is actually the correct solution, though.
Even if that fixes the immediate failure, I'm not sure if the failure of eups distrib install to check out the contents of files from git lfs is a bug or a feature.