Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: stack release
-
Labels:None
-
Story Points:0.1
-
Team:Architecture
Description
The build of w.2018.32 failed during tagging due to new eups products being pulled into the build which did not have the required github team membership. It appears that this was corrected and the weekly build resubmitted.
The 2nd weekly attempt failing attempting to eups distrib install lsst/scarlet as part of the binary tarball build. The failure is clearly from the setup.py.
Writing log to: /build/stack/miniconda3-4.5.4-10a4fa6/EupsBuildDir/Linux64/scarlet-lsst-dev-gc293e6f0ef/build.log |
[build] + eval 'VAL_=$VERSION' |
++ VAL_=lsst-dev-gc293e6f0ef
|
+ '[' -z lsst-dev-gc293e6f0ef ']' |
+ [[ -f SConstruct ]]
|
+ [[ -f configure ]]
|
+ [[ -f Makefile ]]
|
+ [[ -f makefile ]]
|
+ [[ -f GNUmakefile ]]
|
+ [[ -f setup.py ]]
|
+ python setup.py build
|
fatal: Not a git repository (or any of the parent directories): .git
|
Traceback (most recent call last):
|
File "setup.py", line 21, in <module> |
__version__ = '0.3.'+subprocess.check_output(['git', 'rev-parse', 'HEAD'])[:7].decode("utf-8") |
File "/build/python/miniconda3-4.5.4/envs/lsst-scipipe-10a4fa6/lib/python3.6/subprocess.py", line 336, in check_output |
**kwargs).stdout
|
File "/build/python/miniconda3-4.5.4/envs/lsst-scipipe-10a4fa6/lib/python3.6/subprocess.py", line 418, in run |
output=stdout, stderr=stderr)
|
subprocess.CalledProcessError: Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128. |
+ exit -4 |
https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball/detail/tarball/3173/pipeline
The failure appears to be caused by shelling out to run git from setup.py, resulting in an eups product that can be build under lsst-build but not by eups distrib install – which means that binary tarballs can not be built. In order for the weekly to run either lsst/scarlet needs to be updated to not assume the build is running in a git repo or be converted to use standard python package tooling such as setuptools_scm or DM-15104 should be reverted until the packaging is repaired.
Attachments
Issue Links
Activity
Tim Jenness It is unfortunate that end-to-end testing in advance wasn't considered a blocker for the workshop. Also, is scarlet a true 3rd party package? It isn't install-able from pypi in its current state, so it seems likely that LSST pipelines is the only consumer. There don't appear to be any unit tests, which is a bit troubling for a package considered a critical dependency.
scarlet is not considered a critical dependency yet. Currently the default deblender is still the HSC-SDSS deblender that has been running in meas_deblender. The work this week was to integrate scarlet so that it can be setup to run in the stack in parallel with the current deblender for future testing, and so that it can be demonstrated during the workshop.
I admit that the lack of testing is very unfortunate, but in order to have scarlet integrated into the stack before I leave the project at the end of next week we prioritized refactoring the task structure so that scarlet can be run, since Peter Melchior and I still plan on developing scarlet (including writing test code in the near future) but neither of us will be responsible for maintaining the stack implementation.
So it was important to finish DM-15104 and DM-15246 this week. I apologize for all of the problems that this caused, I wasn't aware that there was any danger once all of the Jenkins tests passed and ci_hsc built and ran successfully.
Thank you Tim Jenness for the quick fix.
Fred Moolekamp
added a comment - scarlet is not considered a critical dependency yet. Currently the default deblender is still the HSC-SDSS deblender that has been running in meas_deblender . The work this week was to integrate scarlet so that it can be setup to run in the stack in parallel with the current deblender for future testing, and so that it can be demonstrated during the workshop.
I admit that the lack of testing is very unfortunate, but in order to have scarlet integrated into the stack before I leave the project at the end of next week we prioritized refactoring the task structure so that scarlet can be run, since Peter Melchior and I still plan on developing scarlet (including writing test code in the near future) but neither of us will be responsible for maintaining the stack implementation.
So it was important to finish DM-15104 and DM-15246 this week. I apologize for all of the problems that this caused, I wasn't aware that there was any danger once all of the Jenkins tests passed and ci_hsc built and ran successfully.
Thank you Tim Jenness for the quick fix. Tim Jenness change:
commit fdfbe1c24037acfb2a9f7f87f3f505e4df92f041 (HEAD -> lsst-dev, tag: w.2018.32, origin/lsst-dev)
Author: Tim Jenness <tjenness@lsst.org>
Date: Sat Aug 11 18:30:20 2018 -0700
seems to have solved the problem, since the scarlet binary is now available in EUPS:
We can just wait for confirmation next weekly to be build correctly before closing this issue.
The new weekly did come out. Fred Moolekamp can you please ensure that you make a similar fix upstream (and consider creating proper development release version numbers to prevent the other warning).
Sure, I'll make the same fix upstream so that this will not happen in the future.
Fred Moolekamp
added a comment - Sure, I'll make the same fix upstream so that this will not happen in the future.
I'm pretty sure that
DM-15104is needed for the deblending workshop this week at LSST2018 so I don't think we can revert that. I've put in a quick fix on the lsst-dev branch of scarlet which should trap for git not being available. Obviously the fix is not perfect (regardless, setuptools complains about the version formatting but that's for Fred Moolekamp to fix) but I hope it fixes the immediate problem.