# eupspkg doxygen builds fails on lsst_distrib docker images

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.5
• Epic Link:
• Team:
SQuaRE

## Description

Reported via slack yesterday afternoon, building afw from eupspkg is failing on the w_2018_03 docker image due to dangling references to doxygen artifacts associated with other products.

 buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"]) doxygen /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/Linux64/afw-14.0-35-g29d3a87a7+1/afw-14.0-35g29d3a87a7+1/doc/doxygen.conf error: @INCLUDE = base.inc: not found! scons: *** [doc/html] Error 1 ......................s.................  generated xml file: /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/Linux64/afw-14.0-35-g29d3a87a7+1/afw-14.0-35-g29d3a87a7+1/tests/.tests/pytest-afw.xml 

The fastest solution is to add back some or all of the doxygen doc dir to the installed products. Longer term, there should be a discussion about allowing an end user to disable the doxygen build and/or building the eups' tarballs without docs.

## Activity

Hide
Joshua Hoblitt added a comment -

The existing 7-stack-lsst_distrib-w_2018_03 image is 3.93GB. Removing only the doc/html directory from installs eups products bloats that up to 4.53GB but that's likely the best that can be done on short order.

I was able to build the current sims tagged version of afw on top of the image:

 [lsst@ca4f06936c76 stack]$echo$EUPS_PKGROOT https://eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://eups.lsst.codes/stack/src [lsst@ca4f06936c76 stack]$eups distrib install -t sims afw  [ 1/46 ] apr 1.5.2 (already installed) done.  [ 2/46 ] cfitsio 3360.lsst5 (already installed) done.  [ 3/46 ] doxygen 1.8.13.lsst1 (already installed) done.  [ 4/46 ] eigen 3.2.5.lsst3 (already installed) done.  [ 5/46 ] fftw 3.3.4.lsst2 (already installed) done.  [ 6/46 ] gsl 2.4 (already installed) done.  [ 7/46 ] libyaml 0.1.7 (already installed) done.  [ 8/46 ] mariadbclient 10.1.21.lsst2 (already installed) done.  [ 9/46 ] minuit2 5.34.14 (already installed) done.  [ 10/46 ] pybind11 2.1.1 (already installed) done.  [ 11/46 ] python 0.0.7 (already installed) done.  [ 12/46 ] starlink_ast tickets.DM-13322-gbfdec9744e done.  [ 13/46 ] apr_util 1.5.4 (already installed) done.  [ 14/46 ] astropy 2.0.1 (already installed) done.  [ 15/46 ] boost 1.60.lsst1+2 (already installed) done.  [ 16/46 ] numpy 1.13.1 (already installed) done.  [ 17/46 ] pep8_naming 0.4.1+1 (already installed) done.  [ 18/46 ] pycodestyle 2.3.1+1 (already installed) done.  [ 19/46 ] pyflakes 1.6.0 (already installed) done.  [ 20/46 ] pytest 3.2.0.lsst2 (already installed) done.  [ 21/46 ] python_execnet 1.4.1.lsst2 (already installed) done.  [ 22/46 ] python_future 0.16.0+1 (already installed) done.  [ 23/46 ] python_psutil 4.1.0+4 (already installed) done.  [ 24/46 ] pyyaml 3.11.lsst2 (already installed) done.  [ 25/46 ] scons 3.0.0.lsst1+1 (already installed) done.  [ 26/46 ] wcslib 5.13.lsst1+2 (already installed) done.  [ 27/46 ] log4cxx 0.10.0.lsst7 (already installed) done.  [ 28/46 ] matplotlib 2.0.2 (already installed) done.  [ 29/46 ] ndarray 1.3.3.lsst1 (already installed) done.  [ 30/46 ] pytest_forked 0.2.lsst2 (already installed) done.  [ 31/46 ] pytest_session2file 0.1.9+3 (already installed) done.  [ 32/46 ] python_mccabe 0.6.1+3 (already installed) done.  [ 33/46 ] flake8 3.5.0+2 (already installed) done.  [ 34/46 ] pytest_xdist 1.20.1.lsst2 (already installed) done.  [ 35/46 ] pytest_flake8 0.9.1+2 (already installed) done.  [ 36/46 ] sconsUtils 14.0-15-g79b7e05 (already installed) done.  [ 37/46 ] astshim 14.0-11-g0e62142+3 done.  [ 38/46 ] base 14.0-8-g7f6dd6b+6 (already installed) done.  [ 39/46 ] pex_exceptions 14.0-1-g13ef843+9 (already installed) done.  [ 40/46 ] utils 14.0-8-g1f982c9 (already installed) done.  [ 41/46 ] daf_base 14.0-4-ge2d7f21+8 (already installed) done.  [ 42/46 ] log 14.0-2-ga5af9b6+8 (already installed) done.  [ 43/46 ] pex_policy 14.0-1-g4b114ac+11 (already installed) done.  [ 44/46 ] daf_persistence 14.0-14-g87d16e8+7 (already installed) done.  [ 45/46 ] pex_config 14.0-2-g319577b+8 (already installed) done.  [ 46/46 ] afw 14.0-35-g29d3a87a7+1 done.  Show Joshua Hoblitt added a comment - The existing 7-stack-lsst_distrib-w_2018_03 image is 3.93GB . Removing only the doc/html directory from installs eups products bloats that up to 4.53GB but that's likely the best that can be done on short order. I was able to build the current sims tagged version of afw on top of the image: [lsst @ca4f06936c76 stack]$ echo $EUPS_PKGROOT https: //eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://eups.lsst.codes/stack/src [lsst @ca4f06936c76 stack]$ eups distrib install -t sims afw [ 1 / 46 ] apr 1.5 . 2 (already installed) done. [ 2 / 46 ] cfitsio 3360 .lsst5 (already installed) done. [ 3 / 46 ] doxygen 1.8 . 13 .lsst1 (already installed) done. [ 4 / 46 ] eigen 3.2 . 5 .lsst3 (already installed) done. [ 5 / 46 ] fftw 3.3 . 4 .lsst2 (already installed) done. [ 6 / 46 ] gsl 2.4 (already installed) done. [ 7 / 46 ] libyaml 0.1 . 7 (already installed) done. [ 8 / 46 ] mariadbclient 10.1 . 21 .lsst2 (already installed) done. [ 9 / 46 ] minuit2 5.34 . 14 (already installed) done. [ 10 / 46 ] pybind11 2.1 . 1 (already installed) done. [ 11 / 46 ] python 0.0 . 7 (already installed) done. [ 12 / 46 ] starlink_ast tickets.DM- 13322 -gbfdec9744e done. [ 13 / 46 ] apr_util 1.5 . 4 (already installed) done. [ 14 / 46 ] astropy 2.0 . 1 (already installed) done. [ 15 / 46 ] boost 1.60 .lsst1+ 2 (already installed) done. [ 16 / 46 ] numpy 1.13 . 1 (already installed) done. [ 17 / 46 ] pep8_naming 0.4 . 1 + 1 (already installed) done. [ 18 / 46 ] pycodestyle 2.3 . 1 + 1 (already installed) done. [ 19 / 46 ] pyflakes 1.6 . 0 (already installed) done. [ 20 / 46 ] pytest 3.2 . 0 .lsst2 (already installed) done. [ 21 / 46 ] python_execnet 1.4 . 1 .lsst2 (already installed) done. [ 22 / 46 ] python_future 0.16 . 0 + 1 (already installed) done. [ 23 / 46 ] python_psutil 4.1 . 0 + 4 (already installed) done. [ 24 / 46 ] pyyaml 3.11 .lsst2 (already installed) done. [ 25 / 46 ] scons 3.0 . 0 .lsst1+ 1 (already installed) done. [ 26 / 46 ] wcslib 5.13 .lsst1+ 2 (already installed) done. [ 27 / 46 ] log4cxx 0.10 . 0 .lsst7 (already installed) done. [ 28 / 46 ] matplotlib 2.0 . 2 (already installed) done. [ 29 / 46 ] ndarray 1.3 . 3 .lsst1 (already installed) done. [ 30 / 46 ] pytest_forked 0.2 .lsst2 (already installed) done. [ 31 / 46 ] pytest_session2file 0.1 . 9 + 3 (already installed) done. [ 32 / 46 ] python_mccabe 0.6 . 1 + 3 (already installed) done. [ 33 / 46 ] flake8 3.5 . 0 + 2 (already installed) done. [ 34 / 46 ] pytest_xdist 1.20 . 1 .lsst2 (already installed) done. [ 35 / 46 ] pytest_flake8 0.9 . 1 + 2 (already installed) done. [ 36 / 46 ] sconsUtils 14.0 - 15 -g79b7e05 (already installed) done. [ 37 / 46 ] astshim 14.0 - 11 -g0e62142+ 3 done. [ 38 / 46 ] base 14.0 - 8 -g7f6dd6b+ 6 (already installed) done. [ 39 / 46 ] pex_exceptions 14.0 - 1 -g13ef843+ 9 (already installed) done. [ 40 / 46 ] utils 14.0 - 8 -g1f982c9 (already installed) done. [ 41 / 46 ] daf_base 14.0 - 4 -ge2d7f21+ 8 (already installed) done. [ 42 / 46 ] log 14.0 - 2 -ga5af9b6+ 8 (already installed) done. [ 43 / 46 ] pex_policy 14.0 - 1 -g4b114ac+ 11 (already installed) done. [ 44 / 46 ] daf_persistence 14.0 - 14 -g87d16e8+ 7 (already installed) done. [ 45 / 46 ] pex_config 14.0 - 2 -g319577b+ 8 (already installed) done. [ 46 / 46 ] afw 14.0 - 35 -g29d3a87a7+ 1 done.
Hide
Joshua Hoblitt added a comment -

I've merged a few house keeping updates to the jenkins job that builds the release images, added some basic build "provenance" docker labels to the image, and changed the the doc delete to be only doc/html. Once I've gotten the bugs shaken out of these changes, I'll rebuild the last 4 weeklies.

Show
Joshua Hoblitt added a comment - I've merged a few house keeping updates to the jenkins job that builds the release images, added some basic build "provenance" docker labels to the image, and changed the the doc delete to be only doc/html . Once I've gotten the bugs shaken out of these changes, I'll rebuild the last 4 weeklies.
Hide
Joshua Hoblitt added a comment -

The tags 7-stack-lsst_distrib-w_2018_0[1234] have been repushed. Also note that jenkins is now pushing each image under a second tag that has a timestamp appended to it.

Show
Joshua Hoblitt added a comment - The tags 7-stack-lsst_distrib-w_2018_0 [1234] have been repushed. Also note that jenkins is now pushing each image under a second tag that has a timestamp appended to it.
Hide
Joshua Hoblitt added a comment -

Heather Kelly I have pushed a example of installing lsst_sims from mixed binary + source packages here:

https://github.com/lsst-sqre/docker-tarballs/tree/sims_demo

Note that although the resulting image is ~6GiB, it is not buildable in the default container size of 10GiB.  I changed the dockerd default to 50GiB after the first try ran out of space but didn't work out what the high water mark was.

I've publicly published the resulting image as:

docker.io/lsstsqre/simsdemo:latest

Eg., docker run -ti docker.io/lsstsqre/simsdemo

Show
Joshua Hoblitt added a comment - Heather Kelly I have pushed a example of installing lsst_sims from mixed binary + source packages here: https://github.com/lsst-sqre/docker-tarballs/tree/sims_demo Note that although the resulting image is ~6GiB, it is not buildable in the default container size of 10GiB.  I changed the dockerd default to 50GiB after the first try ran out of space but didn't work out what the high water mark was. I've publicly published the resulting image as: docker.io/lsstsqre/simsdemo:latest Eg., docker run -ti docker.io/lsstsqre/simsdemo
Hide
Joshua Hoblitt added a comment -

I assume that this issue is well resolved but will reopen this issue for any related problems.

Show
Joshua Hoblitt added a comment - I assume that this issue is well resolved but will reopen this issue for any related problems.
Hide
Heather Kelly added a comment -

Thanks Joshua Hoblitt Over the weekend, I took the rebuilt w_2018_03 Docker image as a base and built the lsst_sims on top of it, resulting in a 3 GiB compressed Docker image (it was ~7 GiB before compression) I put it on Dockerhub: https://hub.docker.com/r/lsstdesc/stack-sims/tags/

Is there a reason to prefer grabbing the binaries and building lsst_sims on top of that instead?  In any case, your Dockerfile will allow me to make sure I strip and remove extraneous files in a way that's consistent with your example.

Show
Heather Kelly added a comment - Thanks Joshua Hoblitt Over the weekend, I took the rebuilt w_2018_03 Docker image as a base and built the lsst_sims on top of it, resulting in a 3 GiB compressed Docker image (it was ~7 GiB before compression) I put it on Dockerhub: https://hub.docker.com/r/lsstdesc/stack-sims/tags/ Is there a reason to prefer grabbing the binaries and building lsst_sims on top of that instead?  In any case, your Dockerfile will allow me to make sure I strip and remove extraneous files in a way that's consistent with your example.

## People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Watchers:
Frossie Economou, Heather Kelly, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness
• Votes:
0 Vote for this issue
Watchers:
7 Start watching this issue

## Dates

• Created:
Updated:
Resolved: