Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13474

eupspkg doxygen builds fails on lsst_distrib docker images

    Details

      Description

      Reported via slack yesterday afternoon, building afw from eupspkg is failing on the w_2018_03 docker image due to dangling references to doxygen artifacts associated with other products.

      buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
      doxygen /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/Linux64/afw-14.0-35-g29d3a87a7+1/afw-14.0-35g29d3a87a7+1/doc/doxygen.conf
      error: @INCLUDE = base.inc: not found!
      scons: *** [doc/html] Error 1
      ......................s.................
       generated xml file: /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/EupsBuildDir/Linux64/afw-14.0-35-g29d3a87a7+1/afw-14.0-35-g29d3a87a7+1/tests/.tests/pytest-afw.xml 
      

      The fastest solution is to add back some or all of the doxygen doc dir to the installed products. Longer term, there should be a discussion about allowing an end user to disable the doxygen build and/or building the eups' tarballs without docs.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The existing 7-stack-lsst_distrib-w_2018_03 image is 3.93GB. Removing only the doc/html directory from installs eups products bloats that up to 4.53GB but that's likely the best that can be done on short order.

          I was able to build the current sims tagged version of afw on top of the image:

          [lsst@ca4f06936c76 stack]$ echo $EUPS_PKGROOT
          https://eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://eups.lsst.codes/stack/src
          [lsst@ca4f06936c76 stack]$ eups distrib install -t sims afw
             [  1/46 ]  apr 1.5.2 (already installed)                              done.
             [  2/46 ]  cfitsio 3360.lsst5 (already installed)                     done.
             [  3/46 ]  doxygen 1.8.13.lsst1 (already installed)                   done.
             [  4/46 ]  eigen 3.2.5.lsst3 (already installed)                      done.
             [  5/46 ]  fftw 3.3.4.lsst2 (already installed)                       done.
             [  6/46 ]  gsl 2.4 (already installed)                                done.
             [  7/46 ]  libyaml 0.1.7 (already installed)                          done.
             [  8/46 ]  mariadbclient 10.1.21.lsst2 (already installed)            done.
             [  9/46 ]  minuit2 5.34.14 (already installed)                        done.
             [ 10/46 ]  pybind11 2.1.1 (already installed)                         done.
             [ 11/46 ]  python 0.0.7 (already installed)                           done.
             [ 12/46 ]  starlink_ast tickets.DM-13322-gbfdec9744e                  done.
             [ 13/46 ]  apr_util 1.5.4 (already installed)                         done.
             [ 14/46 ]  astropy 2.0.1 (already installed)                          done.
             [ 15/46 ]  boost 1.60.lsst1+2 (already installed)                     done.
             [ 16/46 ]  numpy 1.13.1 (already installed)                           done.
             [ 17/46 ]  pep8_naming 0.4.1+1 (already installed)                    done.
             [ 18/46 ]  pycodestyle 2.3.1+1 (already installed)                    done.
             [ 19/46 ]  pyflakes 1.6.0 (already installed)                         done.
             [ 20/46 ]  pytest 3.2.0.lsst2 (already installed)                     done.
             [ 21/46 ]  python_execnet 1.4.1.lsst2 (already installed)             done.
             [ 22/46 ]  python_future 0.16.0+1 (already installed)                 done.
             [ 23/46 ]  python_psutil 4.1.0+4 (already installed)                  done.
             [ 24/46 ]  pyyaml 3.11.lsst2 (already installed)                      done.
             [ 25/46 ]  scons 3.0.0.lsst1+1 (already installed)                    done.
             [ 26/46 ]  wcslib 5.13.lsst1+2 (already installed)                    done.
             [ 27/46 ]  log4cxx 0.10.0.lsst7 (already installed)                   done.
             [ 28/46 ]  matplotlib 2.0.2 (already installed)                       done.
             [ 29/46 ]  ndarray 1.3.3.lsst1 (already installed)                    done.
             [ 30/46 ]  pytest_forked 0.2.lsst2 (already installed)                done.
             [ 31/46 ]  pytest_session2file 0.1.9+3 (already installed)            done.
             [ 32/46 ]  python_mccabe 0.6.1+3 (already installed)                  done.
             [ 33/46 ]  flake8 3.5.0+2 (already installed)                         done.
             [ 34/46 ]  pytest_xdist 1.20.1.lsst2 (already installed)              done.
             [ 35/46 ]  pytest_flake8 0.9.1+2 (already installed)                  done.
             [ 36/46 ]  sconsUtils 14.0-15-g79b7e05 (already installed)            done.
             [ 37/46 ]  astshim 14.0-11-g0e62142+3                                 done.
             [ 38/46 ]  base 14.0-8-g7f6dd6b+6 (already installed)                 done.
             [ 39/46 ]  pex_exceptions 14.0-1-g13ef843+9 (already installed)       done.
             [ 40/46 ]  utils 14.0-8-g1f982c9 (already installed)                  done.
             [ 41/46 ]  daf_base 14.0-4-ge2d7f21+8 (already installed)             done.
             [ 42/46 ]  log 14.0-2-ga5af9b6+8 (already installed)                  done.
             [ 43/46 ]  pex_policy 14.0-1-g4b114ac+11 (already installed)          done.
             [ 44/46 ]  daf_persistence 14.0-14-g87d16e8+7 (already installed)     done.
             [ 45/46 ]  pex_config 14.0-2-g319577b+8 (already installed)           done.
             [ 46/46 ]  afw 14.0-35-g29d3a87a7+1                                   done.
          

          Show
          jhoblitt Joshua Hoblitt added a comment - The existing 7-stack-lsst_distrib-w_2018_03 image is 3.93GB . Removing only the doc/html directory from installs eups products bloats that up to 4.53GB but that's likely the best that can be done on short order. I was able to build the current sims tagged version of afw on top of the image: [lsst @ca4f06936c76 stack]$ echo $EUPS_PKGROOT https: //eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://eups.lsst.codes/stack/src [lsst @ca4f06936c76 stack]$ eups distrib install -t sims afw [ 1 / 46 ] apr 1.5 . 2 (already installed) done. [ 2 / 46 ] cfitsio 3360 .lsst5 (already installed) done. [ 3 / 46 ] doxygen 1.8 . 13 .lsst1 (already installed) done. [ 4 / 46 ] eigen 3.2 . 5 .lsst3 (already installed) done. [ 5 / 46 ] fftw 3.3 . 4 .lsst2 (already installed) done. [ 6 / 46 ] gsl 2.4 (already installed) done. [ 7 / 46 ] libyaml 0.1 . 7 (already installed) done. [ 8 / 46 ] mariadbclient 10.1 . 21 .lsst2 (already installed) done. [ 9 / 46 ] minuit2 5.34 . 14 (already installed) done. [ 10 / 46 ] pybind11 2.1 . 1 (already installed) done. [ 11 / 46 ] python 0.0 . 7 (already installed) done. [ 12 / 46 ] starlink_ast tickets.DM- 13322 -gbfdec9744e done. [ 13 / 46 ] apr_util 1.5 . 4 (already installed) done. [ 14 / 46 ] astropy 2.0 . 1 (already installed) done. [ 15 / 46 ] boost 1.60 .lsst1+ 2 (already installed) done. [ 16 / 46 ] numpy 1.13 . 1 (already installed) done. [ 17 / 46 ] pep8_naming 0.4 . 1 + 1 (already installed) done. [ 18 / 46 ] pycodestyle 2.3 . 1 + 1 (already installed) done. [ 19 / 46 ] pyflakes 1.6 . 0 (already installed) done. [ 20 / 46 ] pytest 3.2 . 0 .lsst2 (already installed) done. [ 21 / 46 ] python_execnet 1.4 . 1 .lsst2 (already installed) done. [ 22 / 46 ] python_future 0.16 . 0 + 1 (already installed) done. [ 23 / 46 ] python_psutil 4.1 . 0 + 4 (already installed) done. [ 24 / 46 ] pyyaml 3.11 .lsst2 (already installed) done. [ 25 / 46 ] scons 3.0 . 0 .lsst1+ 1 (already installed) done. [ 26 / 46 ] wcslib 5.13 .lsst1+ 2 (already installed) done. [ 27 / 46 ] log4cxx 0.10 . 0 .lsst7 (already installed) done. [ 28 / 46 ] matplotlib 2.0 . 2 (already installed) done. [ 29 / 46 ] ndarray 1.3 . 3 .lsst1 (already installed) done. [ 30 / 46 ] pytest_forked 0.2 .lsst2 (already installed) done. [ 31 / 46 ] pytest_session2file 0.1 . 9 + 3 (already installed) done. [ 32 / 46 ] python_mccabe 0.6 . 1 + 3 (already installed) done. [ 33 / 46 ] flake8 3.5 . 0 + 2 (already installed) done. [ 34 / 46 ] pytest_xdist 1.20 . 1 .lsst2 (already installed) done. [ 35 / 46 ] pytest_flake8 0.9 . 1 + 2 (already installed) done. [ 36 / 46 ] sconsUtils 14.0 - 15 -g79b7e05 (already installed) done. [ 37 / 46 ] astshim 14.0 - 11 -g0e62142+ 3 done. [ 38 / 46 ] base 14.0 - 8 -g7f6dd6b+ 6 (already installed) done. [ 39 / 46 ] pex_exceptions 14.0 - 1 -g13ef843+ 9 (already installed) done. [ 40 / 46 ] utils 14.0 - 8 -g1f982c9 (already installed) done. [ 41 / 46 ] daf_base 14.0 - 4 -ge2d7f21+ 8 (already installed) done. [ 42 / 46 ] log 14.0 - 2 -ga5af9b6+ 8 (already installed) done. [ 43 / 46 ] pex_policy 14.0 - 1 -g4b114ac+ 11 (already installed) done. [ 44 / 46 ] daf_persistence 14.0 - 14 -g87d16e8+ 7 (already installed) done. [ 45 / 46 ] pex_config 14.0 - 2 -g319577b+ 8 (already installed) done. [ 46 / 46 ] afw 14.0 - 35 -g29d3a87a7+ 1 done.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I've merged a few house keeping updates to the jenkins job that builds the release images, added some basic build "provenance" docker labels to the image, and changed the the doc delete to be only doc/html. Once I've gotten the bugs shaken out of these changes, I'll rebuild the last 4 weeklies.

          Show
          jhoblitt Joshua Hoblitt added a comment - I've merged a few house keeping updates to the jenkins job that builds the release images, added some basic build "provenance" docker labels to the image, and changed the the doc delete to be only doc/html . Once I've gotten the bugs shaken out of these changes, I'll rebuild the last 4 weeklies.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The tags 7-stack-lsst_distrib-w_2018_0[1234] have been repushed. Also note that jenkins is now pushing each image under a second tag that has a timestamp appended to it.

          Show
          jhoblitt Joshua Hoblitt added a comment - The tags 7-stack-lsst_distrib-w_2018_0 [1234] have been repushed. Also note that jenkins is now pushing each image under a second tag that has a timestamp appended to it.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Heather Kelly I have pushed a example of installing lsst_sims from mixed binary + source packages here:

          https://github.com/lsst-sqre/docker-tarballs/tree/sims_demo

          Note that although the resulting image is ~6GiB, it is not buildable in the default container size of 10GiB.  I changed the dockerd default to 50GiB after the first try ran out of space but didn't work out what the high water mark was.

          I've publicly published the resulting image as:

          docker.io/lsstsqre/simsdemo:latest

          Eg., docker run -ti docker.io/lsstsqre/simsdemo

           

          Show
          jhoblitt Joshua Hoblitt added a comment - Heather Kelly I have pushed a example of installing lsst_sims from mixed binary + source packages here: https://github.com/lsst-sqre/docker-tarballs/tree/sims_demo Note that although the resulting image is ~6GiB, it is not buildable in the default container size of 10GiB.  I changed the dockerd default to 50GiB after the first try ran out of space but didn't work out what the high water mark was. I've publicly published the resulting image as: docker.io/lsstsqre/simsdemo:latest Eg., docker run -ti docker.io/lsstsqre/simsdemo  
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I assume that this issue is well resolved but will reopen this issue for any related problems.

          Show
          jhoblitt Joshua Hoblitt added a comment - I assume that this issue is well resolved but will reopen this issue for any related problems.
          Hide
          hkelly Heather Kelly added a comment -

          Thanks Joshua Hoblitt Over the weekend, I took the rebuilt w_2018_03 Docker image as a base and built the lsst_sims on top of it, resulting in a 3 GiB compressed Docker image (it was ~7 GiB before compression) I put it on Dockerhub: https://hub.docker.com/r/lsstdesc/stack-sims/tags/

          Is there a reason to prefer grabbing the binaries and building lsst_sims on top of that instead?  In any case, your Dockerfile will allow me to make sure I strip and remove extraneous files in a way that's consistent with your example.

           

          Show
          hkelly Heather Kelly added a comment - Thanks Joshua Hoblitt Over the weekend, I took the rebuilt w_2018_03 Docker image as a base and built the lsst_sims on top of it, resulting in a 3 GiB compressed Docker image (it was ~7 GiB before compression) I put it on Dockerhub: https://hub.docker.com/r/lsstdesc/stack-sims/tags/ Is there a reason to prefer grabbing the binaries and building lsst_sims on top of that instead?  In any case, your Dockerfile will allow me to make sure I strip and remove extraneous files in a way that's consistent with your example.  

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Frossie Economou, Heather Kelly, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel