Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Continuous Integration
-
Labels:None
-
Story Points:1.5
-
Epic Link:
-
Team:SQuaRE
Description
Sometime between 2017-11-30T00:01:08-0800 and 2017-12-01T08:11:54Z, the index.html page at doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc became the pipe_base mainpage instead of the lsstDoxygen mainpage. No changes were made to either of those packages at that time, and a manual execution of makeDocs lsst_apps produces the correct index.html file. I note that the logs for the release/run-rebuild pipeline before that date (e.g. run 435) show the push to AWS containing both index.html and pipe_base.html, while those after that date (e.g. run 445) show only the former and not the latter. I'm also not sure where the push to doxygen.lsst.codes/stack/doxygen/xlink_master_[date+time] occurs, but that is showing the same problem.
Attachments
Issue Links
Activity
The file on disk does indeed match what is on s3:
/home/jenkins-slave/snowflake/release/home/public_html/doxygen/xlink_master_2017_12_19_10.02.50/index.html
$ sha256sum index.html
|
e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 index.html
|
$ curl -sSL http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/index.html | sha256sum |
e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 -
|
Then there must be something strange about the generation via Jenkins, because the index page is correct when I manually clone, scons, and makeDocs.
Kian-Tat Lim Also note that we have never used lsst-apps as the doxygen base package. AFAIK, it has always been datarel:
https://github.com/lsst-sqre/ci-scripts/blob/master/create_xlinkdocs.sh#L133-L141
This was going to be changed in DM-2948 but it seems to not have been implemented.
create_xlinkdocs.sh is tightly coupled to lsstsw, which irritatingly prevents it from being run against the nightly container. I am doing a clean build in a container trying to isolate this but it will probably take hours to finish.
Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.
Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.
I don't really have any experience with lsst/lsstDoxygen, so I can't say.
If you think you can change the Doxygen build for the better, go for it.
I've attempted to some test builds by hand to isolate the env. master was broken when I tried, w.2017.48 was the oldest build I could get to run, and it also has lsst_base as the landing page. I was unable to build even v14.0 due to the setuptools_scm shenanigans that are being addressed in DM-13061.
Eg.
docker run -ti -v $(pwd):$(pwd) -w $(pwd) docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 |
git clone https://github.com/lsst/lsstsw |
git clone https://github.com/lsst-sqre/ci-scripts |
LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.48 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh |
As I was unable to build a tag prior to december 1st, this is only able to prove the change in behavior isn't related to env vars under jenkins.
I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.
LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh |
LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh |
The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.
# diff -u index.html-lsst_apps index.html-datarel
|
--- index.html-lsst_apps 2017-12-19 23:45:24.930072951 +0000 |
+++ index.html-datarel 2017-12-19 23:49:11.890783323 +0000 |
@@ -28,7 +28,7 @@ |
<tr style="height: 56px;"> |
<td id="projectalign" style="padding-left: 0.5em;"> |
<div id="projectname">LSSTApplications |
-  <span id="projectnumber">11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span> |
+  <span id="projectnumber">11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span> |
</div>
|
<div id="projectbrief">LSSTDataManagementBasePackage</div> |
</td>
|
@@ -214,7 +214,7 @@ |
</div></div><!-- contents -->
|
<!-- start footer part -->
|
<hr class="footer"/><address class="footer"><small> |
-Generated on Tue Dec 19 2017 23:03:34 for LSSTApplications by  <a href="http://www.doxygen.org/index.html"> |
+Generated on Tue Dec 19 2017 23:48:15 for LSSTApplications by  <a href="http://www.doxygen.org/index.html"> |
<img class="footer" src="doxygen.png" alt="doxygen"/> |
</a> 1.8.13 |
</small></address>
|
lsst/lsstsw has not changed
lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
lsst/doxygen has not changed
lsst/lsstDoxygen has not changed
lsst/base has not changed
lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.
The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against.
I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.
LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh |
LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh |
The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.
# diff -u index.html-lsst_apps index.html-datarel
|
--- index.html-lsst_apps 2017-12-19 23:45:24.930072951 +0000 |
+++ index.html-datarel 2017-12-19 23:49:11.890783323 +0000 |
@@ -28,7 +28,7 @@ |
<tr style="height: 56px;"> |
<td id="projectalign" style="padding-left: 0.5em;"> |
<div id="projectname">LSSTApplications |
-  <span id="projectnumber">11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span> |
+  <span id="projectnumber">11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span> |
</div>
|
<div id="projectbrief">LSSTDataManagementBasePackage</div> |
</td>
|
@@ -214,7 +214,7 @@ |
</div></div><!-- contents -->
|
<!-- start footer part -->
|
<hr class="footer"/><address class="footer"><small> |
-Generated on Tue Dec 19 2017 23:03:34 for LSSTApplications by  <a href="http://www.doxygen.org/index.html"> |
+Generated on Tue Dec 19 2017 23:48:15 for LSSTApplications by  <a href="http://www.doxygen.org/index.html"> |
<img class="footer" src="doxygen.png" alt="doxygen"/> |
</a> 1.8.13 |
</small></address>
|
lsst/lsstsw has not changed
lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
lsst/doxygen has not changed
lsst/lsstDoxygen has not changed
lsst/base has not changed
lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.
The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against or perhaps some env vars. The migration to devtoolset-6 came a few days later: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/301 I also can't rule out that a change in some other random product has caused doxygen to change behavior – I need to test with @timj's eups patch to rule that out.
ldd doxygen
|
linux-vdso.so.1 => (0x00007ffdbf1fd000) |
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1e2d836000) |
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1e2d52d000) |
libm.so.6 => /lib64/libm.so.6 (0x00007f1e2d22b000) |
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1e2d015000) |
libc.so.6 => /lib64/libc.so.6 (0x00007f1e2cc51000) |
/lib64/ld-linux-x86-64.so.2 (0x0000555f40fbb000) |
Choices are the linux kernel, glibc, or the gcc 4.8.5 shared libs. (also note that, oddly, doxygen linking against the system gcc libs instead of /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/32/libgcc_s.so).
The kernel changed from 3.10.0-693.5.2 -> 3.10.0-693.5.2 on Nov 28th but I would be shocked if the kernel stub functions would cause this kind of breakage.
The jenkins VM versions:
$ rpm -q glibc gcc
|
glibc-2.17-106.el7_2.1.x86_64 |
gcc-4.8.5-4.el7.x86_64 |
Versions in docker.io/lsstsqre/centos:7-stackbase (Switched to shortly before devtoolset-6 was implemented):
# rpm -q glibc gcc
|
glibc-2.17-196.el7.x86_64 |
gcc-4.8.5-16.el7.x86_64 |
Versions in docker.io/lsstsqre/centos:7-stackbase-devtoolset-6:
# rpm -q glibc gcc
|
glibc-2.17-196.el7.x86_64 |
gcc-4.8.5-16.el7.x86_64 |
I will repeat the same test directly on a jenkins build VM image to check for glibc/gcc changes as a cause.
Kian-Tat Lim Could you provide instructions to reproduce the original doxygen structure on centos-7, preferably in the docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 docker image? I am not very familiar with the doxygen build and I'm at a bit of a loss.
Running directly on the old jenkins el7 image seems produce the expected format.
<p>Mainpages in sub-packages: </p><ul>
|
<li><a class="el" href="afw.html">lsst::afw; the LSST Application Framework</a> </li> |
<li><a class="el" href="astshim.html">astshim</a> </li> |
<li><a class="el" href="base.html">lsst::base; Basic LSST Functionality</a> </li> |
<li><a class="el" href="coadd_chisquared.html">lsst::coadd::chisquared; Support for chi-squared coadds</a> </li> |
<li><a class="el" href="coadd_utils.html">lsst::coadd::utils; Support for coadds</a> </li> |
<li><a class="el" href="log.html">lsst::log; the LSST Logging Framework</a> </li> |
<li><a class="el" href="meas_algorithms.html">lsst::meas::algorithms; support for measurement algorithms</a> </li> |
<li><a class="el" href="meas_astrom.html">lsst::meas::astrom; Astrometric Calibration</a> </li> |
<li><a class="el" href="meas_base.html">lsst::meas::base; redesigned support for measurement tasks and algorithm plugins</a> </li> |
<li><a class="el" href="meas_extensions_astrometry_net.html">lsst::meas::extensions::astromentryNet; Astrometric Calibration using astrometry.net</a> </li> |
<li><a class="el" href="obs_test.html">lsst::obs::test; Simple camera and data repository for testing</a> </li> |
<li><a class="el" href="pex_config.html">lsst::pex::config: configuration data management</a> </li> |
<li><a class="el" href="pex_exceptions.html">lsst::pex::exceptions; LSST Exceptions</a> </li> |
<li><a class="el" href="pex_policy.html">lsst::pex::policy: configuration data management</a> </li> |
<li><a class="el" href="pipe_base.html">lsst::pipe::base; Base package for pipeline tasks</a> </li> |
<li><a class="el" href="pipe_tasks.html">lsst::pipe::tasks: Pipeline tasks</a> </li> |
<li><a class="el" href="skymap.html">lsst::skymap; sky pixelization</a> </li> |
<li><a class="el" href="utils.html">lsst::utils; utility classes</a> </li> |
</ul>
|
Updating the gcc/glibc versions in the vm image to match the container seems to have had no change. Either there is some other package involved (perhaps build time only) or env vars are involved.
As a last resort, I'm testing changing the doxygen base package from datarel -> lsst_distrib, as that would at least resolve DM-2948.
For the record this is still broken in March. Are we now using lsst_distrib as the anchor package? I'd like to remove datarel because it brings in some mariadbclient dependencies that we'd rather get rid of.
Per discussion on DM-2948, I have merged the removal of datarel from create_xlinkdocs.sh.
I'm at a loss as for how to address this and am returning the issue to the backlog for the time being.
Fixing the Python 3.7 incompatibility (having to do with a regexp substitution for \mainpage commands) has fixed this!
I would be quiet surprised if pipe_base.html was being accidentally renamed to index.html and the console output does show index.html being pushed to s3 last night:
https://ci.lsst.codes/blue/rest/organizations/jenkins/pipelines/release/pipelines/run-rebuild/runs/506/nodes/261/steps/274/log/?start=0
I'm wonder if there is some sort of environmental sensitivity...