# doxygen.lsst.codes master index.html page broken

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.5
• Team:
SQuaRE

#### Description

Sometime between 2017-11-30T00:01:08-0800 and 2017-12-01T08:11:54Z, the index.html page at doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc became the pipe_base mainpage instead of the lsstDoxygen mainpage. No changes were made to either of those packages at that time, and a manual execution of makeDocs lsst_apps produces the correct index.html file. I note that the logs for the release/run-rebuild pipeline before that date (e.g. run 435) show the push to AWS containing both index.html and pipe_base.html, while those after that date (e.g. run 445) show only the former and not the latter. I'm also not sure where the push to doxygen.lsst.codes/stack/doxygen/xlink_master_[date+time] occurs, but that is showing the same problem.

#### Attachments

1. index.html
26 kB

#### Activity

No builds found.
Kian-Tat Lim created issue -
John Swinbank made changes -
Field Original Value New Value
Team SQuaRE [ 10302 ]
Priority Undefined [ 10000 ] Critical [ 2 ]
Hide
Joshua Hoblitt added a comment -

I would be quiet surprised if pipe_base.html was being accidentally renamed to index.html and the console output does show index.html being pushed to s3 last night:

https://ci.lsst.codes/blue/rest/organizations/jenkins/pipelines/release/pipelines/run-rebuild/runs/506/nodes/261/steps/274/log/?start=0

 upload: home/public_html/doxygen/x_masterDoxyDoc/index.html to s3://****/stack/doxygen/x_masterDoxyDoc/index.html 

I'm wonder if there is some sort of environmental sensitivity...

Show
Joshua Hoblitt added a comment - I would be quiet surprised if pipe_base.html was being accidentally renamed to index.html and the console output does show index.html being pushed to s3 last night: https://ci.lsst.codes/blue/rest/organizations/jenkins/pipelines/release/pipelines/run-rebuild/runs/506/nodes/261/steps/274/log/?start=0 upload: home/public_html/doxygen/x_masterDoxyDoc/index.html to s3: //****/stack/doxygen/x_masterDoxyDoc/index.html I'm wonder if there is some sort of environmental sensitivity...
Joshua Hoblitt made changes -
 Watchers John Swinbank, Joshua Hoblitt, Kian-Tat Lim [ John Swinbank, Joshua Hoblitt, Kian-Tat Lim ] John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim [ John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim ]
Hide
Joshua Hoblitt added a comment - - edited

The file on disk does indeed match what is on s3:

 $sha256sum index.html  e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 index.html   $ curl -sSL http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/index.html | sha256sum  e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 - 

Show
Joshua Hoblitt added a comment - - edited The file on disk does indeed match what is on s3: /home/jenkins-slave/snowflake/release/home/public_html/doxygen/xlink_master_2017_12_19_10.02.50/index.html $sha256sum index.html e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 index.html$ curl -sSL http: //doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/index.html | sha256sum e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 -
Joshua Hoblitt made changes -
 Attachment index.html [ 30650 ]
Joshua Hoblitt made changes -
 Status To Do [ 10001 ] In Progress [ 3 ]
Joshua Hoblitt made changes -
 Epic Link DM-12784 [ 36402 ]
Hide
Kian-Tat Lim added a comment -

Then there must be something strange about the generation via Jenkins, because the index page is correct when I manually clone, scons, and makeDocs.

Show
Kian-Tat Lim added a comment - Then there must be something strange about the generation via Jenkins, because the index page is correct when I manually clone, scons, and makeDocs.
Hide
Joshua Hoblitt added a comment -

Kian-Tat Lim Also note that we have never used lsst-apps as the doxygen base package. AFAIK, it has always been datarel:

This was going to be changed in DM-2948 but it seems to not have been implemented.

Show
Joshua Hoblitt added a comment - Kian-Tat Lim Also note that we have never used lsst-apps as the doxygen base package. AFAIK, it has always been datarel: https://github.com/lsst-sqre/ci-scripts/blob/master/create_xlinkdocs.sh#L133-L141 This was going to be changed in DM-2948 but it seems to not have been implemented.
Hide
Joshua Hoblitt added a comment -

create_xlinkdocs.sh is tightly coupled to lsstsw, which irritatingly prevents it from being run against the nightly container. I am doing a clean build in a container trying to isolate this but it will probably take hours to finish.

Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.

Show
Joshua Hoblitt added a comment - create_xlinkdocs.sh is tightly coupled to lsstsw , which irritatingly prevents it from being run against the nightly container. I am doing a clean build in a container trying to isolate this but it will probably take hours to finish. Jonathan Sick is there any reason we can't install lsst/lsstDoxygen ? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.
Hide
Jonathan Sick added a comment -

Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.

I don't really have any experience with lsst/lsstDoxygen, so I can't say.

If you think you can change the Doxygen build for the better, go for it.

Show
Jonathan Sick added a comment - Jonathan Sick is there any reason we can't install lsst/lsstDoxygen ? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images. I don't really have any experience with lsst/lsstDoxygen , so I can't say. If you think you can change the Doxygen build for the better, go for it.
Hide
Joshua Hoblitt added a comment -

I've attempted to some test builds by hand to isolate the env. master was broken when I tried, w.2017.48 was the oldest build I could get to run, and it also has lsst_base as the landing page. I was unable to build even v14.0 due to the setuptools_scm shenanigans that are being addressed in DM-13061.

Eg.

 docker run -ti -v $(pwd):$(pwd) -w $(pwd) docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 git clone https://github.com/lsst/lsstsw git clone https://github.com/lsst-sqre/ci-scripts LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.48 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh 

As I was unable to build a tag prior to december 1st, this is only able to prove the change in behavior isn't related to env vars under jenkins.

Show
Joshua Hoblitt added a comment - I've attempted to some test builds by hand to isolate the env. master was broken when I tried, w.2017.48 was the oldest build I could get to run, and it also has lsst_base as the landing page. I was unable to build even v14.0 due to the setuptools_scm shenanigans that are being addressed in DM-13061 . Eg. docker run -ti -v $(pwd):$(pwd) -w $(pwd) docker.io/lsstsqre/centos: 7 -stackbase-devtoolset- 6 git clone https: //github.com/lsst/lsstsw git clone https: //github.com/lsst-sqre/ci-scripts LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.48 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh As I was unable to build a tag prior to december 1st, this is only able to prove the change in behavior isn't related to env vars under jenkins.
Hide
Joshua Hoblitt added a comment -

I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.

 LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh   LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh 

The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.

 # diff -u index.html-lsst_apps index.html-datarel  --- index.html-lsst_apps 2017-12-19 23:45:24.930072951 +0000 +++ index.html-datarel 2017-12-19 23:49:11.890783323 +0000 @@ -28,7 +28,7 @@ 

LSSTApplications -  11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5 +  11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5 

LSSTDataManagementBasePackage

  @@ -214,7 +214,7 @@      

lsst/lsstsw has not changed
lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
lsst/doxygen has not changed
lsst/lsstDoxygen has not changed
lsst/base has not changed
lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.

The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against.

Show
Joshua Hoblitt added a comment - I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps. LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=lsst_apps BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps . # diff -u index.html-lsst_apps index.html-datarel --- index.html-lsst_apps 2017 - 12 - 19 23 : 45 : 24.930072951 + 0000 +++ index.html-datarel 2017 - 12 - 19 23 : 49 : 11.890783323 + 0000 @@ - 28 , 7 + 28 , 7 @@ <tr style= "height: 56px;" > <td id= "projectalign" style= "padding-left: 0.5em;" > <div id= "projectname" >LSSTApplications - &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 19 , 14.0 + 28 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g2fa83af+ 26 , 14.0 - 1 -g3019bb2+ 13 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 1 -ge6e5c2d+ 17 , 14.0 - 1 -ge993229+ 26 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> + &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 28 , 14.0 + 31 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> </div> <div id= "projectbrief" >LSSTDataManagementBasePackage</div> </td> @@ - 214 , 7 + 214 , 7 @@ </div></div><!-- contents --> <!-- start footer part --> <hr class = "footer" /><address class = "footer" ><small> -Generated on Tue Dec 19 2017 23 : 03 : 34 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > +Generated on Tue Dec 19 2017 23 : 48 : 15 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > <img class = "footer" src= "doxygen.png" alt= "doxygen" /> </a> 1.8 . 13 </small></address> lsst/lsstsw has not changed lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed lsst/doxygen has not changed lsst/lsstDoxygen has not changed lsst/base has not changed lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st. The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against.
Hide
Joshua Hoblitt added a comment -

I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.

 LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh   LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh 

The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.

 # diff -u index.html-lsst_apps index.html-datarel  --- index.html-lsst_apps 2017-12-19 23:45:24.930072951 +0000 +++ index.html-datarel 2017-12-19 23:49:11.890783323 +0000 @@ -28,7 +28,7 @@ 

LSSTApplications -  11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5 +  11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5 

LSSTDataManagementBasePackage

  @@ -214,7 +214,7 @@      

lsst/lsstsw has not changed
lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
lsst/doxygen has not changed
lsst/lsstDoxygen has not changed
lsst/base has not changed
lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.

The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against or perhaps some env vars. The migration to devtoolset-6 came a few days later: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/301 I also can't rule out that a change in some other random product has caused doxygen to change behavior – I need to test with @timj's eups patch to rule that out.

  ldd doxygen   linux-vdso.so.1 => (0x00007ffdbf1fd000)  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1e2d836000)  libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1e2d52d000)  libm.so.6 => /lib64/libm.so.6 (0x00007f1e2d22b000)  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1e2d015000)  libc.so.6 => /lib64/libc.so.6 (0x00007f1e2cc51000)  /lib64/ld-linux-x86-64.so.2 (0x0000555f40fbb000) 

Choices are the linux kernel, glibc, or the gcc 4.8.5 shared libs. (also note that, oddly, doxygen linking against the system gcc libs instead of /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/32/libgcc_s.so).

The kernel changed from 3.10.0-693.5.2 -> 3.10.0-693.5.2 on Nov 28th but I would be shocked if the kernel stub functions would cause this kind of breakage.

The jenkins VM versions:

 $rpm -q glibc gcc glibc-2.17-106.el7_2.1.x86_64 gcc-4.8.5-4.el7.x86_64  Versions in docker.io/lsstsqre/centos:7-stackbase (Switched to shortly before devtoolset-6 was implemented):  # rpm -q glibc gcc glibc-2.17-196.el7.x86_64 gcc-4.8.5-16.el7.x86_64  Versions in docker.io/lsstsqre/centos:7-stackbase-devtoolset-6:  # rpm -q glibc gcc glibc-2.17-196.el7.x86_64 gcc-4.8.5-16.el7.x86_64  I will repeat the same test directly on a jenkins build VM image to check for glibc/gcc changes as a cause. Kian-Tat Lim Could you provide instructions to reproduce the original doxygen structure on centos-7, preferably in the docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 docker image? I am not very familiar with the doxygen build and I'm at a bit of a loss. Show Joshua Hoblitt added a comment - I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps. LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=lsst_apps BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.49 WORKSPACE=PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps . # diff -u index.html-lsst_apps index.html-datarel --- index.html-lsst_apps 2017 - 12 - 19 23 : 45 : 24.930072951 + 0000 +++ index.html-datarel 2017 - 12 - 19 23 : 49 : 11.890783323 + 0000 @@ - 28 , 7 + 28 , 7 @@ <tr style= "height: 56px;" > <td id= "projectalign" style= "padding-left: 0.5em;" > <div id= "projectname" >LSSTApplications - &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 19 , 14.0 + 28 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g2fa83af+ 26 , 14.0 - 1 -g3019bb2+ 13 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 1 -ge6e5c2d+ 17 , 14.0 - 1 -ge993229+ 26 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> + &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 28 , 14.0 + 31 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> </div> <div id= "projectbrief" >LSSTDataManagementBasePackage</div> </td> @@ - 214 , 7 + 214 , 7 @@ </div></div><!-- contents --> <!-- start footer part --> <hr class = "footer" /><address class = "footer" ><small> -Generated on Tue Dec 19 2017 23 : 03 : 34 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > +Generated on Tue Dec 19 2017 23 : 48 : 15 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > <img class = "footer" src= "doxygen.png" alt= "doxygen" /> </a> 1.8 . 13 </small></address> lsst/lsstsw has not changed lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed lsst/doxygen has not changed lsst/lsstDoxygen has not changed lsst/base has not changed lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st. The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against or perhaps some env vars. The migration to devtoolset-6 came a few days later: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/301 I also can't rule out that a change in some other random product has caused doxygen to change behavior – I need to test with @timj's eups patch to rule that out. ldd doxygen linux-vdso.so. 1 => ( 0x00007ffdbf1fd000 ) libpthread.so. 0 => /lib64/libpthread.so. 0 ( 0x00007f1e2d836000 ) libstdc++.so. 6 => /lib64/libstdc++.so. 6 ( 0x00007f1e2d52d000 ) libm.so. 6 => /lib64/libm.so. 6 ( 0x00007f1e2d22b000 ) libgcc_s.so. 1 => /lib64/libgcc_s.so. 1 ( 0x00007f1e2d015000 ) libc.so. 6 => /lib64/libc.so. 6 ( 0x00007f1e2cc51000 ) /lib64/ld-linux-x86- 64 .so. 2 ( 0x0000555f40fbb000 ) Choices are the linux kernel, glibc, or the gcc 4.8.5 shared libs. (also note that, oddly, doxygen linking against the system gcc libs instead of /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/32/libgcc_s.so ). The kernel changed from 3.10.0-693.5.2 -> 3.10.0-693.5.2 on Nov 28th but I would be shocked if the kernel stub functions would cause this kind of breakage. The jenkins VM versions: rpm -q glibc gcc glibc- 2.17 - 106 .el7_2. 1 .x86_64 gcc- 4.8 . 5 - 4 .el7.x86_64 Versions in docker.io/lsstsqre/centos:7-stackbase (Switched to shortly before devtoolset-6 was implemented): # rpm -q glibc gcc glibc- 2.17 - 196 .el7.x86_64 gcc- 4.8 . 5 - 16 .el7.x86_64 Versions in docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 : # rpm -q glibc gcc glibc- 2.17 - 196 .el7.x86_64 gcc- 4.8 . 5 - 16 .el7.x86_64 I will repeat the same test directly on a jenkins build VM image to check for glibc/gcc changes as a cause. Kian-Tat Lim Could you provide instructions to reproduce the original doxygen structure on centos-7, preferably in the docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 docker image? I am not very familiar with the doxygen build and I'm at a bit of a loss.
Joshua Hoblitt made changes -
 Story Points 1.5
Show
Hide
Joshua Hoblitt added a comment -

Updating the gcc/glibc versions in the vm image to match the container seems to have had no change. Either there is some other package involved (perhaps build time only) or env vars are involved.

Show
Joshua Hoblitt added a comment - Updating the gcc/glibc versions in the vm image to match the container seems to have had no change. Either there is some other package involved (perhaps build time only) or env vars are involved.
Hide
Joshua Hoblitt added a comment - - edited

As a last resort, I'm testing changing the doxygen base package from datarel -> lsst_distrib, as that would at least resolve DM-2948.

Show
Joshua Hoblitt added a comment - - edited As a last resort, I'm testing changing the doxygen base package from datarel -> lsst_distrib , as that would at least resolve DM-2948 .
Joshua Hoblitt made changes -
 Link This issue relates to DM-2948 [ DM-2948 ]
Hide
Tim Jenness added a comment -

For the record this is still broken in March. Are we now using lsst_distrib as the anchor package? I'd like to remove datarel because it brings in some mariadbclient dependencies that we'd rather get rid of.

Show
Tim Jenness added a comment - For the record this is still broken in March. Are we now using lsst_distrib as the anchor package? I'd like to remove datarel because it brings in some mariadbclient dependencies that we'd rather get rid of.
Tim Jenness made changes -
 Link This issue relates to DM-1040 [ DM-1040 ]
Hide
Joshua Hoblitt added a comment -

Per discussion on DM-2948, I have merged the removal of datarel from create_xlinkdocs.sh.

Show
Joshua Hoblitt added a comment - Per discussion on DM-2948 , I have merged the removal of datarel from create_xlinkdocs.sh .
Joshua Hoblitt made changes -
 Epic Link DM-12784 [ 36402 ] DM-13784 [ 39331 ]
Joshua Hoblitt made changes -
 Status In Progress [ 3 ] To Do [ 10001 ]
Joshua Hoblitt made changes -
 Risk Score 0
Joshua Hoblitt made changes -
 Epic Link DM-13784 [ 39331 ] DM-11756 [ 34208 ]
Hide
Joshua Hoblitt added a comment -

I'm at a loss as for how to address this and am returning the issue to the backlog for the time being.

Show
Joshua Hoblitt added a comment - I'm at a loss as for how to address this and am returning the issue to the backlog for the time being.
Joshua Hoblitt made changes -
 Epic Link DM-11756 [ 34208 ] DM-5417 [ 23241 ]
Hide
Kian-Tat Lim added a comment -

Fixing the Python 3.7 incompatibility (having to do with a regexp substitution for \mainpage commands) has fixed this!

Show
Kian-Tat Lim added a comment - Fixing the Python 3.7 incompatibility (having to do with a regexp substitution for \mainpage commands) has fixed this!
Kian-Tat Lim made changes -
 Resolution Done [ 10000 ] Status To Do [ 10001 ] Done [ 10002 ]

#### People

Assignee:
Joshua Hoblitt
Reporter:
Kian-Tat Lim
Watchers:
John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Krzysztof Findeisen, Tim Jenness