Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13074

doxygen.lsst.codes master index.html page broken

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Sometime between 2017-11-30T00:01:08-0800 and 2017-12-01T08:11:54Z, the index.html page at doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc became the pipe_base mainpage instead of the lsstDoxygen mainpage. No changes were made to either of those packages at that time, and a manual execution of makeDocs lsst_apps produces the correct index.html file. I note that the logs for the release/run-rebuild pipeline before that date (e.g. run 435) show the push to AWS containing both index.html and pipe_base.html, while those after that date (e.g. run 445) show only the former and not the latter. I'm also not sure where the push to doxygen.lsst.codes/stack/doxygen/xlink_master_[date+time] occurs, but that is showing the same problem.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I would be quiet surprised if pipe_base.html was being accidentally renamed to index.html and the console output does show index.html being pushed to s3 last night:

            https://ci.lsst.codes/blue/rest/organizations/jenkins/pipelines/release/pipelines/run-rebuild/runs/506/nodes/261/steps/274/log/?start=0

            upload: home/public_html/doxygen/x_masterDoxyDoc/index.html to s3://****/stack/doxygen/x_masterDoxyDoc/index.html
            

            I'm wonder if there is some sort of environmental sensitivity...

            Show
            jhoblitt Joshua Hoblitt added a comment - I would be quiet surprised if pipe_base.html was being accidentally renamed to index.html and the console output does show index.html being pushed to s3 last night: https://ci.lsst.codes/blue/rest/organizations/jenkins/pipelines/release/pipelines/run-rebuild/runs/506/nodes/261/steps/274/log/?start=0 upload: home/public_html/doxygen/x_masterDoxyDoc/index.html to s3: //****/stack/doxygen/x_masterDoxyDoc/index.html I'm wonder if there is some sort of environmental sensitivity...
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            The file on disk does indeed match what is on s3:

            /home/jenkins-slave/snowflake/release/home/public_html/doxygen/xlink_master_2017_12_19_10.02.50/index.html

            $ sha256sum index.html 
            e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8  index.html
            

            $ curl -sSL http://doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/index.html | sha256sum 
            e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8  -
            

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited The file on disk does indeed match what is on s3: /home/jenkins-slave/snowflake/release/home/public_html/doxygen/xlink_master_2017_12_19_10.02.50/index.html $ sha256sum index.html e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 index.html $ curl -sSL http: //doxygen.lsst.codes/stack/doxygen/x_masterDoxyDoc/index.html | sha256sum e2b90965df36f25d6121bcd5c75d384b0c41a6b855e54c56b205e436c6a51dc8 -
            Hide
            ktl Kian-Tat Lim added a comment -

            Then there must be something strange about the generation via Jenkins, because the index page is correct when I manually clone, scons, and makeDocs.

            Show
            ktl Kian-Tat Lim added a comment - Then there must be something strange about the generation via Jenkins, because the index page is correct when I manually clone, scons, and makeDocs.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Kian-Tat Lim Also note that we have never used lsst-apps as the doxygen base package. AFAIK, it has always been datarel:

            https://github.com/lsst-sqre/ci-scripts/blob/master/create_xlinkdocs.sh#L133-L141

            This was going to be changed in DM-2948 but it seems to not have been implemented.

            Show
            jhoblitt Joshua Hoblitt added a comment - Kian-Tat Lim Also note that we have never used lsst-apps as the doxygen base package. AFAIK, it has always been datarel: https://github.com/lsst-sqre/ci-scripts/blob/master/create_xlinkdocs.sh#L133-L141 This was going to be changed in DM-2948 but it seems to not have been implemented.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            create_xlinkdocs.sh is tightly coupled to lsstsw, which irritatingly prevents it from being run against the nightly container. I am doing a clean build in a container trying to isolate this but it will probably take hours to finish.

            Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.

            Show
            jhoblitt Joshua Hoblitt added a comment - create_xlinkdocs.sh is tightly coupled to lsstsw , which irritatingly prevents it from being run against the nightly container. I am doing a clean build in a container trying to isolate this but it will probably take hours to finish. Jonathan Sick is there any reason we can't install lsst/lsstDoxygen ? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.
            Hide
            jsick Jonathan Sick added a comment -

            Jonathan Sick is there any reason we can't install lsst/lsstDoxygen? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images.

            I don't really have any experience with lsst/lsstDoxygen, so I can't say.

            If you think you can change the Doxygen build for the better, go for it.

            Show
            jsick Jonathan Sick added a comment - Jonathan Sick is there any reason we can't install lsst/lsstDoxygen ? If we inject it into the product build list when the doc build is enabled, it would allow a slight simplification of create_xlinkdocs.sh and the exact version used would be eups published / end up in the docker release images. I don't really have any experience with lsst/lsstDoxygen , so I can't say. If you think you can change the Doxygen build for the better, go for it.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've attempted to some test builds by hand to isolate the env. master was broken when I tried, w.2017.48 was the oldest build I could get to run, and it also has lsst_base as the landing page. I was unable to build even v14.0 due to the setuptools_scm shenanigans that are being addressed in DM-13061.

            Eg.

            docker run -ti -v $(pwd):$(pwd) -w $(pwd) docker.io/lsstsqre/centos:7-stackbase-devtoolset-6
            git clone https://github.com/lsst/lsstsw
            git clone https://github.com/lsst-sqre/ci-scripts
            LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.48 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh
            

            As I was unable to build a tag prior to december 1st, this is only able to prove the change in behavior isn't related to env vars under jenkins.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've attempted to some test builds by hand to isolate the env. master was broken when I tried, w.2017.48 was the oldest build I could get to run, and it also has lsst_base as the landing page. I was unable to build even v14.0 due to the setuptools_scm shenanigans that are being addressed in DM-13061 . Eg. docker run -ti -v $(pwd):$(pwd) -w $(pwd) docker.io/lsstsqre/centos: 7 -stackbase-devtoolset- 6 git clone https: //github.com/lsst/lsstsw git clone https: //github.com/lsst-sqre/ci-scripts LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.48 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh As I was unable to build a tag prior to december 1st, this is only able to prove the change in behavior isn't related to env vars under jenkins.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.

            LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh
            

            LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh
            

            The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.

            # diff -u index.html-lsst_apps index.html-datarel 
            --- index.html-lsst_apps	2017-12-19 23:45:24.930072951 +0000
            +++ index.html-datarel	2017-12-19 23:49:11.890783323 +0000
            @@ -28,7 +28,7 @@
              <tr style="height: 56px;">
               <td id="projectalign" style="padding-left: 0.5em;">
                <div id="projectname">LSSTApplications
            -   &#160;<span id="projectnumber">11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span>
            +   &#160;<span id="projectnumber">11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span>
                </div>
                <div id="projectbrief">LSSTDataManagementBasePackage</div>
               </td>
            @@ -214,7 +214,7 @@
             </div></div><!-- contents -->
             <!-- start footer part -->
             <hr class="footer"/><address class="footer"><small>
            -Generated on Tue Dec 19 2017 23:03:34 for LSSTApplications by &#160;<a href="http://www.doxygen.org/index.html">
            +Generated on Tue Dec 19 2017 23:48:15 for LSSTApplications by &#160;<a href="http://www.doxygen.org/index.html">
             <img class="footer" src="doxygen.png" alt="doxygen"/>
             </a> 1.8.13
             </small></address>
            

            lsst/lsstsw has not changed
            lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
            lsst/doxygen has not changed
            lsst/lsstDoxygen has not changed
            lsst/base has not changed
            lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.

            The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps. LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=lsst_apps BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps . # diff -u index.html-lsst_apps index.html-datarel --- index.html-lsst_apps 2017 - 12 - 19 23 : 45 : 24.930072951 + 0000 +++ index.html-datarel 2017 - 12 - 19 23 : 49 : 11.890783323 + 0000 @@ - 28 , 7 + 28 , 7 @@ <tr style= "height: 56px;" > <td id= "projectalign" style= "padding-left: 0.5em;" > <div id= "projectname" >LSSTApplications - &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 19 , 14.0 + 28 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g2fa83af+ 26 , 14.0 - 1 -g3019bb2+ 13 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 1 -ge6e5c2d+ 17 , 14.0 - 1 -ge993229+ 26 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> + &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 28 , 14.0 + 31 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> </div> <div id= "projectbrief" >LSSTDataManagementBasePackage</div> </td> @@ - 214 , 7 + 214 , 7 @@ </div></div><!-- contents --> <!-- start footer part --> <hr class = "footer" /><address class = "footer" ><small> -Generated on Tue Dec 19 2017 23 : 03 : 34 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > +Generated on Tue Dec 19 2017 23 : 48 : 15 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > <img class = "footer" src= "doxygen.png" alt= "doxygen" /> </a> 1.8 . 13 </small></address> lsst/lsstsw has not changed lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed lsst/doxygen has not changed lsst/lsstDoxygen has not changed lsst/base has not changed lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st. The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps.

            LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=lsst_apps BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh
            

            LSST_COMPILER=devtoolset-6 LSST_PYTHON_VERSION=3 PRODUCT=datarel BRANCH=w.2017.49 WORKSPACE=$PWD SKIP_DEMO=true ./ci-scripts/jenkins_wrapper.sh
            

            The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps.

            # diff -u index.html-lsst_apps index.html-datarel 
            --- index.html-lsst_apps	2017-12-19 23:45:24.930072951 +0000
            +++ index.html-datarel	2017-12-19 23:49:11.890783323 +0000
            @@ -28,7 +28,7 @@
              <tr style="height: 56px;">
               <td id="projectalign" style="padding-left: 0.5em;">
                <div id="projectname">LSSTApplications
            -   &#160;<span id="projectnumber">11.0-22-g33de520,14.0+19,14.0+28,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g2fa83af+26,14.0-1-g3019bb2+13,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-1-ge6e5c2d+17,14.0-1-ge993229+26,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span>
            +   &#160;<span id="projectnumber">11.0-22-g33de520,14.0+28,14.0+31,14.0+4,14.0-1-g013352c+11,14.0-1-g13ef843+8,14.0-1-g4b114ac+9,14.0-1-g7257b6a+8,14.0-1-g8b7e855+27,14.0-10-ga7aaa25+4,14.0-12-g233aa8e+9,14.0-14-g87d16e8+5,14.0-18-g0eec5f5+10,14.0-2-g0af5a6c+27,14.0-2-g319577b+6,14.0-2-g8373656+6,14.0-2-ga5af9b6+6,14.0-25-g0e8fc950e+7,14.0-28-gf7234cf5+2,14.0-4-g4cc409d+17,14.0-4-ge2d7f21+6,14.0-5-g86eb1bd+7,14.0-6-g4f52afe+5,14.0-6-ge2c9487+18,14.0-6-gf4bc96c+27,14.0-8-g7f6dd6b+5,14.0-8-gf6dacf9e+5</span>
                </div>
                <div id="projectbrief">LSSTDataManagementBasePackage</div>
               </td>
            @@ -214,7 +214,7 @@
             </div></div><!-- contents -->
             <!-- start footer part -->
             <hr class="footer"/><address class="footer"><small>
            -Generated on Tue Dec 19 2017 23:03:34 for LSSTApplications by &#160;<a href="http://www.doxygen.org/index.html">
            +Generated on Tue Dec 19 2017 23:48:15 for LSSTApplications by &#160;<a href="http://www.doxygen.org/index.html">
             <img class="footer" src="doxygen.png" alt="doxygen"/>
             </a> 1.8.13
             </small></address>
            

            lsst/lsstsw has not changed
            lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed
            lsst/doxygen has not changed
            lsst/lsstDoxygen has not changed
            lsst/base has not changed
            lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st.

            The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against or perhaps some env vars. The migration to devtoolset-6 came a few days later: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/301 I also can't rule out that a change in some other random product has caused doxygen to change behavior – I need to test with @timj's eups patch to rule that out.

             ldd doxygen 
            	linux-vdso.so.1 =>  (0x00007ffdbf1fd000)
            	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1e2d836000)
            	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1e2d52d000)
            	libm.so.6 => /lib64/libm.so.6 (0x00007f1e2d22b000)
            	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1e2d015000)
            	libc.so.6 => /lib64/libc.so.6 (0x00007f1e2cc51000)
            	/lib64/ld-linux-x86-64.so.2 (0x0000555f40fbb000)
            

            Choices are the linux kernel, glibc, or the gcc 4.8.5 shared libs. (also note that, oddly, doxygen linking against the system gcc libs instead of /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/32/libgcc_s.so).

            The kernel changed from 3.10.0-693.5.2 -> 3.10.0-693.5.2 on Nov 28th but I would be shocked if the kernel stub functions would cause this kind of breakage.

            The jenkins VM versions:

            $ rpm -q glibc gcc
            glibc-2.17-106.el7_2.1.x86_64
            gcc-4.8.5-4.el7.x86_64
            

            Versions in docker.io/lsstsqre/centos:7-stackbase (Switched to shortly before devtoolset-6 was implemented):

            # rpm -q glibc gcc
            glibc-2.17-196.el7.x86_64
            gcc-4.8.5-16.el7.x86_64
            

            Versions in docker.io/lsstsqre/centos:7-stackbase-devtoolset-6:

            # rpm -q glibc gcc
            glibc-2.17-196.el7.x86_64
            gcc-4.8.5-16.el7.x86_64
            

            I will repeat the same test directly on a jenkins build VM image to check for glibc/gcc changes as a cause.

            Kian-Tat Lim Could you provide instructions to reproduce the original doxygen structure on centos-7, preferably in the docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 docker image? I am not very familiar with the doxygen build and I'm at a bit of a loss.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've tested producing the docs with both datarel and lsst_apps as makeDocs targets. The results are identical except for eups +versions and timestamps. LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=lsst_apps BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh LSST_COMPILER=devtoolset- 6 LSST_PYTHON_VERSION= 3 PRODUCT=datarel BRANCH=w. 2017.49 WORKSPACE=$PWD SKIP_DEMO= true ./ci-scripts/jenkins_wrapper.sh The former had all of the references to the datarel package in ci-scripts/create_xlinkdocs.sh replaced with lsst_apps . # diff -u index.html-lsst_apps index.html-datarel --- index.html-lsst_apps 2017 - 12 - 19 23 : 45 : 24.930072951 + 0000 +++ index.html-datarel 2017 - 12 - 19 23 : 49 : 11.890783323 + 0000 @@ - 28 , 7 + 28 , 7 @@ <tr style= "height: 56px;" > <td id= "projectalign" style= "padding-left: 0.5em;" > <div id= "projectname" >LSSTApplications - &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 19 , 14.0 + 28 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g2fa83af+ 26 , 14.0 - 1 -g3019bb2+ 13 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 1 -ge6e5c2d+ 17 , 14.0 - 1 -ge993229+ 26 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> + &# 160 ;<span id= "projectnumber" > 11.0 - 22 -g33de520, 14.0 + 28 , 14.0 + 31 , 14.0 + 4 , 14.0 - 1 -g013352c+ 11 , 14.0 - 1 -g13ef843+ 8 , 14.0 - 1 -g4b114ac+ 9 , 14.0 - 1 -g7257b6a+ 8 , 14.0 - 1 -g8b7e855+ 27 , 14.0 - 10 -ga7aaa25+ 4 , 14.0 - 12 -g233aa8e+ 9 , 14.0 - 14 -g87d16e8+ 5 , 14.0 - 18 -g0eec5f5+ 10 , 14.0 - 2 -g0af5a6c+ 27 , 14.0 - 2 -g319577b+ 6 , 14.0 - 2 -g8373656+ 6 , 14.0 - 2 -ga5af9b6+ 6 , 14.0 - 25 -g0e8fc950e+ 7 , 14.0 - 28 -gf7234cf5+ 2 , 14.0 - 4 -g4cc409d+ 17 , 14.0 - 4 -ge2d7f21+ 6 , 14.0 - 5 -g86eb1bd+ 7 , 14.0 - 6 -g4f52afe+ 5 , 14.0 - 6 -ge2c9487+ 18 , 14.0 - 6 -gf4bc96c+ 27 , 14.0 - 8 -g7f6dd6b+ 5 , 14.0 - 8 -gf6dacf9e+ 5 </span> </div> <div id= "projectbrief" >LSSTDataManagementBasePackage</div> </td> @@ - 214 , 7 + 214 , 7 @@ </div></div><!-- contents --> <!-- start footer part --> <hr class = "footer" /><address class = "footer" ><small> -Generated on Tue Dec 19 2017 23 : 03 : 34 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > +Generated on Tue Dec 19 2017 23 : 48 : 15 for LSSTApplications by &# 160 ;<a href= "http://www.doxygen.org/index.html" > <img class = "footer" src= "doxygen.png" alt= "doxygen" /> </a> 1.8 . 13 </small></address> lsst/lsstsw has not changed lsst-sqre/ci-scripts/create_xlinkdocs.sh has not changed lsst/doxygen has not changed lsst/lsstDoxygen has not changed lsst/base has not changed lsst/sconsUtils has recent changes but are too recent to have affected Dec. 1st. The only changed I can think of that was around that time was the switch to running inside of a container: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/284 I suppose that could have slightly changed the version of something doxygen is linking against or perhaps some env vars. The migration to devtoolset-6 came a few days later: https://github.com/lsst-sqre/jenkins-dm-jobs/pull/301 I also can't rule out that a change in some other random product has caused doxygen to change behavior – I need to test with @timj's eups patch to rule that out. ldd doxygen linux-vdso.so. 1 => ( 0x00007ffdbf1fd000 ) libpthread.so. 0 => /lib64/libpthread.so. 0 ( 0x00007f1e2d836000 ) libstdc++.so. 6 => /lib64/libstdc++.so. 6 ( 0x00007f1e2d52d000 ) libm.so. 6 => /lib64/libm.so. 6 ( 0x00007f1e2d22b000 ) libgcc_s.so. 1 => /lib64/libgcc_s.so. 1 ( 0x00007f1e2d015000 ) libc.so. 6 => /lib64/libc.so. 6 ( 0x00007f1e2cc51000 ) /lib64/ld-linux-x86- 64 .so. 2 ( 0x0000555f40fbb000 ) Choices are the linux kernel, glibc, or the gcc 4.8.5 shared libs. (also note that, oddly, doxygen linking against the system gcc libs instead of /opt/rh/devtoolset-6/root/usr/lib/gcc/x86_64-redhat-linux/6.3.1/32/libgcc_s.so ). The kernel changed from 3.10.0-693.5.2 -> 3.10.0-693.5.2 on Nov 28th but I would be shocked if the kernel stub functions would cause this kind of breakage. The jenkins VM versions: $ rpm -q glibc gcc glibc- 2.17 - 106 .el7_2. 1 .x86_64 gcc- 4.8 . 5 - 4 .el7.x86_64 Versions in docker.io/lsstsqre/centos:7-stackbase (Switched to shortly before devtoolset-6 was implemented): # rpm -q glibc gcc glibc- 2.17 - 196 .el7.x86_64 gcc- 4.8 . 5 - 16 .el7.x86_64 Versions in docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 : # rpm -q glibc gcc glibc- 2.17 - 196 .el7.x86_64 gcc- 4.8 . 5 - 16 .el7.x86_64 I will repeat the same test directly on a jenkins build VM image to check for glibc/gcc changes as a cause. Kian-Tat Lim Could you provide instructions to reproduce the original doxygen structure on centos-7, preferably in the docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 docker image? I am not very familiar with the doxygen build and I'm at a bit of a loss.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Running directly on the old jenkins el7 image seems produce the expected format.

            <p>Mainpages in sub-packages: </p><ul>
            <li><a class="el" href="afw.html">lsst::afw; the LSST Application Framework</a> </li>
            <li><a class="el" href="astshim.html">astshim</a> </li>
            <li><a class="el" href="base.html">lsst::base; Basic LSST Functionality</a> </li>
            <li><a class="el" href="coadd_chisquared.html">lsst::coadd::chisquared; Support for chi-squared coadds</a> </li>
            <li><a class="el" href="coadd_utils.html">lsst::coadd::utils; Support for coadds</a> </li>
            <li><a class="el" href="log.html">lsst::log; the LSST Logging Framework</a> </li>
            <li><a class="el" href="meas_algorithms.html">lsst::meas::algorithms; support for measurement algorithms</a> </li>
            <li><a class="el" href="meas_astrom.html">lsst::meas::astrom; Astrometric Calibration</a> </li>
            <li><a class="el" href="meas_base.html">lsst::meas::base; redesigned support for measurement tasks and algorithm plugins</a> </li>
            <li><a class="el" href="meas_extensions_astrometry_net.html">lsst::meas::extensions::astromentryNet; Astrometric Calibration using astrometry.net</a> </li>
            <li><a class="el" href="obs_test.html">lsst::obs::test; Simple camera and data repository for testing</a> </li>
            <li><a class="el" href="pex_config.html">lsst::pex::config: configuration data management</a> </li>
            <li><a class="el" href="pex_exceptions.html">lsst::pex::exceptions; LSST Exceptions</a> </li>
            <li><a class="el" href="pex_policy.html">lsst::pex::policy: configuration data management</a> </li>
            <li><a class="el" href="pipe_base.html">lsst::pipe::base; Base package for pipeline tasks</a> </li>
            <li><a class="el" href="pipe_tasks.html">lsst::pipe::tasks: Pipeline tasks</a> </li>
            <li><a class="el" href="skymap.html">lsst::skymap; sky pixelization</a> </li>
            <li><a class="el" href="utils.html">lsst::utils; utility classes</a> </li>
            </ul>
            

            Show
            jhoblitt Joshua Hoblitt added a comment - Running directly on the old jenkins el7 image seems produce the expected format. <p>Mainpages in sub-packages: </p><ul> <li><a class = "el" href= "afw.html" >lsst::afw; the LSST Application Framework</a> </li> <li><a class = "el" href= "astshim.html" >astshim</a> </li> <li><a class = "el" href= "base.html" >lsst::base; Basic LSST Functionality</a> </li> <li><a class = "el" href= "coadd_chisquared.html" >lsst::coadd::chisquared; Support for chi-squared coadds</a> </li> <li><a class = "el" href= "coadd_utils.html" >lsst::coadd::utils; Support for coadds</a> </li> <li><a class = "el" href= "log.html" >lsst::log; the LSST Logging Framework</a> </li> <li><a class = "el" href= "meas_algorithms.html" >lsst::meas::algorithms; support for measurement algorithms</a> </li> <li><a class = "el" href= "meas_astrom.html" >lsst::meas::astrom; Astrometric Calibration</a> </li> <li><a class = "el" href= "meas_base.html" >lsst::meas::base; redesigned support for measurement tasks and algorithm plugins</a> </li> <li><a class = "el" href= "meas_extensions_astrometry_net.html" >lsst::meas::extensions::astromentryNet; Astrometric Calibration using astrometry.net</a> </li> <li><a class = "el" href= "obs_test.html" >lsst::obs::test; Simple camera and data repository for testing</a> </li> <li><a class = "el" href= "pex_config.html" >lsst::pex::config: configuration data management</a> </li> <li><a class = "el" href= "pex_exceptions.html" >lsst::pex::exceptions; LSST Exceptions</a> </li> <li><a class = "el" href= "pex_policy.html" >lsst::pex::policy: configuration data management</a> </li> <li><a class = "el" href= "pipe_base.html" >lsst::pipe::base; Base package for pipeline tasks</a> </li> <li><a class = "el" href= "pipe_tasks.html" >lsst::pipe::tasks: Pipeline tasks</a> </li> <li><a class = "el" href= "skymap.html" >lsst::skymap; sky pixelization</a> </li> <li><a class = "el" href= "utils.html" >lsst::utils; utility classes</a> </li> </ul>
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Updating the gcc/glibc versions in the vm image to match the container seems to have had no change. Either there is some other package involved (perhaps build time only) or env vars are involved.

            Show
            jhoblitt Joshua Hoblitt added a comment - Updating the gcc/glibc versions in the vm image to match the container seems to have had no change. Either there is some other package involved (perhaps build time only) or env vars are involved.
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            As a last resort, I'm testing changing the doxygen base package from datarel -> lsst_distrib, as that would at least resolve DM-2948.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited As a last resort, I'm testing changing the doxygen base package from datarel -> lsst_distrib , as that would at least resolve DM-2948 .
            Hide
            tjenness Tim Jenness added a comment -

            For the record this is still broken in March. Are we now using lsst_distrib as the anchor package? I'd like to remove datarel because it brings in some mariadbclient dependencies that we'd rather get rid of.

            Show
            tjenness Tim Jenness added a comment - For the record this is still broken in March. Are we now using lsst_distrib as the anchor package? I'd like to remove datarel because it brings in some mariadbclient dependencies that we'd rather get rid of.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Per discussion on DM-2948, I have merged the removal of datarel from create_xlinkdocs.sh.

            Show
            jhoblitt Joshua Hoblitt added a comment - Per discussion on DM-2948 , I have merged the removal of datarel from create_xlinkdocs.sh .
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm at a loss as for how to address this and am returning the issue to the backlog for the time being.

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm at a loss as for how to address this and am returning the issue to the backlog for the time being.
            Hide
            ktl Kian-Tat Lim added a comment -

            Fixing the Python 3.7 incompatibility (having to do with a regexp substitution for \mainpage commands) has fixed this!

            Show
            ktl Kian-Tat Lim added a comment - Fixing the Python 3.7 incompatibility (having to do with a regexp substitution for \mainpage commands) has fixed this!

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              ktl Kian-Tat Lim
              Watchers:
              John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.