Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33157

Fix doxygen errors in pipe_tasks

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP S22-5 (April)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Examining the build logs for pipe_tasks demonstrated that we are still using doxygen to build the docs and there are some unexplained slow downs during that process. In particular between CharacterizeImageTask and MakeCoaddTempExpTask there is a 2 minute pause in logging on a Jenkins run:

      [2022-01-06T16:03:51.943911Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:245: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
      		 
      [2022-01-06T16:03:51.944330Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:246: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
      		 
      [2022-01-06T16:03:51.944774Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: Found recursive @copybrief or @copydoc relation for argument 'CharacterizeImageTask'.
      		 
      [2022-01-06T16:03:51.944782Z] 
      [2022-01-06T16:03:51.945163Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
      		 
      [2022-01-06T16:03:51.945543Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:250: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
      		 
      [2022-01-06T16:05:26.867714Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
      		 
      [2022-01-06T16:05:26.867727Z] 
      [2022-01-06T16:05:26.868102Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
      		 
      [2022-01-06T16:05:26.868478Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:116: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
      		 
      [2022-01-06T16:05:26.868903Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:119: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
      		 
      [2022-01-06T16:05:26.868910Z]

      It's possibly that fixing the errors will fix the slow down. Ideally we would convert to sphinx but that is beyond the scope of this ticket.

      See https://ci.lsst.codes/job/stack-os-matrix/35647/artifact/macos/lsstsw/build/pipe_tasks/_build.log

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          krzys Krzysztof Findeisen added a comment -

          I've started work on this, but I can't reproduce the slowdown locally. scons doc runs noisily but quickly. I do see a big slowdown right before

          passed
          		 
          running global pytest...

          which in the linked Jenkins log seems to have no delay.

          Show
          krzys Krzysztof Findeisen added a comment - I've started work on this, but I can't reproduce the slowdown locally. scons doc runs noisily but quickly. I do see a big slowdown right before passed running global pytest... which in the linked Jenkins log seems to have no delay.
          Hide
          Permalink
          tjenness Tim Jenness added a comment -

          Running locally with scons my log says:

          scons: Nothing to be done for `examples'.
          		 
          buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
          		 
          running tests/nopytest_test_coadds.py... doxygen /Users/timj/work/lsstsw/build/pipe_tasks/doc/doxygen.conf
          		 
          error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/utils/g617c0b0dc2+9633a190c8/doc/utils.tag' does not exist or is not a file. Skipping it...
          		 
          error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/obs_base/g7a69c27ea0+bead29cdf2/doc/obs_base.tag' does not exist or is not a file. Skipping it...
          		 
          error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/daf_butler/g8243fdef33+f83c363f3e/doc/daf_butler.tag' does not exist or is not a file. Skipping it...
          		 
          /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py:173: warning: multiple use of section label 'pipe_tasks_transform_Contents' while adding section, (first occurrence: /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py, line 61)
          		 
          /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/calibrate.py:339: warning: Found recursive @copybrief or @copydoc relation for argument 'CalibrateTask'.

          and then there is a pause right at the end where it says "passed". That gap is caused by the nopytest_test_coadds.py running. We block the pytest startup until the standalone tests are run. This is all consistent with the doxygen and nopytest test running in parallel. The confusing thing about the Jenkins output is that the pause is in the middle of the oxygen output, not at the end of it. Is it possible that eupspkg capturing the standard out from scons is causing the confusing timestamps and the gap is from the test but the doxygen log entries are buffered? I think Eli Rykoff saw the same problem.

          Show
          tjenness Tim Jenness added a comment - Running locally with scons my log says: scons: Nothing to be done for `examples'. buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"]) running tests/nopytest_test_coadds.py... doxygen /Users/timj/work/lsstsw/build/pipe_tasks/doc/doxygen.conf error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/utils/g617c0b0dc2+9633a190c8/doc/utils.tag' does not exist or is not a file. Skipping it... error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/obs_base/g7a69c27ea0+bead29cdf2/doc/obs_base.tag' does not exist or is not a file. Skipping it... error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/daf_butler/g8243fdef33+f83c363f3e/doc/daf_butler.tag' does not exist or is not a file. Skipping it... /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py:173: warning: multiple use of section label 'pipe_tasks_transform_Contents' while adding section, (first occurrence: /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py, line 61) /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/calibrate.py:339: warning: Found recursive @copybrief or @copydoc relation for argument 'CalibrateTask'. and then there is a pause right at the end where it says "passed". That gap is caused by the nopytest_test_coadds.py running. We block the pytest startup until the standalone tests are run. This is all consistent with the doxygen and nopytest test running in parallel. The confusing thing about the Jenkins output is that the pause is in the middle of the oxygen output, not at the end of it. Is it possible that eupspkg capturing the standard out from scons is causing the confusing timestamps and the gap is from the test but the doxygen log entries are buffered? I think Eli Rykoff saw the same problem.
          Hide
          Permalink
          erykoff Eli Rykoff added a comment -

          Yes, I agree that this pause is due to nopytest_test_coadds.py and not from doxygen. That said, the log timestamps are confusing and Tim's hypothesis (or something related to different log buffers) might explain that part.

          Show
          erykoff Eli Rykoff added a comment - Yes, I agree that this pause is due to nopytest_test_coadds.py and not from doxygen . That said, the log timestamps are confusing and Tim's hypothesis (or something related to different log buffers) might explain that part.
          Hide
          Permalink
          krzys Krzysztof Findeisen added a comment -

          Thanks for agreeing to review this, Kian-Tat Lim!

          Show
          krzys Krzysztof Findeisen added a comment - Thanks for agreeing to review this, Kian-Tat Lim !
          Hide
          Permalink
          ktl Kian-Tat Lim added a comment -

          It looked like there was one still-correct line that was maybe inadvertently removed.  Otherwise, looks fine.

          Show
          ktl Kian-Tat Lim added a comment - It looked like there was one still-correct line that was maybe inadvertently removed.  Otherwise, looks fine.

            People

            Assignee:
            krzys Krzysztof Findeisen
            Reporter:
            tjenness Tim Jenness
            Reviewers:
            Kian-Tat Lim
            Watchers:
            Eli Rykoff, Ian Sullivan, Kian-Tat Lim, Krzysztof Findeisen, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.