[DM-33157] Fix doxygen errors in pipe_tasks - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: pipe_tasks
Labels:
None

Story Points:
4
Epic Link:
s22b-ap-emergent
Sprint:
AP S22-5 (April)
Team:
Alert Production
Urgent?:
No

Description

[2022-01-06T16:03:51.943911Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:245: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command

[2022-01-06T16:03:51.944330Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:246: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command

[2022-01-06T16:03:51.944774Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: Found recursive @copybrief or @copydoc relation for argument 'CharacterizeImageTask'.

[2022-01-06T16:03:51.944782Z]

[2022-01-06T16:03:51.945163Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command

[2022-01-06T16:03:51.945543Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:250: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command

[2022-01-06T16:05:26.867714Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.

[2022-01-06T16:05:26.867727Z]

[2022-01-06T16:05:26.868102Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command

[2022-01-06T16:05:26.868478Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:116: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command

[2022-01-06T16:05:26.868903Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:119: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.

[2022-01-06T16:05:26.868910Z]

It's possibly that fixing the errors will fix the slow down. Ideally we would convert to sphinx but that is beyond the scope of this ticket.

See https://ci.lsst.codes/job/stack-os-matrix/35647/artifact/macos/lsstsw/build/pipe_tasks/_build.log

Attachments

Activity

Ascending order - Click to sort in descending order

No builds found.

Tim Jenness created issue - 06/Jan/22 10:23 PM

Tim Jenness made changes - 06/Jan/22 10:42 PM

Field	Original Value	New Value
Summary	Migrate pipe_tasks documentation to sphinx	Fix doxygen errors in pipe_tasks

Tim Jenness made changes - 06/Jan/22 10:43 PM

Description

Examining the build logs for pipe_tasks demonstrated that we are still using doxygen to build the docs and there are some unexplained slow downs during that process. In particular between CharacterizeImageTask and MakeCoaddTempExpTask there is a 2 minute pause in logging on a Jenkins run:

{code}
[2022-01-06T16:03:51.943911Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:245: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.944330Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:246: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.944774Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: Found recursive @copybrief or @copydoc relation for argument 'CharacterizeImageTask'.
[2022-01-06T16:03:51.944782Z]
[2022-01-06T16:03:51.945163Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.945543Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:250: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:05:26.867714Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
[2022-01-06T16:05:26.867727Z]
[2022-01-06T16:05:26.868102Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
[2022-01-06T16:05:26.868478Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:116: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
[2022-01-06T16:05:26.868903Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:119: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
[2022-01-06T16:05:26.868910Z]
{code}

Removing doxygen completely is probably the better answer rather than continuing to run doxygen.

See https://ci.lsst.codes/job/stack-os-matrix/35647/artifact/macos/lsstsw/build/pipe_tasks/_build.log

Examining the build logs for pipe_tasks demonstrated that we are still using doxygen to build the docs and there are some unexplained slow downs during that process. In particular between CharacterizeImageTask and MakeCoaddTempExpTask there is a 2 minute pause in logging on a Jenkins run:

{code}
[2022-01-06T16:03:51.943911Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:245: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.944330Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:246: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.944774Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: Found recursive @copybrief or @copydoc relation for argument 'CharacterizeImageTask'.
[2022-01-06T16:03:51.944782Z]
[2022-01-06T16:03:51.945163Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:249: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:03:51.945543Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/characterizeImage.py:250: warning: unable to resolve reference to 'CharacterizeImageTask_' for \ref command
[2022-01-06T16:05:26.867714Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
[2022-01-06T16:05:26.867727Z]
[2022-01-06T16:05:26.868102Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:115: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
[2022-01-06T16:05:26.868478Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:116: warning: unable to resolve reference to 'MakeCoaddTempExpTask_' for \ref command
[2022-01-06T16:05:26.868903Z] /Users/square/j/ws/stack-os-matrix/macos/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/makeCoaddTempExp.py:119: warning: Found recursive @copybrief or @copydoc relation for argument 'MakeCoaddTempExpTask'.
[2022-01-06T16:05:26.868910Z]
{code}

It's possibly that fixing the errors will fix the slow down. Ideally we would convert to sphinx but that is beyond the scope of this ticket.

See https://ci.lsst.codes/job/stack-os-matrix/35647/artifact/macos/lsstsw/build/pipe_tasks/_build.log

Krzysztof Findeisen made changes - 06/Jan/22 10:57 PM

Epic Link		~~DM-30502~~ [ 510160 ]
Sprint		AP S22-3 (February) [ 1142 ]

Ian Sullivan made changes - 03/Mar/22 8:14 PM

Sprint

AP S22-3 (February) [ 1142 ]

AP S22-5 (April) [ 1156 ]

Ian Sullivan made changes - 03/Mar/22 8:14 PM

Rank

Ranked higher

Ian Sullivan made changes - 09/Mar/22 11:54 PM

Epic Link

~~DM-30502~~ [ 510160 ]

~~DM-30506~~ [ 510172 ]

Krzysztof Findeisen made changes - 29/Mar/22 9:08 PM

Rank

Ranked higher

Krzysztof Findeisen made changes - 30/Mar/22 9:57 PM

Rank

Ranked lower

Ian Sullivan made changes - 30/Mar/22 11:31 PM

Rank

Ranked higher

Krzysztof Findeisen made changes - 04/Apr/22 6:46 PM

Status

To Do [ 10001 ]

In Progress [ 3 ]

Hide

Permalink

Krzysztof Findeisen added a comment - 04/Apr/22 7:39 PM

I've started work on this, but I can't reproduce the slowdown locally. scons doc runs noisily but quickly. I do see a big slowdown right before

passed

running global pytest...

which in the linked Jenkins log seems to have no delay.

Show

Krzysztof Findeisen added a comment - 04/Apr/22 7:39 PM I've started work on this, but I can't reproduce the slowdown locally. scons doc runs noisily but quickly. I do see a big slowdown right before passed running global pytest... which in the linked Jenkins log seems to have no delay.

Hide

Permalink

Tim Jenness added a comment - 04/Apr/22 8:02 PM

Running locally with scons my log says:

scons: Nothing to be done for `examples'.

buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])

running tests/nopytest_test_coadds.py... doxygen /Users/timj/work/lsstsw/build/pipe_tasks/doc/doxygen.conf

error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/utils/g617c0b0dc2+9633a190c8/doc/utils.tag' does not exist or is not a file. Skipping it...

error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/obs_base/g7a69c27ea0+bead29cdf2/doc/obs_base.tag' does not exist or is not a file. Skipping it...

error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/daf_butler/g8243fdef33+f83c363f3e/doc/daf_butler.tag' does not exist or is not a file. Skipping it...

/Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py:173: warning: multiple use of section label 'pipe_tasks_transform_Contents' while adding section, (first occurrence: /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py, line 61)

/Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/calibrate.py:339: warning: Found recursive @copybrief or @copydoc relation for argument 'CalibrateTask'.

and then there is a pause right at the end where it says "passed". That gap is caused by the nopytest_test_coadds.py running. We block the pytest startup until the standalone tests are run. This is all consistent with the doxygen and nopytest test running in parallel. The confusing thing about the Jenkins output is that the pause is in the middle of the oxygen output, not at the end of it. Is it possible that eupspkg capturing the standard out from scons is causing the confusing timestamps and the gap is from the test but the doxygen log entries are buffered? I think Eli Rykoff saw the same problem.

Show

Tim Jenness added a comment - 04/Apr/22 8:02 PM Running locally with scons my log says: scons: Nothing to be done for `examples'. buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"]) running tests/nopytest_test_coadds.py... doxygen /Users/timj/work/lsstsw/build/pipe_tasks/doc/doxygen.conf error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/utils/g617c0b0dc2+9633a190c8/doc/utils.tag' does not exist or is not a file. Skipping it... error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/obs_base/g7a69c27ea0+bead29cdf2/doc/obs_base.tag' does not exist or is not a file. Skipping it... error: Tag file '/Users/timj/work/lsstsw/stack/lsst-scipipe-3.0.0/Darwin/daf_butler/g8243fdef33+f83c363f3e/doc/daf_butler.tag' does not exist or is not a file. Skipping it... /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py:173: warning: multiple use of section label 'pipe_tasks_transform_Contents' while adding section, (first occurrence: /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/transformMeasurement.py, line 61) /Users/timj/work/lsstsw/build/pipe_tasks/python/lsst/pipe/tasks/calibrate.py:339: warning: Found recursive @copybrief or @copydoc relation for argument 'CalibrateTask'. and then there is a pause right at the end where it says "passed". That gap is caused by the nopytest_test_coadds.py running. We block the pytest startup until the standalone tests are run. This is all consistent with the doxygen and nopytest test running in parallel. The confusing thing about the Jenkins output is that the pause is in the middle of the oxygen output, not at the end of it. Is it possible that eupspkg capturing the standard out from scons is causing the confusing timestamps and the gap is from the test but the doxygen log entries are buffered? I think Eli Rykoff saw the same problem.

Hide

Permalink

Eli Rykoff added a comment - 04/Apr/22 8:05 PM

Yes, I agree that this pause is due to nopytest_test_coadds.py and not from doxygen. That said, the log timestamps are confusing and Tim's hypothesis (or something related to different log buffers) might explain that part.

Show

Eli Rykoff added a comment - 04/Apr/22 8:05 PM Yes, I agree that this pause is due to nopytest_test_coadds.py and not from doxygen . That said, the log timestamps are confusing and Tim's hypothesis (or something related to different log buffers) might explain that part.

Hide

Permalink

Krzysztof Findeisen added a comment - 11/Apr/22 6:14 PM

Thanks for agreeing to review this, Kian-Tat Lim!

Show

Krzysztof Findeisen added a comment - 11/Apr/22 6:14 PM Thanks for agreeing to review this, Kian-Tat Lim !

Krzysztof Findeisen made changes - 11/Apr/22 6:14 PM

Reviewers		Kian-Tat Lim [ ktl ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

Hide

Permalink

Kian-Tat Lim added a comment - 11/Apr/22 10:13 PM

It looked like there was one still-correct line that was maybe inadvertently removed. Otherwise, looks fine.

Show

Kian-Tat Lim added a comment - 11/Apr/22 10:13 PM It looked like there was one still-correct line that was maybe inadvertently removed. Otherwise, looks fine.

Kian-Tat Lim made changes - 11/Apr/22 10:13 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Krzysztof Findeisen made changes - 12/Apr/22 6:54 PM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

People

Assignee:: Krzysztof Findeisen

Reporter:: Tim Jenness

Reviewers:: Kian-Tat Lim

Watchers:: Eli Rykoff, Ian Sullivan, Kian-Tat Lim, Krzysztof Findeisen, Tim Jenness

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 06/Jan/22 10:23 PM

Updated:: 12/Apr/22 6:54 PM

Resolved:: 12/Apr/22 6:54 PM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Activity

People

Dates

Jenkins