# Fix package-docs recursion depth error from rebuilds

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.4
• Team:
SQuaRE

#### Description

When running package-docs multiple times to rebuild documentation for a single package multiple times, it's common to see a recursion depth error from Sphinx:

 package-docs build jsick@Tay ~VERIFY_DIR Running Sphinx v1.7.9 /Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/matplotlib/__init__.py:846: MatplotlibDeprecationWarning: The text.latex.unicode rcparam was deprecated in Matplotlib 2.2 and will be removed in 3.1.  "2.2", name=key, obj_type="rcparam", addendum=addendum) loading pickled environment... done [autosummary] generating autosummary for: index.rst, lsst.verify/index.rst, lsst.verify/scripts/dispatch_verify.py.rst, lsst.verify/scripts/inspect_job.py.rst, lsst.verify/scripts/lint_metrics.py.rst, lsst.verify/tasks/lsst.verify.gen2tasks.MetricTask.rst, lsst.verify/tasks/lsst.verify.gen2tasks.MetricsControllerTask.rst, lsst.verify/tasks/lsst.verify.gen2tasks.SquashMetadataTask.rst, lsst.verify/tasks/lsst.verify.tasks.ConfigPpdbLoader.rst, lsst.verify/tasks/lsst.verify.tasks.MetadataMetricTask.rst, ..., py-api/lsst.verify.squash.post.rst, py-api/lsst.verify.squash.reset_endpoint_cache.rst, py-api/lsst.verify.tasks.ConfigPpdbLoader.rst, py-api/lsst.verify.tasks.MetadataMetricConfig.rst, py-api/lsst.verify.tasks.MetadataMetricTask.rst, py-api/lsst.verify.tasks.MetricComputationError.rst, py-api/lsst.verify.tasks.PpdbMetricConfig.rst, py-api/lsst.verify.tasks.PpdbMetricTask.rst, py-api/lsst.verify.tasks.TimingMetricConfig.rst, py-api/lsst.verify.tasks.TimingMetricTask.rst [automodsumm] lsst.verify/index.rst: found 51 automodsumm entries to generate building [mo]: all of 0 po files building [html]: all source files   Traceback (most recent call last):  File "/Users/jsick/lsst/pipelines_lsst_io_stack/documenteer/documenteer/sphinxrunner.py", line 85, in run_sphinx  app.build(force_all, filenames)  File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/application.py", line 325, in build  self.builder.build_all()  File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_all  self.build(None, summary='all source files', method='all')  File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 355, in build  updated_docnames = set(self.env.update(self.config, self.srcdir, self.doctreedir))  File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/environment/__init__.py", line 511, in update  if self.config[confval.name] != confval.value: RecursionError: maximum recursion depth exceeded in comparison   Recursion error: maximum recursion depth exceeded in comparison   This can happen with very large or deeply nested source files. You can carefully increase the default Python recursion limit of 1000 in conf.py with e.g.:  import sys; sys.setrecursionlimit(1500) 

For the the Pipelines documentation we do use that sys.setrecursionlimit trick, but I don't think this has been done yet for the builds of individual packages.

#### Activity

Hide
Jonathan Sick added a comment -

My hunch right now is that the current approach of injecting configuring variables is working against us. Right now the config code produces a dictionary with all the configuration variables, and these variables get added to the global member. It seems that this approach isn't reliably pickle-able. The better approach seems to be to have a full module that is simply imported by the conf.py file:

 from documenteer.conf.stack import *

This is the approach that projects like astropy are making and it seems to be far more reliable for them.

Show
Jonathan Sick added a comment - My hunch right now is that the current approach of injecting configuring variables is working against us. Right now the config code produces a dictionary with all the configuration variables, and these variables get added to the global member. It seems that this approach isn't reliably pickle-able. The better approach seems to be to have a full module that is simply imported by the conf.py file: from documenteer.conf.stack import * This is the approach that projects like astropy are making and it seems to be far more reliable for them.
Hide
Jonathan Sick added a comment -

It seems that this issue blocks real progress on DM-20866, which is turn blocks the Doxygen integration work.

I'm targeting this for a documenteer 0.6 release, which will specifically be incompatible with documenteer 0.5.

Show
Jonathan Sick added a comment - It seems that this issue blocks real progress on DM-20866 , which is turn blocks the Doxygen integration work. I'm targeting this for a documenteer 0.6 release, which will specifically be incompatible with documenteer 0.5.
Hide
Jonathan Sick added a comment -

This work will be part of the upcoming 0.6.0 release of Documenteer. Additional docs will need to be written to coordinate the migration of any existing projects' config.py that want to use documenteer >= 0.6.0.

Show
Jonathan Sick added a comment - This work will be part of the upcoming 0.6.0 release of Documenteer. Additional docs will need to be written to coordinate the migration of any existing projects' config.py that want to use documenteer >= 0.6.0.

#### People

Assignee:
Jonathan Sick
Reporter:
Jonathan Sick
Watchers:
John Swinbank, Jonathan Sick