Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20866

Fix package-docs recursion depth error from rebuilds

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: documenteer
    • Labels:
      None

      Description

      When running package-docs multiple times to rebuild documentation for a single package multiple times, it's common to see a recursion depth error from Sphinx:

      package-docs build                                                                                                                                                                                                  jsick@Tay ~VERIFY_DIR
      Running Sphinx v1.7.9
      /Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/matplotlib/__init__.py:846: MatplotlibDeprecationWarning:
      The text.latex.unicode rcparam was deprecated in Matplotlib 2.2 and will be removed in 3.1.
        "2.2", name=key, obj_type="rcparam", addendum=addendum)
      loading pickled environment... done
      [autosummary] generating autosummary for: index.rst, lsst.verify/index.rst, lsst.verify/scripts/dispatch_verify.py.rst, lsst.verify/scripts/inspect_job.py.rst, lsst.verify/scripts/lint_metrics.py.rst, lsst.verify/tasks/lsst.verify.gen2tasks.MetricTask.rst, lsst.verify/tasks/lsst.verify.gen2tasks.MetricsControllerTask.rst, lsst.verify/tasks/lsst.verify.gen2tasks.SquashMetadataTask.rst, lsst.verify/tasks/lsst.verify.tasks.ConfigPpdbLoader.rst, lsst.verify/tasks/lsst.verify.tasks.MetadataMetricTask.rst, ..., py-api/lsst.verify.squash.post.rst, py-api/lsst.verify.squash.reset_endpoint_cache.rst, py-api/lsst.verify.tasks.ConfigPpdbLoader.rst, py-api/lsst.verify.tasks.MetadataMetricConfig.rst, py-api/lsst.verify.tasks.MetadataMetricTask.rst, py-api/lsst.verify.tasks.MetricComputationError.rst, py-api/lsst.verify.tasks.PpdbMetricConfig.rst, py-api/lsst.verify.tasks.PpdbMetricTask.rst, py-api/lsst.verify.tasks.TimingMetricConfig.rst, py-api/lsst.verify.tasks.TimingMetricTask.rst
      [automodsumm] lsst.verify/index.rst: found 51 automodsumm entries to generate
      building [mo]: all of 0 po files
      building [html]: all source files
       
      Traceback (most recent call last):
        File "/Users/jsick/lsst/pipelines_lsst_io_stack/documenteer/documenteer/sphinxrunner.py", line 85, in run_sphinx
          app.build(force_all, filenames)
        File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/application.py", line 325, in build
          self.builder.build_all()
        File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_all
          self.build(None, summary='all source files', method='all')
        File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 355, in build
          updated_docnames = set(self.env.update(self.config, self.srcdir, self.doctreedir))
        File "/Users/jsick/lsst/pipelines_lsst_io_stack/stacks/w_2019_29/python/miniconda3-4.5.12/envs/lsst-scipipe-1172c30/lib/python3.7/site-packages/sphinx/environment/__init__.py", line 511, in update
          if self.config[confval.name] != confval.value:
      RecursionError: maximum recursion depth exceeded in comparison
       
      Recursion error:
      maximum recursion depth exceeded in comparison
       
      This can happen with very large or deeply nested source files.  You can carefully increase the default Python recursion limit of 1000 in conf.py with e.g.:
          import sys; sys.setrecursionlimit(1500)
      

      For the the Pipelines documentation we do use that sys.setrecursionlimit trick, but I don't think this has been done yet for the builds of individual packages.

        Attachments

          Issue Links

            Activity

            Hide
            jsick Jonathan Sick added a comment -

            My hunch right now is that the current approach of injecting configuring variables is working against us. Right now the config code produces a dictionary with all the configuration variables, and these variables get added to the global member. It seems that this approach isn't reliably pickle-able. The better approach seems to be to have a full module that is simply imported by the conf.py file:

            from documenteer.conf.stack import * 

            This is the approach that projects like astropy are making and it seems to be far more reliable for them.

            Show
            jsick Jonathan Sick added a comment - My hunch right now is that the current approach of injecting configuring variables is working against us. Right now the config code produces a dictionary with all the configuration variables, and these variables get added to the global member. It seems that this approach isn't reliably pickle-able. The better approach seems to be to have a full module that is simply imported by the conf.py file: from documenteer.conf.stack import * This is the approach that projects like astropy are making and it seems to be far more reliable for them.
            Hide
            jsick Jonathan Sick added a comment -

            It seems that this issue blocks real progress on DM-20866, which is turn blocks the Doxygen integration work.

            I'm targeting this for a documenteer 0.6 release, which will specifically be incompatible with documenteer 0.5.

            Show
            jsick Jonathan Sick added a comment - It seems that this issue blocks real progress on DM-20866 , which is turn blocks the Doxygen integration work. I'm targeting this for a documenteer 0.6 release, which will specifically be incompatible with documenteer 0.5.
            Hide
            jsick Jonathan Sick added a comment -

            This work will be part of the upcoming 0.6.0 release of Documenteer. Additional docs will need to be written to coordinate the migration of any existing projects' config.py that want to use documenteer >= 0.6.0.

            Show
            jsick Jonathan Sick added a comment - This work will be part of the upcoming 0.6.0 release of Documenteer. Additional docs will need to be written to coordinate the migration of any existing projects' config.py that want to use documenteer >= 0.6.0.

              People

              Assignee:
              jsick Jonathan Sick
              Reporter:
              jsick Jonathan Sick
              Watchers:
              John Swinbank, Jonathan Sick
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: