Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12569

Missing pandoc in Travis docs build

    Details

      Description

      When I push to LDM-534, a Travis build is started and (apparently) runs successfully. However, the new version of the document doesn't appear at https://ldm-534.lsst.io/v/.

      On further investigation, Lander is failing as follows:

      5.62s$ lander --pdf LDM-534*.pdf --upload --lsstdoc LDM-534.tex --env=[secure] --ltd-product $PRODUCT --handle $HANDLE --title "$TITLE"
      2017-11-09 03:35:38,557     INFO lander.main | Lander version 0.1.8
      Maybe try:
          sudo apt-get install pandoc
      See http://johnmacfarlane.net/pandoc/installing.html
      for installation options
      ---------------------------------------------------------------
      2017-11-09 03:35:38,575  WARNING metasrc.pandoc.convert | pandoc needed but not found. Now installing it for you.
      tar (child): data.tar.gz: Cannot open: No such file or directory
      tar (child): Error is not recoverable: exiting now
      tar: Child returned status 2
      tar: Error is not recoverable: exiting now
      * Downloading pandoc from https://github.com/jgm/pandoc/releases/download/2.0.1.1/pandoc-2.0.1.1-1-amd64.deb ...
      * Unpacking pandoc-2.0.1.1-1-amd64.deb to tempfolder...
      Traceback (most recent call last):
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 24, in _install_and_run
          result = func(*args, **kwargs)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 124, in convert_text
          extra_args=extra_args)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 103, in convert_text
          outputfile=outputfile, filters=filters)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 260, in _convert_input
          _ensure_pandoc_path()
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 544, in _ensure_pandoc_path
          raise OSError("No pandoc was found: either install pandoc and add it\n"
      OSError: No pandoc was found: either install pandoc and add it
      to your PATH or or call pypandoc.download_pandoc(...) or
      install pypandoc wheels with included pandoc.
      During handling of the above exception, another exception occurred:
      Traceback (most recent call last):
        File "/home/[secure]/virtualenv/python3.5.3/bin/lander", line 11, in <module>
          sys.exit(main())
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/lander/main.py", line 185, in main
          config = Configuration(args=args)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/lander/config.py", line 91, in __init__
          self['title_html'] = lsstdoc.html_title
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/tex/lsstdoc.py", line 79, in html_title
          mathjax=False, smart=True)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/tex/lsstdoc.py", line 281, in format_title
          extra_args=extra_args)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 190, in convert_lsstdoc_tex
          smart=smart, extra_args=extra_args)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 29, in _install_and_run
          pypandoc.download_pandoc()
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/pandoc_download.py", line 207, in download_pandoc
          unpack(filename, targetfolder)
        File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/pandoc_download.py", line 86, in _handle_linux
          subprocess.check_call(cmd)
        File "/opt/python/3.5.3/lib/python3.5/subprocess.py", line 271, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['tar', 'xzf', 'data.tar.gz']' returned non-zero exit status 2
      

      This is due to a known issue with pypandoc, which attempts (and fails) to install pandoc for itself if it doesn't already exist.

      I also tried installing my own pandoc using APT. That doesn't help:

      RuntimeError: Pandoc died with exitcode "2" during conversion: b"pandoc: unrecognized option `--wrap=none'\nTry pandoc --help for more information.\n"
      

      I guess this tells us that pypandoc is assuming some version of pandoc other than that provided by the Ubuntu repository.

        Attachments

          Activity

          Hide
          jsick Jonathan Sick added a comment -

          I'm looking into this issue. I think pypandoc issue #146 might be a red herring since that was opened in April 2017 and Lander hasn't been broken for that long.

          I did notice that pandoc created a 2.0 release at the end of October (https://pandoc.org/releases.html#pandoc-2.0-29-oct-2017), which fits the timeline a bit better.

          I'm going to experiment with constraining the pandoc version that pypandoc (via metasrc/lander) attempts to install.

          Show
          jsick Jonathan Sick added a comment - I'm looking into this issue. I think pypandoc issue #146 might be a red herring since that was opened in April 2017 and Lander hasn't been broken for that long. I did notice that pandoc created a 2.0 release at the end of October ( https://pandoc.org/releases.html#pandoc-2.0-29-oct-2017 ), which fits the timeline a bit better. I'm going to experiment with constraining the pandoc version that pypandoc (via metasrc/lander) attempts to install.
          Hide
          swinbank John Swinbank added a comment - - edited

          Not sure where the "April 2017" comes from — according to GitHub, pypandoc #146 was opened "16 days ago".

          But anyway, I don't think this is due to a version change of pypandoc. What's happening is that Pypandoc interrogates the Pandoc github site to find the latest deb format package for Pandoc. It then downloads and unpacks that package, making certain assumptions about what it contains (specifically, that it contains a data.tar.gz file). Current Pandoc debs don't contain that file: they contain data.tar.xz instead. This likely due to a change in the Pandoc build chain (presumably they upgraded to a newer version of dpkg-deb which defaults to xz compression), so it could have happened at any time.

          Looks like version 1.4 of Pypandoc introduced this "grab the latest Pandoc deb" functionality: before that, they used hard-coded versions. So I guess dropping back to a version of Pypandoc earlier than that would do the trick, unless Pandoc has regenerated all their debs. (Fixing Pypandoc to not make unwarranted assumptions about what you can find in deb packages would seem to be the right approach, but that's likely a job for upstream rather than for us.)

          Show
          swinbank John Swinbank added a comment - - edited Not sure where the "April 2017" comes from — according to GitHub, pypandoc #146 was opened "16 days ago". But anyway, I don't think this is due to a version change of pypandoc. What's happening is that Pypandoc interrogates the Pandoc github site to find the latest deb format package for Pandoc. It then downloads and unpacks that package, making certain assumptions about what it contains (specifically, that it contains a data.tar.gz file). Current Pandoc debs don't contain that file: they contain data.tar.xz instead. This likely due to a change in the Pandoc build chain (presumably they upgraded to a newer version of dpkg-deb which defaults to xz compression), so it could have happened at any time. Looks like version 1.4 of Pypandoc introduced this "grab the latest Pandoc deb" functionality: before that, they used hard-coded versions. So I guess dropping back to a version of Pypandoc earlier than that would do the trick, unless Pandoc has regenerated all their debs. (Fixing Pypandoc to not make unwarranted assumptions about what you can find in deb packages would seem to be the right approach, but that's likely a job for upstream rather than for us.)
          Hide
          jsick Jonathan Sick added a comment -

          I meant to say pypandoc 1.4 was released in April 2017 and we've been using that functionality successfully until Pandoc 2.0+ came out. PRs with fixes are on the way.

          Show
          jsick Jonathan Sick added a comment - I meant to say pypandoc 1.4 was released in April 2017 and we've been using that functionality successfully until Pandoc 2.0+ came out. PRs with fixes are on the way.
          Hide
          jsick Jonathan Sick added a comment -

          Fixes are in place:

          Once I release lander 0.1.9 on PyPI the fix will automatically become available to all existing Lander-enabled documents.

          Show
          jsick Jonathan Sick added a comment - Fixes are in place: https://github.com/lsst-sqre/metasrc/pull/8 https://github.com/lsst-sqre/lander/pull/7 Once I release lander 0.1.9 on PyPI the fix will automatically become available to all existing Lander-enabled documents.
          Hide
          swinbank John Swinbank added a comment -

          Seems fine; thanks.

          Minor points:

          • Can we be specific about the "apparent version incompatibility" in flake8 land? I'm not clear if that's referring to another ticket, or if it's just something you happened to notice.
          • I think you're needlessly conservative re the diagnosis of the pandoc downloading problem. This seems totally well defined to me: the pypandoc folks have screwed up by making incorrect assumptions about how dpkg-deb works, and they have a ticket in to fix it. Not sure why we're dancing around that issue.

          But those are both fairly trivial: good to merge as is if you disagree that they are worth addressing.

          Show
          swinbank John Swinbank added a comment - Seems fine; thanks. Minor points: Can we be specific about the "apparent version incompatibility" in flake8 land? I'm not clear if that's referring to another ticket, or if it's just something you happened to notice. I think you're needlessly conservative re the diagnosis of the pandoc downloading problem. This seems totally well defined to me: the pypandoc folks have screwed up by making incorrect assumptions about how dpkg-deb works, and they have a ticket in to fix it. Not sure why we're dancing around that issue. But those are both fairly trivial: good to merge as is if you disagree that they are worth addressing.
          Hide
          jsick Jonathan Sick added a comment - - edited

          Thanks. I've released metasrc==0.2.2 and lander==0.1.9 to PyPI.

          I adjusted the change log comments about the pytest stack update. It's one of those awkward situations where we've pinned dependencies, but there are floating indirect dependencies that introduced an incompatibility, as far as I can tell.

          And it's absolutely true that I could have solved the issue by sending a PR upstream to pypandoc, but this route seemed the most expedient to fix DM's immediate production issue.

          Show
          jsick Jonathan Sick added a comment - - edited Thanks. I've released metasrc==0.2.2 and lander==0.1.9 to PyPI. I adjusted the change log comments about the pytest stack update. It's one of those awkward situations where we've pinned dependencies, but there are floating indirect dependencies that introduced an incompatibility, as far as I can tell. And it's absolutely true that I could have solved the issue by sending a PR upstream to pypandoc, but this route seemed the most expedient to fix DM's immediate production issue.

            People

            • Assignee:
              jsick Jonathan Sick
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Swinbank
              Watchers:
              John Swinbank, Jonathan Sick, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel