# Missing pandoc in Travis docs build

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
0.7
• Team:
SQuaRE

## Description

When I push to LDM-534, a Travis build is started and (apparently) runs successfully. However, the new version of the document doesn't appear at https://ldm-534.lsst.io/v/.

On further investigation, Lander is failing as follows:

 5.62s$lander --pdf LDM-534*.pdf --upload --lsstdoc LDM-534.tex --env=[secure] --ltd-product$PRODUCT --handle $HANDLE --title "$TITLE" 2017-11-09 03:35:38,557 INFO lander.main | Lander version 0.1.8 Maybe try:  sudo apt-get install pandoc See http://johnmacfarlane.net/pandoc/installing.html for installation options --------------------------------------------------------------- 2017-11-09 03:35:38,575 WARNING metasrc.pandoc.convert | pandoc needed but not found. Now installing it for you. tar (child): data.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now * Downloading pandoc from https://github.com/jgm/pandoc/releases/download/2.0.1.1/pandoc-2.0.1.1-1-amd64.deb ... * Unpacking pandoc-2.0.1.1-1-amd64.deb to tempfolder... Traceback (most recent call last):  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 24, in _install_and_run  result = func(*args, **kwargs)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 124, in convert_text  extra_args=extra_args)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 103, in convert_text  outputfile=outputfile, filters=filters)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 260, in _convert_input  _ensure_pandoc_path()  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/__init__.py", line 544, in _ensure_pandoc_path  raise OSError("No pandoc was found: either install pandoc and add it\n" OSError: No pandoc was found: either install pandoc and add it to your PATH or or call pypandoc.download_pandoc(...) or install pypandoc wheels with included pandoc. During handling of the above exception, another exception occurred: Traceback (most recent call last):  File "/home/[secure]/virtualenv/python3.5.3/bin/lander", line 11, in   sys.exit(main())  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/lander/main.py", line 185, in main  config = Configuration(args=args)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/lander/config.py", line 91, in __init__  self['title_html'] = lsstdoc.html_title  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/tex/lsstdoc.py", line 79, in html_title  mathjax=False, smart=True)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/tex/lsstdoc.py", line 281, in format_title  extra_args=extra_args)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 190, in convert_lsstdoc_tex  smart=smart, extra_args=extra_args)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/metasrc/pandoc/convert.py", line 29, in _install_and_run  pypandoc.download_pandoc()  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/pandoc_download.py", line 207, in download_pandoc  unpack(filename, targetfolder)  File "/home/[secure]/virtualenv/python3.5.3/lib/python3.5/site-packages/pypandoc/pandoc_download.py", line 86, in _handle_linux  subprocess.check_call(cmd)  File "/opt/python/3.5.3/lib/python3.5/subprocess.py", line 271, in check_call  raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['tar', 'xzf', 'data.tar.gz']' returned non-zero exit status 2 

This is due to a known issue with pypandoc, which attempts (and fails) to install pandoc for itself if it doesn't already exist.

I also tried installing my own pandoc using APT. That doesn't help:

 RuntimeError: Pandoc died with exitcode "2" during conversion: b"pandoc: unrecognized option --wrap=none'\nTry pandoc --help for more information.\n" `

I guess this tells us that pypandoc is assuming some version of pandoc other than that provided by the Ubuntu repository.

## Activity

Hide
Jonathan Sick added a comment -

I'm looking into this issue. I think pypandoc issue #146 might be a red herring since that was opened in April 2017 and Lander hasn't been broken for that long.

I did notice that pandoc created a 2.0 release at the end of October (https://pandoc.org/releases.html#pandoc-2.0-29-oct-2017), which fits the timeline a bit better.

I'm going to experiment with constraining the pandoc version that pypandoc (via metasrc/lander) attempts to install.

Show
Jonathan Sick added a comment - I'm looking into this issue. I think pypandoc issue #146 might be a red herring since that was opened in April 2017 and Lander hasn't been broken for that long. I did notice that pandoc created a 2.0 release at the end of October ( https://pandoc.org/releases.html#pandoc-2.0-29-oct-2017 ), which fits the timeline a bit better. I'm going to experiment with constraining the pandoc version that pypandoc (via metasrc/lander) attempts to install.
Hide
John Swinbank added a comment - - edited

Not sure where the "April 2017" comes from — according to GitHub, pypandoc #146 was opened "16 days ago".

But anyway, I don't think this is due to a version change of pypandoc. What's happening is that Pypandoc interrogates the Pandoc github site to find the latest deb format package for Pandoc. It then downloads and unpacks that package, making certain assumptions about what it contains (specifically, that it contains a data.tar.gz file). Current Pandoc debs don't contain that file: they contain data.tar.xz instead. This likely due to a change in the Pandoc build chain (presumably they upgraded to a newer version of dpkg-deb which defaults to xz compression), so it could have happened at any time.

Looks like version 1.4 of Pypandoc introduced this "grab the latest Pandoc deb" functionality: before that, they used hard-coded versions. So I guess dropping back to a version of Pypandoc earlier than that would do the trick, unless Pandoc has regenerated all their debs. (Fixing Pypandoc to not make unwarranted assumptions about what you can find in deb packages would seem to be the right approach, but that's likely a job for upstream rather than for us.)

Show
John Swinbank added a comment - - edited Not sure where the "April 2017" comes from — according to GitHub, pypandoc #146 was opened "16 days ago". But anyway, I don't think this is due to a version change of pypandoc. What's happening is that Pypandoc interrogates the Pandoc github site to find the latest deb format package for Pandoc. It then downloads and unpacks that package, making certain assumptions about what it contains (specifically, that it contains a data.tar.gz file). Current Pandoc debs don't contain that file: they contain data.tar.xz instead. This likely due to a change in the Pandoc build chain (presumably they upgraded to a newer version of dpkg-deb which defaults to xz compression), so it could have happened at any time. Looks like version 1.4 of Pypandoc introduced this "grab the latest Pandoc deb" functionality: before that, they used hard-coded versions. So I guess dropping back to a version of Pypandoc earlier than that would do the trick, unless Pandoc has regenerated all their debs. (Fixing Pypandoc to not make unwarranted assumptions about what you can find in deb packages would seem to be the right approach, but that's likely a job for upstream rather than for us.)
Hide
Jonathan Sick added a comment -

I meant to say pypandoc 1.4 was released in April 2017 and we've been using that functionality successfully until Pandoc 2.0+ came out. PRs with fixes are on the way.

Show
Jonathan Sick added a comment - I meant to say pypandoc 1.4 was released in April 2017 and we've been using that functionality successfully until Pandoc 2.0+ came out. PRs with fixes are on the way.
Hide
Jonathan Sick added a comment -

Fixes are in place:

Once I release lander 0.1.9 on PyPI the fix will automatically become available to all existing Lander-enabled documents.

Show
Jonathan Sick added a comment - Fixes are in place: https://github.com/lsst-sqre/metasrc/pull/8 https://github.com/lsst-sqre/lander/pull/7 Once I release lander 0.1.9 on PyPI the fix will automatically become available to all existing Lander-enabled documents.
Hide
John Swinbank added a comment -

Seems fine; thanks.

Minor points:

• Can we be specific about the "apparent version incompatibility" in flake8 land? I'm not clear if that's referring to another ticket, or if it's just something you happened to notice.
• I think you're needlessly conservative re the diagnosis of the pandoc downloading problem. This seems totally well defined to me: the pypandoc folks have screwed up by making incorrect assumptions about how dpkg-deb works, and they have a ticket in to fix it. Not sure why we're dancing around that issue.

But those are both fairly trivial: good to merge as is if you disagree that they are worth addressing.

Show
John Swinbank added a comment - Seems fine; thanks. Minor points: Can we be specific about the "apparent version incompatibility" in flake8 land? I'm not clear if that's referring to another ticket, or if it's just something you happened to notice. I think you're needlessly conservative re the diagnosis of the pandoc downloading problem. This seems totally well defined to me: the pypandoc folks have screwed up by making incorrect assumptions about how dpkg-deb works, and they have a ticket in to fix it. Not sure why we're dancing around that issue. But those are both fairly trivial: good to merge as is if you disagree that they are worth addressing.
Hide
Jonathan Sick added a comment - - edited

Thanks. I've released metasrc==0.2.2 and lander==0.1.9 to PyPI.

I adjusted the change log comments about the pytest stack update. It's one of those awkward situations where we've pinned dependencies, but there are floating indirect dependencies that introduced an incompatibility, as far as I can tell.

And it's absolutely true that I could have solved the issue by sending a PR upstream to pypandoc, but this route seemed the most expedient to fix DM's immediate production issue.

Show
Jonathan Sick added a comment - - edited Thanks. I've released metasrc==0.2.2 and lander==0.1.9 to PyPI. I adjusted the change log comments about the pytest stack update. It's one of those awkward situations where we've pinned dependencies, but there are floating indirect dependencies that introduced an incompatibility, as far as I can tell. And it's absolutely true that I could have solved the issue by sending a PR upstream to pypandoc, but this route seemed the most expedient to fix DM's immediate production issue.

## People

• Assignee:
Jonathan Sick
Reporter:
John Swinbank
Reviewers:
John Swinbank
Watchers:
John Swinbank, Jonathan Sick, Tim Jenness