Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6033

Support sphinxcontrib-bibtex in technotes

    Details

      Description

      Allow bibtex-based references in technotes using sphinxcontrib-bibtex. Includes:

      • Changes to documenteer Sphinx configuration for technotes.
      • Addition of bibliography in technote boilerplate.
      • Documentation of our requirements to use bibtex for different types of references.

      DMTN-010 will be used as a pilot case.

        Attachments

          Issue Links

            Activity

            jsick Jonathan Sick created issue -
            jsick Jonathan Sick made changes -
            Field Original Value New Value
            Epic Link DM-5644 [ 23494 ]
            jsick Jonathan Sick made changes -
            Labels technote-platform documenteer technote-platform
            Hide
            jsick Jonathan Sick added a comment -

            Upgraded documenteer to support sphinxcontrib-bibtex (v0.1.1) and issued a PR to DMTN-010 demonstrating how bibtex support works.

            Show
            jsick Jonathan Sick added a comment - Upgraded documenteer to support sphinxcontrib-bibtex (v0.1.1) and issued a PR to DMTN-010 demonstrating how bibtex support works.
            jsick Jonathan Sick made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jsick Jonathan Sick made changes -
            Story Points 0.5
            jsick Jonathan Sick made changes -
            Description Allow bibtex-based in technotes using [sphinxcontrib-bibtex|https://github.com/mcmtroffaes/sphinxcontrib-bibtex].

            DMTN-010 will be used as a pilot case.
            Allow bibtex-based references in technotes using [sphinxcontrib-bibtex|https://github.com/mcmtroffaes/sphinxcontrib-bibtex].

            DMTN-010 will be used as a pilot case.
            Hide
            Parejkoj John Parejko added a comment -

            Thank you: this works nicely to add references to a technote.

            Show
            Parejkoj John Parejko added a comment - Thank you: this works nicely to add references to a technote.
            frossie Frossie Economou made changes -
            Epic Link DM-5644 [ 23494 ] DM-6198 [ 24714 ]
            jsick Jonathan Sick made changes -
            Epic Link DM-6198 [ 24714 ] DM-7496 [ 26625 ]
            jsick Jonathan Sick made changes -
            Epic Link DM-7496 [ 26625 ] DM-5644 [ 23494 ]
            jsick Jonathan Sick made changes -
            Watchers John Parejko, Jonathan Sick [ John Parejko, Jonathan Sick ] John Parejko, Jonathan Sick, Tim Jenness [ John Parejko, Jonathan Sick, Tim Jenness ]
            jsick Jonathan Sick made changes -
            Description Allow bibtex-based references in technotes using [sphinxcontrib-bibtex|https://github.com/mcmtroffaes/sphinxcontrib-bibtex].

            DMTN-010 will be used as a pilot case.
            Allow bibtex-based references in technotes using [sphinxcontrib-bibtex|https://github.com/mcmtroffaes/sphinxcontrib-bibtex]. Includes:

            - Changes to documenteer Sphinx configuration for technotes.
            - Addition of bibliography in technote boilerplate.
            - Documentation of our requirements to use bibtex for different types of references.

            DMTN-010 will be used as a pilot case.
            Hide
            jsick Jonathan Sick added a comment -

            See also an issue about references in DMTN-021: https://github.com/lsst-dm/dmtn-021/issues/2

            Show
            jsick Jonathan Sick added a comment - See also an issue about references in DMTN-021: https://github.com/lsst-dm/dmtn-021/issues/2
            Hide
            tjenness Tim Jenness added a comment -

            I used this scheme in DM-8142 and it worked well.

            Show
            tjenness Tim Jenness added a comment - I used this scheme in DM-8142 and it worked well.
            Hide
            jsick Jonathan Sick added a comment -

            It's probably long due to get this done properly. The big question in my mind is how to integrate properly with bibtex files that we're maintaining at https://github.com/lsst/lsst-texmf/tree/master/texmf/bibtex/bib. Here's my proposal:

            • lsst-technote-bootstrap includes a copy of the lsst-texmf bib files and installs them with a new technote.
            • lsst-technote-bootstrap also includes a local.bib file with directions on adding new bibtex entries.
            • The technote template also includes boilerplate for the bibliography directive that points to these bib files.
            • To help technote authors update the lsst-texmf-maintain bib files in their own technote's repo, we have a command line program installed through documenteer, refresh-lsst-bib that redownloads the lsst-texmf bib files from master on GitHub. The author can commit these changed files.
            Show
            jsick Jonathan Sick added a comment - It's probably long due to get this done properly. The big question in my mind is how to integrate properly with bibtex files that we're maintaining at https://github.com/lsst/lsst-texmf/tree/master/texmf/bibtex/bib . Here's my proposal: lsst-technote-bootstrap includes a copy of the lsst-texmf bib files and installs them with a new technote. lsst-technote-bootstrap also includes a local.bib file with directions on adding new bibtex entries. The technote template also includes boilerplate for the bibliography directive that points to these bib files. To help technote authors update the lsst-texmf-maintain bib files in their own technote's repo, we have a command line program installed through documenteer, refresh-lsst-bib that redownloads the lsst-texmf bib files from master on GitHub. The author can commit these changed files.
            jsick Jonathan Sick made changes -
            Story Points 0.5 1.7
            jsick Jonathan Sick made changes -
            Link This issue is duplicated by DM-11195 [ DM-11195 ]
            Hide
            jsick Jonathan Sick added a comment -

            There's potentially an issue with lsst-dm.bib that I'm tracking down:

            During handling of the above exception, another exception occurred:
             
            Traceback (most recent call last):
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/cmdline.py", line 296, in main
                app.build(opts.force_all, filenames)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/application.py", line 333, in build
                self.builder.build_update()
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 251, in build_update
                'out of date' % len(to_build))
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 265, in build
                self.doctreedir, self.app))
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 556, in update
                self._read_serial(docnames, app)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 576, in _read_serial
                self.read_doc(docname, app)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 684, in read_doc
                pub.publish()
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/core.py", line 217, in publish
                self.settings)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/io.py", line 55, in read
                self.parse()
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/readers/__init__.py", line 78, in parse
                self.parser.parse(self.input, document)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/__init__.py", line 185, in parse
                self.statemachine.run(inputlines, document, inliner=self.inliner)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 170, in run
                input_source=document['source'])
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
                context, state, transitions)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
                return method(match, context, next_state)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2320, in explicit_markup
                self.explicit_list(blank_finish)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2350, in explicit_list
                match_titles=self.state_machine.match_titles)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
                node=node, match_titles=match_titles)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 195, in run
                results = StateMachineWS.run(self, input_lines, input_offset)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run
                context, state, transitions)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line
                return method(match, context, next_state)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2623, in explicit_markup
                nodelist, blank_finish = self.explicit_construct(match)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2330, in explicit_construct
                return method(self, expmatch)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2073, in directive
                directive_class, match, type_name, option_presets)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2122, in run_directive
                result = directive_instance.run()
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 140, in run
                self.process_bibfile(bibfile, bibcache.encoding)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 214, in process_bibfile
                self.update_bibfile_cache(bibfile, mtime, encoding)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 175, in update_bibfile_cache
                data = self.parse_bibfile(bibfile, encoding)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 158, in parse_bibfile
                parser.parse_file(bibfile)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/__init__.py", line 51, in parse_file
                self.parse_stream(f)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 385, in parse_stream
                return self.parse_string(text)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 372, in parse_string
                for entry in entry_iterator:
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 185, in parse_bibliography
                self.handle_error(error)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 358, in handle_error
                report_error(error)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/errors.py", line 77, in report_error
                raise exception
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 183, in parse_bibliography
                yield tuple(self.parse_command())
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 216, in parse_command
                self.handle_error(error)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 358, in handle_error
                report_error(error)
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/errors.py", line 77, in report_error
                raise exception
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 214, in parse_command
                self.required([body_end])
              File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/scanner.py", line 118, in required
                raise TokenRequired(description, self)
            pybtex.scanner.TokenRequired: syntax error in line 165: '}' expected
            

            pybtex can read lsst-dm.bib just fine using

            from pybtex.database import parse_file
            parse_file('lsstdm.bib')
            

            But it runs into the above exception when reading it through the bibliography directive with Sphinx. Potentially its an incompatibility with the assumed latex+latin encoding.

            Debugging in process...

            Show
            jsick Jonathan Sick added a comment - There's potentially an issue with lsst-dm.bib that I'm tracking down: During handling of the above exception, another exception occurred:   Traceback (most recent call last): File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/cmdline.py", line 296, in main app.build(opts.force_all, filenames) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/application.py", line 333, in build self.builder.build_update() File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 251, in build_update 'out of date' % len(to_build)) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/builders/__init__.py", line 265, in build self.doctreedir, self.app)) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 556, in update self._read_serial(docnames, app) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 576, in _read_serial self.read_doc(docname, app) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/environment/__init__.py", line 684, in read_doc pub.publish() File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/core.py", line 217, in publish self.settings) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinx/io.py", line 55, in read self.parse() File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/readers/__init__.py", line 78, in parse self.parser.parse(self.input, document) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/__init__.py", line 185, in parse self.statemachine.run(inputlines, document, inliner=self.inliner) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 170, in run input_source=document['source']) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run context, state, transitions) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line return method(match, context, next_state) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2320, in explicit_markup self.explicit_list(blank_finish) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2350, in explicit_list match_titles=self.state_machine.match_titles) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse node=node, match_titles=match_titles) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 195, in run results = StateMachineWS.run(self, input_lines, input_offset) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 239, in run context, state, transitions) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/statemachine.py", line 460, in check_line return method(match, context, next_state) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2623, in explicit_markup nodelist, blank_finish = self.explicit_construct(match) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2330, in explicit_construct return method(self, expmatch) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2073, in directive directive_class, match, type_name, option_presets) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/docutils/parsers/rst/states.py", line 2122, in run_directive result = directive_instance.run() File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 140, in run self.process_bibfile(bibfile, bibcache.encoding) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 214, in process_bibfile self.update_bibfile_cache(bibfile, mtime, encoding) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 175, in update_bibfile_cache data = self.parse_bibfile(bibfile, encoding) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/sphinxcontrib/bibtex/directives.py", line 158, in parse_bibfile parser.parse_file(bibfile) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/__init__.py", line 51, in parse_file self.parse_stream(f) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 385, in parse_stream return self.parse_string(text) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 372, in parse_string for entry in entry_iterator: File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 185, in parse_bibliography self.handle_error(error) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 358, in handle_error report_error(error) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/errors.py", line 77, in report_error raise exception File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 183, in parse_bibliography yield tuple(self.parse_command()) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 216, in parse_command self.handle_error(error) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 358, in handle_error report_error(error) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/errors.py", line 77, in report_error raise exception File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/database/input/bibtex.py", line 214, in parse_command self.required([body_end]) File "/Users/jsick/.virtualenvs/lsst-technote-bootstrap/lib/python3.5/site-packages/pybtex/scanner.py", line 118, in required raise TokenRequired(description, self) pybtex.scanner.TokenRequired: syntax error in line 165: '}' expected pybtex can read lsst-dm.bib just fine using from pybtex.database import parse_file parse_file('lsstdm.bib') But it runs into the above exception when reading it through the bibliography directive with Sphinx. Potentially its an incompatibility with the assumed latex+latin encoding. Debugging in process...
            Hide
            jsick Jonathan Sick added a comment -

            Traced the issue to

            @Unpublished{2012lim-xldbasia,
              author =       {K.-T. Lim},
              title =        {XLDB and the Large Synoptic Survey Telescope},
              note =         {XLDB Asia, Beijing, China, June 22-23 2012 },
              month =        jun,
              year =         2012,
              url =
                              {http://idke.ruc.edu.cn/xldb/www.xldb-asia.org/slides/XLDB%20Asia%20-%20LSST.pdf},
            }
            

            The issue is solved if I escape the percent characters in the URL:

            @Unpublished{2012lim-xldbasia,
              author =       {K.-T. Lim},
              title =        {XLDB and the Large Synoptic Survey Telescope},
              note =         {XLDB Asia, Beijing, China, June 22-23 2012 },
              month =        jun,
              year =         2012,
              url =          {http://idke.ruc.edu.cn/xldb/www.xldb-asia.org/slides/XLDB\%20Asia\%20-\%20LSST.pdf},
            }
            

            Show
            jsick Jonathan Sick added a comment - Traced the issue to @Unpublished{2012lim-xldbasia, author = {K.-T. Lim}, title = {XLDB and the Large Synoptic Survey Telescope}, note = {XLDB Asia, Beijing, China, June 22-23 2012 }, month = jun, year = 2012, url = {http://idke.ruc.edu.cn/xldb/www.xldb-asia.org/slides/XLDB%20Asia%20-%20LSST.pdf}, } The issue is solved if I escape the percent characters in the URL: @Unpublished{2012lim-xldbasia, author = {K.-T. Lim}, title = {XLDB and the Large Synoptic Survey Telescope}, note = {XLDB Asia, Beijing, China, June 22-23 2012 }, month = jun, year = 2012, url = {http://idke.ruc.edu.cn/xldb/www.xldb-asia.org/slides/XLDB\%20Asia\%20-\%20LSST.pdf}, }
            Hide
            jsick Jonathan Sick added a comment -

            Additional fixes needed for lsst-dm.bib:

            • 2016juric-lssteurope

            Fixes needed for refs.bib:

            • Lindegren1993
            • zucker:2004

            Fixes needed for refs_ads.bib:

            • Everything by A&A
            Show
            jsick Jonathan Sick added a comment - Additional fixes needed for lsst-dm.bib : 2016juric-lssteurope Fixes needed for refs.bib : Lindegren1993 zucker:2004 Fixes needed for refs_ads.bib : Everything by A&A
            Hide
            jsick Jonathan Sick added a comment -

            Fixes needed for gaia_livelink_valid.bib:

            • Duplicate: LL:CD-013
            • Duplicate: LL:SAG-CUO-077
            • Duplicate: LL:AKM-001
            • Duplicate: LL:VLE-023 (note, the duplicated keys seem to point to different datasets)
            • Duplicate: LL:IBV-014 (again, different data)
            • Duplicate: LL:IBV-013 (software vs. release note for the same)

            ... There are many more gaia_livelink_valid.bib duplicate keys. Given that it's not be tested by https://github.com/lsst/lsst-texmf/blob/master/tests/test-bibtex.tex it seems unclear whether this bibliography is even relevant to LSST. I'll just ignore fixing it for now.

            Meanwhile, I will submit matches for LSST bibliographies and write a pybtest script for CI to ensure that the bibliographies continue to be compatible with Sphinx.

            Show
            jsick Jonathan Sick added a comment - Fixes needed for gaia_livelink_valid.bib : Duplicate: LL:CD-013 Duplicate: LL:SAG-CUO-077 Duplicate: LL:AKM-001 Duplicate: LL:VLE-023 (note, the duplicated keys seem to point to different datasets) Duplicate: LL:IBV-014 (again, different data) Duplicate: LL:IBV-013 (software vs. release note for the same) ... There are many more gaia_livelink_valid.bib duplicate keys. Given that it's not be tested by https://github.com/lsst/lsst-texmf/blob/master/tests/test-bibtex.tex it seems unclear whether this bibliography is even relevant to LSST. I'll just ignore fixing it for now. Meanwhile, I will submit matches for LSST bibliographies and write a pybtest script for CI to ensure that the bibliographies continue to be compatible with Sphinx.
            Hide
            tjenness Tim Jenness added a comment -

            I'm explicitly ignoring the livelink bibliography. It has so many problems and so many irrelevant entries that I don't think it's worth spending any time on it.

            As for the fixes for percentages. That's fine so long as the URLs in test-bibtex.pdf still go to the right place. I'm inclined to think that it's a bug in the python code if it's valid for bibtex. If we have to fix ADS entries then it's not something I would want to do. I want to reserve the right to be able to regenerate ADS bibtex entries from ADS programmatically based on the bibcodes. We should be able to use ADS bib entries without modification.

            Show
            tjenness Tim Jenness added a comment - I'm explicitly ignoring the livelink bibliography. It has so many problems and so many irrelevant entries that I don't think it's worth spending any time on it. As for the fixes for percentages. That's fine so long as the URLs in test-bibtex.pdf still go to the right place. I'm inclined to think that it's a bug in the python code if it's valid for bibtex . If we have to fix ADS entries then it's not something I would want to do. I want to reserve the right to be able to regenerate ADS bibtex entries from ADS programmatically based on the bibcodes. We should be able to use ADS bib entries without modification.
            Hide
            jsick Jonathan Sick added a comment -

            I agree with the need to use ADS entries verbatim.

            It looks like the pybtex folks plane sphinxcontrib-bibtex for this (https://bitbucket.org/pybtex-devs/pybtex/issues/65/percent-sign-in-url-field-raises-an) and sphinxcontrib-bibtex points to latexcodec (http://sphinxcontrib-bibtex.readthedocs.io/en/latest/usage.html#encoding-percent-signs). Maybe we can PR a fix to latexdoc?

            The immediate ways forward for this ticket at are:

            1. Accept manual patches to the bibtex to escape %, or
            2. Don't automatically populate lsst-bibtex bibliographies in Sphinx technotes and instead recommend that bibliographies be curated separately in each technote.

            I'm hoping to do 1, but will be happy to do 2 to ship this ticket.

            Show
            jsick Jonathan Sick added a comment - I agree with the need to use ADS entries verbatim. It looks like the pybtex folks plane sphinxcontrib-bibtex for this ( https://bitbucket.org/pybtex-devs/pybtex/issues/65/percent-sign-in-url-field-raises-an ) and sphinxcontrib-bibtex points to latexcodec ( http://sphinxcontrib-bibtex.readthedocs.io/en/latest/usage.html#encoding-percent-signs ). Maybe we can PR a fix to latexdoc? The immediate ways forward for this ticket at are: 1. Accept manual patches to the bibtex to escape % , or 2. Don't automatically populate lsst-bibtex bibliographies in Sphinx technotes and instead recommend that bibliographies be curated separately in each technote. I'm hoping to do 1, but will be happy to do 2 to ship this ticket.
            Hide
            tjenness Tim Jenness added a comment -

            Option 1. is fine as a temporary fix if the URLs in PDFs still go to the right place. I think it has to be sphinxcontrib-bibtex that is fixed. URL fields in bib files should not be treated as latex at all. They have to be treated as valid URLs.

            Show
            tjenness Tim Jenness added a comment - Option 1. is fine as a temporary fix if the URLs in PDFs still go to the right place. I think it has to be sphinxcontrib-bibtex that is fixed. URL fields in bib files should not be treated as latex at all. They have to be treated as valid URLs.
            jsick Jonathan Sick made changes -
            Link This issue is triggering DM-11358 [ DM-11358 ]
            Hide
            jsick Jonathan Sick added a comment -

            These PRs are involved:

            1. https://github.com/lsst-sqre/lsst-technote-bootstrap/pull/12 (technote template additions)
            2. https://github.com/lsst-sqre/documenteer/pull/16 (Sphinx extension and bib management tool)
            3. https://github.com/lsst/lsst-texmf/pull/88 (sphinxcontrib-bibtex compatibility for common bib files)
            Show
            jsick Jonathan Sick added a comment - These PRs are involved: https://github.com/lsst-sqre/lsst-technote-bootstrap/pull/12 (technote template additions) https://github.com/lsst-sqre/documenteer/pull/16 (Sphinx extension and bib management tool) https://github.com/lsst/lsst-texmf/pull/88 (sphinxcontrib-bibtex compatibility for common bib files)
            jsick Jonathan Sick made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jsick Jonathan Sick made changes -
            Story Points 1.7 3.1
            jsick Jonathan Sick made changes -
            Epic Link DM-5644 [ 23494 ] DM-9892 [ 31304 ]
            Hide
            tjenness Tim Jenness added a comment -

            Looks okay for now. Comments on PR.

            Show
            tjenness Tim Jenness added a comment - Looks okay for now. Comments on PR.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            jsick Jonathan Sick added a comment -

            Thanks for the astute comments Tim Jenness. I'll put out a community.lsst.org post outlining the new technote features (after I've upgraded technotes to use documenteer 0.2).

            Show
            jsick Jonathan Sick added a comment - Thanks for the astute comments Tim Jenness . I'll put out a community.lsst.org post outlining the new technote features (after I've upgraded technotes to use documenteer 0.2).
            jsick Jonathan Sick made changes -
            Story Points 3.1 3.6
            jsick Jonathan Sick made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                jsick Jonathan Sick
                Reporter:
                jsick Jonathan Sick
                Reviewers:
                Tim Jenness
                Watchers:
                John Parejko, Jonathan Sick, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel