I think the top two choices for having source documentation in the code repo, but nicely browsable are:

markdown + jekyll
restructuredtext + sphinx

Markdown vs ReST:

Markdown is simpler, but less powerful and criticized as immature. Most implementations use their own variant of markdown with (incompatible) extensions specific to that implementation. It's quick to learn, but perhaps limiting and not great for larger linked docs.

ReST: More mature, but still simple. Designed for problems of greater scale, but has "less natural constructs", which can look a little ugly when reading source in plain text. Unclear how frequently we need these constructs. Python's own doc is done in ReST with a Sphinx viewer.

Both are designed to have (more or less) human-readable source text that looks fine (more or less) in plain text.

I think we will go with ReST and Sphinx.
Reasons:

• Mathematical expressions are directly supported in Sphinx. ReST is cleanly extendable to allow LaTeX math to be used, and Sphinx does this. Apparently there is no obvious single method for doing this in markdown, so the implementations that do math in markdown do it slightly differently.
• There is a Sphinx-doxygen bridge called "breathe" ( https://github.com/michaeljones/breathe/tree/master ) that can pull doxygen-generated XML into sphinx, providing some mild unification between user docs and API docs. Doxygen can read markdown directly, but it is unclear how well it can read markdown with a particular variant's math support. ReST+Sphinx seems more workable.

Okay, for this ticket, I'll aim to:

• Test-convert one or two Trac wiki pages into ReST
• Publish the pages with sphinx either at SLAC or some other web-space
• Produce a script (as necessary) to keep the browsable version up-to-date, runnable via cron or a git hook, etc.

I hope this is sufficient for us to get going.

After having discussed with Daniel, we decide I'll try to generate sphinx documentation aside distribution server, on lsst-dev.

Qserv documentation should eventually be automatically produced by buildbot, along with the LSST Stack documentation. Please make sure that your efforts on this ticket mesh with that goal. (Sphinx+ReST is aligned well.)

A. Salnikov interesting question :

is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration (i.e. reST migration) ?

I'll try to setup a minimalist documentation framework with documentation of configuration tool architecture.

Everything is explained in README.txt.

Hi Fabrice,

Sphinx pages look nice

If people look at the new README.txt and go to http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ they may get confused by what they see there. It currently contains a directory listing with only u.fjammes.DM-649-g3b7fe40918 folder in it. I think we should provide more details on that page and explain what that cryptic name means. Can we replace directory listing with a nicer web page with some basic information about releases and what they mean? Also it would be nice to add instructions there for how to install one of the recent production (monthly) releases - 2014_07.0 or 2014_06.0.

Regarding documentation itself:

• quick-start-devel.rst says - "Then re-run configuration and test process, as described in README.txt." but README does not have that info anymore, you should link to configuration section in quick-start.rst instead.
• My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies.
• Prerequisites section - we should probably expand a bit on what to do with older Ubuntu versions, we have script for 13.10, and 12.04 still seems to be popular.
• Installation section - say few words about what INSTALL_DIR is?
• I think in general more details in every section could help, for example in Integration tests it may not be very clear that qserv-benchmark.py --case=01 --load runs a subset of tests from qserv-test-integration.py
• BTW, is there a particular reason why we have two scripts for integration tests (qserv-test-integration.py and qserv-benchmark.py), cannot we do everything in one script? I think that two scripts that do essentially the same may confuse people (and require more things to remember).

Otherwise I think this moves in the right direction, we may want to think more about how to organize information on the top page, but that can be done later on.

eups 1.5.0 is released and is now the default with newinstall.sh

VERY NICE!

One of native speakers in our group should review it, I spotted several language issues:

• The procedure below install --> The procedure below installs
• developpers --> developers (this is misspelled in several places)
• Configuration data are --> Configuration data is
• give additional informations --> provides addition information

"# script below will ask some questions, answer 'yes' everywhere". If that is the case, why is the script even asking?

Do we need space before every ":"? (I always noticed and always wondered)

"as described in README.txt." - do we still have the readme.txt?

We should also explain that "setup qserv_testdata" needs to be followed by setting up and configuring qserv, otherwise it is not really set up

Hi,

Thanks for your remarks, i've adressed them in my last commits.

Two precisions :

• About Ubuntu 12.04 : could you please provide me the dependencies list so that I integrate it in the documentation ? Nevertheless, I think that while we're in active development process, we should only support a limited number of distributions. And Ubuntu 14.04 is the new "Long Term Support" version since april.
• About "setup qserv_testdata", add of --testdata_dir option in qserv-test-integration.py solve this, please see help about this option below.
• Difference between qserv-test-integration.py and qserv-benchmark.py (renamed to qserv-check-integration.py as a proposal) :
  • The first one relies on unittest python framework and aims at being used in other tools like continuous integration platform (which performs and monitor many tests after each build) or cluster management tool (to check if everything is well installed). It aims at launching automatically a large panel of tests and retrieve their results in unittest module standard way.
  • The second one, whereas it runs the same algorithm, is more development and debugging oriented as it allow to run more fine grained tests (on one dataset, or, why not, on one query). Furthermore it isn't encapsulated in unittest framework. I tried to explains that in these scripts --help options.

fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/DM-649 %) $ qserv-test-integration.py --help

usage: qserv-test-integration.py [-h] [-v {DEBUG,INFO,WARNING,FATAL,ERROR}]

                                 [-t TESTDATA_DIR]

Qserv integration tests suite. Relies on python unit testing framework,

provide test meta-data which can be used for example in a continuous

integration framework or by a cluster management tool. Configuration values

are read from ~/.lsst/qserv.conf.

optional arguments:

  -h, --help            show this help message and exit

  -v {DEBUG,INFO,WARNING,FATAL,ERROR}, --verbose-level {DEBUG,INFO,WARNING,FATAL,ERROR}

                        verbosity level (default: INFO)

  -t TESTDATA_DIR, --testdata-dir TESTDATA_DIR

                        full path to directory containing test datasets. This

                        value is set, by precedence, by this option, then by

                        QSERV_TESTDATA_DIR environment variable if not empty,

                        and finally by setting testdata_dir value in

                        ~/.lsst/qserv.conf (default:

                        /data/fjammes/stack/Linux64/qserv_testdata/2014_06.0)

Please also note that DM-930 needs to be integrated in these ticket in order to close this review.

Thanks,

Hi Fabrice,

thanks for all fixes. One minor thing, in the piece of "code":

INSTALL_DIR = root/directory/where/qserv/stack/will/be/installed

it's better to remove spaces around assignment, should help people who copy and paste this (they will modify it anyway but still).

I still do not understand the reason to keep qserv-test-integration.py and qserv-check-integration.py separate, they do essentially the same job, it seems logical to me that they can be unified.

Regarding Ubuntu 12.04 - let me try to install VM and figure out exactly what is needed.

Anyways, I'm satisfied and remove myself from reviewers.

Cheers,
Andy

Fabrice, I just remembered one more thing: the "-r <qserv run dir>" option is very handy, it needs to be well documented (and explained that the default is set to ~/qserv-run)

Fabrice, I am pasting comments from Daniel here. I believe this completes the review

"Welcome to Qserv's documentation!" should just be "Qserv 2014_08.0 documentation <http://lsst-web.ncsa.illinois.edu/%7Efjammes/qserv-doc/u.fjammes.DM-649-gee2bed707c/index.html#>" or just "Qserv documentation"

"Software architecture" --> "Administration reference"
---------------------------
quickstart page:

"FOR DEVELOPERS:..." --> "This procedure is for RELEASED and PUBLISHED qserv software. Developers interested in modifying code or running unreleased versions should see Quick start guide for developers "

Configuration
The code block should just be:
qserv-configure.py --all

The alert:"Qserv commands listed below provides advanced options, use –help option to discover how to use it." is unnecessary. Just put a note below the command saying something like: "For more options, run qserv-configure.py --help". In the future, we should at least list, of not document some of the options. The whole point of this page is to "document".

In general, the alert blocks ( that have ) should only be used to point out "gotchas" or ways in which the reader may possibly screw up. The block at the top of the page is appropriate--it just says "Hey, watch out" and is not really adding to the essentials of the page.

Integration tests
"Integration tests" -> "Testing"

The "Default value for ..." alert should probably be a comment in the code block below.

Quick start guide for developers

You want something to say: "Using Qserv with your own custom code or arbitrary versions can be done by connecting your local git clone with an existing released qserv installation (e.g. from Quick start guide). This should be stated above "Setup current Qserv version in eups".

Setup current Qserv version in eups:

text should say: "These commands point the infrastructure of an existing qserv installation to work with a local Qserv git repository."

below code block:
Once the qserv eups stack is integrated with your local Qserv repository, you will need to re-configure and (if desired) re-test as described in the Quick start guide.

I'll have to look at "Software architecture" later-it's a bit messy and not really user documentation-it looks like like a Trac/JIRA ticket or epic.

Hope this helps,
-Daniel

Looking at the documentation again it occurs to me that instructions as written now are for installing the most recent qserv version. Should they have explicit release name in the "eups distrib unstall qserv" and "setup qserv"?

One more comment (sorry I continue commenting on a closed issue, I don't feel this deserves a new ticket yet). I managed to build and run qserv on Ubuntu 12.04 clean server install after I added these packages:

make

g++

libz-dev

gettext

flex

bison

scons

libreadline-dev

libncurses5-dev

cmake

python-dev

python-numpy

swig

libbz2-dev

python-setuptools

pkg-config

libglib2.0-dev

openjdk-7-jdk

Hi, i've integrated most of your post-review comment directly into master. About Daniel comments, i've only replaced "existing released qserv installation " with "eups software stack containing Qserv dependencies". Indeed, in last developer doc updates, it is no more required to install a Qserv release prior to installing it from user git repository.