framework for documenting "how to run qserv"

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2.5
• Sprint:
DB_S14_06, DB_S14_07, DB_S14_08
• Team:
Data Access and Database

Description

We need to have permanent location for how-to-run-qserv, currently we keep it in https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/Hackathon2/howToRunQserv. It should be in the code repo. Need to decide on how we format it, and need to expose it on our Qserv trac/confluence pages

Activity

Hide
Daniel Wang [X] (Inactive) added a comment -

I think the top two choices for having source documentation in the code repo, but nicely browsable are:

markdown + jekyll
restructuredtext + sphinx

Markdown vs ReST:

Markdown is simpler, but less powerful and criticized as immature. Most implementations use their own variant of markdown with (incompatible) extensions specific to that implementation. It's quick to learn, but perhaps limiting and not great for larger linked docs.

ReST: More mature, but still simple. Designed for problems of greater scale, but has "less natural constructs", which can look a little ugly when reading source in plain text. Unclear how frequently we need these constructs. Python's own doc is done in ReST with a Sphinx viewer.

Both are designed to have (more or less) human-readable source text that looks fine (more or less) in plain text.

Show
Daniel Wang [X] (Inactive) added a comment - I think the top two choices for having source documentation in the code repo, but nicely browsable are: markdown + jekyll restructuredtext + sphinx Markdown vs ReST: Markdown is simpler, but less powerful and criticized as immature. Most implementations use their own variant of markdown with (incompatible) extensions specific to that implementation. It's quick to learn, but perhaps limiting and not great for larger linked docs. ReST: More mature, but still simple. Designed for problems of greater scale, but has "less natural constructs", which can look a little ugly when reading source in plain text. Unclear how frequently we need these constructs. Python's own doc is done in ReST with a Sphinx viewer. Both are designed to have (more or less) human-readable source text that looks fine (more or less) in plain text.
Hide
Daniel Wang [X] (Inactive) added a comment -

I think we will go with ReST and Sphinx.
Reasons:

• Mathematical expressions are directly supported in Sphinx. ReST is cleanly extendable to allow LaTeX math to be used, and Sphinx does this. Apparently there is no obvious single method for doing this in markdown, so the implementations that do math in markdown do it slightly differently.
• There is a Sphinx-doxygen bridge called "breathe" ( https://github.com/michaeljones/breathe/tree/master ) that can pull doxygen-generated XML into sphinx, providing some mild unification between user docs and API docs. Doxygen can read markdown directly, but it is unclear how well it can read markdown with a particular variant's math support. ReST+Sphinx seems more workable.

Okay, for this ticket, I'll aim to:

• Test-convert one or two Trac wiki pages into ReST
• Publish the pages with sphinx either at SLAC or some other web-space
• Produce a script (as necessary) to keep the browsable version up-to-date, runnable via cron or a git hook, etc.

I hope this is sufficient for us to get going.

Show
Daniel Wang [X] (Inactive) added a comment - I think we will go with ReST and Sphinx. Reasons: Mathematical expressions are directly supported in Sphinx. ReST is cleanly extendable to allow LaTeX math to be used, and Sphinx does this. Apparently there is no obvious single method for doing this in markdown, so the implementations that do math in markdown do it slightly differently. There is a Sphinx-doxygen bridge called "breathe" ( https://github.com/michaeljones/breathe/tree/master ) that can pull doxygen-generated XML into sphinx, providing some mild unification between user docs and API docs. Doxygen can read markdown directly, but it is unclear how well it can read markdown with a particular variant's math support. ReST+Sphinx seems more workable. Okay, for this ticket, I'll aim to: Test-convert one or two Trac wiki pages into ReST Publish the pages with sphinx either at SLAC or some other web-space Produce a script (as necessary) to keep the browsable version up-to-date, runnable via cron or a git hook, etc. I hope this is sufficient for us to get going.
Hide
Fabrice Jammes added a comment -

After having discussed with Daniel, we decide I'll try to generate sphinx documentation aside distribution server, on lsst-dev.

Show
Fabrice Jammes added a comment - After having discussed with Daniel, we decide I'll try to generate sphinx documentation aside distribution server, on lsst-dev.
Hide
Kian-Tat Lim added a comment -

Qserv documentation should eventually be automatically produced by buildbot, along with the LSST Stack documentation. Please make sure that your efforts on this ticket mesh with that goal. (Sphinx+ReST is aligned well.)

Show
Kian-Tat Lim added a comment - Qserv documentation should eventually be automatically produced by buildbot, along with the LSST Stack documentation. Please make sure that your efforts on this ticket mesh with that goal. (Sphinx+ReST is aligned well.)
Hide
Fabrice Jammes added a comment -

A. Salnikov interesting question :

is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration (i.e. reST migration) ?

Show
Fabrice Jammes added a comment - A. Salnikov interesting question : is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration (i.e. reST migration) ?
Hide
Fabrice Jammes added a comment - - edited

I'll try to setup a minimalist documentation framework with documentation of configuration tool architecture.

Show
Fabrice Jammes added a comment - - edited I'll try to setup a minimalist documentation framework with documentation of configuration tool architecture.
Hide
Fabrice Jammes added a comment -

Show
Hide
Andy Salnikov added a comment -

Hi Fabrice,

Sphinx pages look nice

If people look at the new README.txt and go to http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ they may get confused by what they see there. It currently contains a directory listing with only u.fjammes.DM-649-g3b7fe40918 folder in it. I think we should provide more details on that page and explain what that cryptic name means. Can we replace directory listing with a nicer web page with some basic information about releases and what they mean? Also it would be nice to add instructions there for how to install one of the recent production (monthly) releases - 2014_07.0 or 2014_06.0.

Regarding documentation itself:

• quick-start-devel.rst says - "Then re-run configuration and test process, as described in README.txt." but README does not have that info anymore, you should link to configuration section in quick-start.rst instead.
• My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies.
• Prerequisites section - we should probably expand a bit on what to do with older Ubuntu versions, we have script for 13.10, and 12.04 still seems to be popular.
• Installation section - say few words about what INSTALL_DIR is?
• I think in general more details in every section could help, for example in Integration tests it may not be very clear that qserv-benchmark.py --case=01 --load runs a subset of tests from qserv-test-integration.py
• BTW, is there a particular reason why we have two scripts for integration tests (qserv-test-integration.py and qserv-benchmark.py), cannot we do everything in one script? I think that two scripts that do essentially the same may confuse people (and require more things to remember).

Otherwise I think this moves in the right direction, we may want to think more about how to organize information on the top page, but that can be done later on.

Show
Andy Salnikov added a comment - Hi Fabrice, Sphinx pages look nice If people look at the new README.txt and go to http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ they may get confused by what they see there. It currently contains a directory listing with only u.fjammes. DM-649 -g3b7fe40918 folder in it. I think we should provide more details on that page and explain what that cryptic name means. Can we replace directory listing with a nicer web page with some basic information about releases and what they mean? Also it would be nice to add instructions there for how to install one of the recent production (monthly) releases - 2014_07.0 or 2014_06.0. Regarding documentation itself: quick-start-devel.rst says - "Then re-run configuration and test process, as described in README.txt." but README does not have that info anymore, you should link to configuration section in quick-start.rst instead. My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies. Prerequisites section - we should probably expand a bit on what to do with older Ubuntu versions, we have script for 13.10, and 12.04 still seems to be popular. Installation section - say few words about what INSTALL_DIR is? I think in general more details in every section could help, for example in Integration tests it may not be very clear that qserv-benchmark.py --case=01 --load runs a subset of tests from qserv-test-integration.py BTW, is there a particular reason why we have two scripts for integration tests ( qserv-test-integration.py and qserv-benchmark.py ), cannot we do everything in one script? I think that two scripts that do essentially the same may confuse people (and require more things to remember). Otherwise I think this moves in the right direction, we may want to think more about how to organize information on the top page, but that can be done later on.
Hide
Kian-Tat Lim added a comment -

eups 1.5.0 is released and is now the default with newinstall.sh

Show
Kian-Tat Lim added a comment - eups 1.5.0 is released and is now the default with newinstall.sh
Hide
Jacek Becla added a comment -

VERY NICE!

One of native speakers in our group should review it, I spotted several language issues:

• The procedure below install --> The procedure below installs
• developpers --> developers (this is misspelled in several places)
• Configuration data are --> Configuration data is

"# script below will ask some questions, answer 'yes' everywhere". If that is the case, why is the script even asking?

Do we need space before every ":"? (I always noticed and always wondered)

Show
Jacek Becla added a comment - VERY NICE! One of native speakers in our group should review it, I spotted several language issues: The procedure below install --> The procedure below installs developpers --> developers (this is misspelled in several places) Configuration data are --> Configuration data is give additional informations --> provides addition information "# script below will ask some questions, answer 'yes' everywhere". If that is the case, why is the script even asking? Do we need space before every ":"? (I always noticed and always wondered) "as described in README.txt." - do we still have the readme.txt?
Hide
Jacek Becla added a comment -

We should also explain that "setup qserv_testdata" needs to be followed by setting up and configuring qserv, otherwise it is not really set up

Show
Jacek Becla added a comment - We should also explain that "setup qserv_testdata" needs to be followed by setting up and configuring qserv, otherwise it is not really set up
Hide
Fabrice Jammes added a comment - - edited

Hi,

Two precisions :

• About Ubuntu 12.04 : could you please provide me the dependencies list so that I integrate it in the documentation ? Nevertheless, I think that while we're in active development process, we should only support a limited number of distributions. And Ubuntu 14.04 is the new "Long Term Support" version since april.
• Difference between qserv-test-integration.py and qserv-benchmark.py (renamed to qserv-check-integration.py as a proposal) :
• The first one relies on unittest python framework and aims at being used in other tools like continuous integration platform (which performs and monitor many tests after each build) or cluster management tool (to check if everything is well installed). It aims at launching automatically a large panel of tests and retrieve their results in unittest module standard way.
• The second one, whereas it runs the same algorithm, is more development and debugging oriented as it allow to run more fine grained tests (on one dataset, or, why not, on one query). Furthermore it isn't encapsulated in unittest framework. I tried to explains that in these scripts --help options.

 fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/DM-649 %) $qserv-test-integration.py --help usage: qserv-test-integration.py [-h] [-v {DEBUG,INFO,WARNING,FATAL,ERROR}]  [-t TESTDATA_DIR]   Qserv integration tests suite. Relies on python unit testing framework, provide test meta-data which can be used for example in a continuous integration framework or by a cluster management tool. Configuration values are read from ~/.lsst/qserv.conf.   optional arguments:  -h, --help show this help message and exit  -v {DEBUG,INFO,WARNING,FATAL,ERROR}, --verbose-level {DEBUG,INFO,WARNING,FATAL,ERROR}  verbosity level (default: INFO)  -t TESTDATA_DIR, --testdata-dir TESTDATA_DIR  full path to directory containing test datasets. This  value is set, by precedence, by this option, then by  QSERV_TESTDATA_DIR environment variable if not empty,  and finally by setting testdata_dir value in  ~/.lsst/qserv.conf (default:  /data/fjammes/stack/Linux64/qserv_testdata/2014_06.0) Please also note that DM-930 needs to be integrated in these ticket in order to close this review. Thanks, Show Fabrice Jammes added a comment - - edited Hi, Thanks for your remarks, i've adressed them in my last commits. Two precisions : About Ubuntu 12.04 : could you please provide me the dependencies list so that I integrate it in the documentation ? Nevertheless, I think that while we're in active development process, we should only support a limited number of distributions. And Ubuntu 14.04 is the new "Long Term Support" version since april. About "setup qserv_testdata", add of --testdata_dir option in qserv-test-integration.py solve this, please see help about this option below. Difference between qserv-test-integration.py and qserv-benchmark.py (renamed to qserv-check-integration.py as a proposal) : The first one relies on unittest python framework and aims at being used in other tools like continuous integration platform (which performs and monitor many tests after each build) or cluster management tool (to check if everything is well installed). It aims at launching automatically a large panel of tests and retrieve their results in unittest module standard way. The second one, whereas it runs the same algorithm, is more development and debugging oriented as it allow to run more fine grained tests (on one dataset, or, why not, on one query). Furthermore it isn't encapsulated in unittest framework. I tried to explains that in these scripts --help options. fjammes@clrlsst-dbmaster-vm:~ /src/qserv (u /fjammes/DM-649 %)$ qserv- test -integration.py --help usage: qserv- test -integration.py [-h] [- v {DEBUG,INFO,WARNING,FATAL,ERROR}] [-t TESTDATA_DIR]   Qserv integration tests suite. Relies on python unit testing framework, provide test meta-data which can be used for example in a continuous integration framework or by a cluster management tool. Configuration values are read from ~/.lsst /qserv .conf.   optional arguments: -h, --help show this help message and exit - v {DEBUG,INFO,WARNING,FATAL,ERROR}, --verbose-level {DEBUG,INFO,WARNING,FATAL,ERROR} verbosity level (default: INFO) -t TESTDATA_DIR, --testdata- dir TESTDATA_DIR full path to directory containing test datasets. This value is set , by precedence, by this option, then by QSERV_TESTDATA_DIR environment variable if not empty, and finally by setting testdata_dir value in ~/.lsst /qserv .conf (default: /data/fjammes/stack/Linux64/qserv_testdata/2014_06 .0) Please also note that DM-930 needs to be integrated in these ticket in order to close this review. Thanks,
Hide
Andy Salnikov added a comment -

Hi Fabrice,

thanks for all fixes. One minor thing, in the piece of "code":

 INSTALL_DIR = root/directory/where/qserv/stack/will/be/installed

it's better to remove spaces around assignment, should help people who copy and paste this (they will modify it anyway but still).

I still do not understand the reason to keep qserv-test-integration.py and qserv-check-integration.py separate, they do essentially the same job, it seems logical to me that they can be unified.

Regarding Ubuntu 12.04 - let me try to install VM and figure out exactly what is needed.

Anyways, I'm satisfied and remove myself from reviewers.

Cheers,
Andy

Show
Andy Salnikov added a comment - Hi Fabrice, thanks for all fixes. One minor thing, in the piece of "code": INSTALL_DIR = root/directory/where/qserv/stack/will/be/installed it's better to remove spaces around assignment, should help people who copy and paste this (they will modify it anyway but still). I still do not understand the reason to keep qserv-test-integration.py and qserv-check-integration.py separate, they do essentially the same job, it seems logical to me that they can be unified. Regarding Ubuntu 12.04 - let me try to install VM and figure out exactly what is needed. Anyways, I'm satisfied and remove myself from reviewers. Cheers, Andy
Hide
Jacek Becla added a comment -

Fabrice, I just remembered one more thing: the "-r <qserv run dir>" option is very handy, it needs to be well documented (and explained that the default is set to ~/qserv-run)

Show
Jacek Becla added a comment - Fabrice, I just remembered one more thing: the "-r <qserv run dir>" option is very handy, it needs to be well documented (and explained that the default is set to ~/qserv-run)
Hide
Jacek Becla added a comment -

Fabrice, I am pasting comments from Daniel here. I believe this completes the review

"Welcome to Qserv's documentation!" should just be "Qserv 2014_08.0 documentation <http://lsst-web.ncsa.illinois.edu/%7Efjammes/qserv-doc/u.fjammes.DM-649-gee2bed707c/index.html#>" or just "Qserv documentation"

---------------------------
quickstart page:

"FOR DEVELOPERS:..." --> "This procedure is for RELEASED and PUBLISHED qserv software. Developers interested in modifying code or running unreleased versions should see Quick start guide for developers "

Configuration
The code block should just be:
qserv-configure.py --all

The alert:"Qserv commands listed below provides advanced options, use –help option to discover how to use it." is unnecessary. Just put a note below the command saying something like: "For more options, run qserv-configure.py --help". In the future, we should at least list, of not document some of the options. The whole point of this page is to "document".

In general, the alert blocks ( that have ) should only be used to point out "gotchas" or ways in which the reader may possibly screw up. The block at the top of the page is appropriate--it just says "Hey, watch out" and is not really adding to the essentials of the page.

Integration tests
"Integration tests" -> "Testing"

The "Default value for ..." alert should probably be a comment in the code block below.

Quick start guide for developers

You want something to say: "Using Qserv with your own custom code or arbitrary versions can be done by connecting your local git clone with an existing released qserv installation (e.g. from Quick start guide). This should be stated above "Setup current Qserv version in eups".

Setup current Qserv version in eups:

text should say: "These commands point the infrastructure of an existing qserv installation to work with a local Qserv git repository."

below code block:
Once the qserv eups stack is integrated with your local Qserv repository, you will need to re-configure and (if desired) re-test as described in the Quick start guide.

I'll have to look at "Software architecture" later-it's a bit messy and not really user documentation-it looks like like a Trac/JIRA ticket or epic.

Hope this helps,
-Daniel

Show
Jacek Becla added a comment - Fabrice, I am pasting comments from Daniel here. I believe this completes the review "Welcome to Qserv's documentation!" should just be "Qserv 2014_08.0 documentation < http://lsst-web.ncsa.illinois.edu/%7Efjammes/qserv-doc/u.fjammes.DM-649-gee2bed707c/index.html# >" or just "Qserv documentation" "Software architecture" --> "Administration reference" --------------------------- quickstart page: "FOR DEVELOPERS:..." --> "This procedure is for RELEASED and PUBLISHED qserv software. Developers interested in modifying code or running unreleased versions should see Quick start guide for developers " Configuration The code block should just be: qserv-configure.py --all The alert:"Qserv commands listed below provides advanced options, use –help option to discover how to use it." is unnecessary. Just put a note below the command saying something like: "For more options, run qserv-configure.py --help". In the future, we should at least list, of not document some of the options. The whole point of this page is to "document". In general, the alert blocks ( that have ) should only be used to point out "gotchas" or ways in which the reader may possibly screw up. The block at the top of the page is appropriate--it just says "Hey, watch out" and is not really adding to the essentials of the page. Integration tests "Integration tests" -> "Testing" The "Default value for ..." alert should probably be a comment in the code block below. Quick start guide for developers You want something to say: "Using Qserv with your own custom code or arbitrary versions can be done by connecting your local git clone with an existing released qserv installation (e.g. from Quick start guide). This should be stated above "Setup current Qserv version in eups". Setup current Qserv version in eups: text should say: "These commands point the infrastructure of an existing qserv installation to work with a local Qserv git repository." below code block: Once the qserv eups stack is integrated with your local Qserv repository, you will need to re-configure and (if desired) re-test as described in the Quick start guide. I'll have to look at "Software architecture" later- it's a bit messy and not really user documentation -it looks like like a Trac/JIRA ticket or epic. Hope this helps, -Daniel
Hide
Andy Salnikov added a comment -

Looking at the documentation again it occurs to me that instructions as written now are for installing the most recent qserv version. Should they have explicit release name in the "eups distrib unstall qserv" and "setup qserv"?

Show
Andy Salnikov added a comment - Looking at the documentation again it occurs to me that instructions as written now are for installing the most recent qserv version. Should they have explicit release name in the "eups distrib unstall qserv" and "setup qserv"?
Hide
Andy Salnikov added a comment -

One more comment (sorry I continue commenting on a closed issue, I don't feel this deserves a new ticket yet). I managed to build and run qserv on Ubuntu 12.04 clean server install after I added these packages:

 make g++ libz-dev gettext flex bison scons libreadline-dev libncurses5-dev cmake python-dev python-numpy swig libbz2-dev python-setuptools pkg-config libglib2.0-dev openjdk-7-jdk

Show
Andy Salnikov added a comment - One more comment (sorry I continue commenting on a closed issue, I don't feel this deserves a new ticket yet). I managed to build and run qserv on Ubuntu 12.04 clean server install after I added these packages: make g++ libz-dev gettext flex bison scons libreadline-dev libncurses5-dev cmake python-dev python-numpy swig libbz2-dev python-setuptools pkg-config libglib2.0-dev openjdk-7-jdk
Hide
Fabrice Jammes added a comment -

Hi, i've integrated most of your post-review comment directly into master. About Daniel comments, i've only replaced "existing released qserv installation " with "eups software stack containing Qserv dependencies". Indeed, in last developer doc updates, it is no more required to install a Qserv release prior to installing it from user git repository.

Show
Fabrice Jammes added a comment - Hi, i've integrated most of your post-review comment directly into master. About Daniel comments, i've only replaced "existing released qserv installation " with "eups software stack containing Qserv dependencies". Indeed, in last developer doc updates, it is no more required to install a Qserv release prior to installing it from user git repository.

People

Assignee:
Fabrice Jammes
Reporter:
Fritz Mueller
Reviewers:
Daniel Wang [X] (Inactive), Jacek Becla
Watchers:
Andy Salnikov, Daniel Wang [X] (Inactive), Fabrice Jammes, Jacek Becla, Kian-Tat Lim