Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-649

framework for documenting "how to run qserv"

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • Qserv
    • None
    • 2.5
    • DB_S14_06, DB_S14_07, DB_S14_08
    • Data Access and Database

    Description

      We need to have permanent location for how-to-run-qserv, currently we keep it in https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/Hackathon2/howToRunQserv. It should be in the code repo. Need to decide on how we format it, and need to expose it on our Qserv trac/confluence pages

      Attachments

        Issue Links

          Activity

            I think the top two choices for having source documentation in the code repo, but nicely browsable are:

            markdown + jekyll
            restructuredtext + sphinx

            Markdown vs ReST:

            Markdown is simpler, but less powerful and criticized as immature. Most implementations use their own variant of markdown with (incompatible) extensions specific to that implementation. It's quick to learn, but perhaps limiting and not great for larger linked docs.

            ReST: More mature, but still simple. Designed for problems of greater scale, but has "less natural constructs", which can look a little ugly when reading source in plain text. Unclear how frequently we need these constructs. Python's own doc is done in ReST with a Sphinx viewer.

            Both are designed to have (more or less) human-readable source text that looks fine (more or less) in plain text.

            danielw Daniel Wang [X] (Inactive) added a comment - I think the top two choices for having source documentation in the code repo, but nicely browsable are: markdown + jekyll restructuredtext + sphinx Markdown vs ReST: Markdown is simpler, but less powerful and criticized as immature. Most implementations use their own variant of markdown with (incompatible) extensions specific to that implementation. It's quick to learn, but perhaps limiting and not great for larger linked docs. ReST: More mature, but still simple. Designed for problems of greater scale, but has "less natural constructs", which can look a little ugly when reading source in plain text. Unclear how frequently we need these constructs. Python's own doc is done in ReST with a Sphinx viewer. Both are designed to have (more or less) human-readable source text that looks fine (more or less) in plain text.

            I think we will go with ReST and Sphinx.
            Reasons:

            • Mathematical expressions are directly supported in Sphinx. ReST is cleanly extendable to allow LaTeX math to be used, and Sphinx does this. Apparently there is no obvious single method for doing this in markdown, so the implementations that do math in markdown do it slightly differently.
            • There is a Sphinx-doxygen bridge called "breathe" ( https://github.com/michaeljones/breathe/tree/master ) that can pull doxygen-generated XML into sphinx, providing some mild unification between user docs and API docs. Doxygen can read markdown directly, but it is unclear how well it can read markdown with a particular variant's math support. ReST+Sphinx seems more workable.

            Okay, for this ticket, I'll aim to:

            • Test-convert one or two Trac wiki pages into ReST
            • Publish the pages with sphinx either at SLAC or some other web-space
            • Produce a script (as necessary) to keep the browsable version up-to-date, runnable via cron or a git hook, etc.

            I hope this is sufficient for us to get going.

            danielw Daniel Wang [X] (Inactive) added a comment - I think we will go with ReST and Sphinx. Reasons: Mathematical expressions are directly supported in Sphinx. ReST is cleanly extendable to allow LaTeX math to be used, and Sphinx does this. Apparently there is no obvious single method for doing this in markdown, so the implementations that do math in markdown do it slightly differently. There is a Sphinx-doxygen bridge called "breathe" ( https://github.com/michaeljones/breathe/tree/master ) that can pull doxygen-generated XML into sphinx, providing some mild unification between user docs and API docs. Doxygen can read markdown directly, but it is unclear how well it can read markdown with a particular variant's math support. ReST+Sphinx seems more workable. Okay, for this ticket, I'll aim to: Test-convert one or two Trac wiki pages into ReST Publish the pages with sphinx either at SLAC or some other web-space Produce a script (as necessary) to keep the browsable version up-to-date, runnable via cron or a git hook, etc. I hope this is sufficient for us to get going.

            After having discussed with Daniel, we decide I'll try to generate sphinx documentation aside distribution server, on lsst-dev.

            jammes Fabrice Jammes added a comment - After having discussed with Daniel, we decide I'll try to generate sphinx documentation aside distribution server, on lsst-dev.
            ktl Kian-Tat Lim added a comment -

            Qserv documentation should eventually be automatically produced by buildbot, along with the LSST Stack documentation. Please make sure that your efforts on this ticket mesh with that goal. (Sphinx+ReST is aligned well.)

            ktl Kian-Tat Lim added a comment - Qserv documentation should eventually be automatically produced by buildbot, along with the LSST Stack documentation. Please make sure that your efforts on this ticket mesh with that goal. (Sphinx+ReST is aligned well.)

            A. Salnikov interesting question :

            is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration (i.e. reST migration) ?

            jammes Fabrice Jammes added a comment - A. Salnikov interesting question : is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration (i.e. reST migration) ?
            jammes Fabrice Jammes added a comment - - edited

            I'll try to setup a minimalist documentation framework with documentation of configuration tool architecture.

            jammes Fabrice Jammes added a comment - - edited I'll try to setup a minimalist documentation framework with documentation of configuration tool architecture.

            Everything is explained in README.txt.

            jammes Fabrice Jammes added a comment - Everything is explained in README.txt.

            Hi Fabrice,

            Sphinx pages look nice

            If people look at the new README.txt and go to http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ they may get confused by what they see there. It currently contains a directory listing with only u.fjammes.DM-649-g3b7fe40918 folder in it. I think we should provide more details on that page and explain what that cryptic name means. Can we replace directory listing with a nicer web page with some basic information about releases and what they mean? Also it would be nice to add instructions there for how to install one of the recent production (monthly) releases - 2014_07.0 or 2014_06.0.

            Regarding documentation itself:

            • quick-start-devel.rst says - "Then re-run configuration and test process, as described in README.txt." but README does not have that info anymore, you should link to configuration section in quick-start.rst instead.
            • My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies.
            • Prerequisites section - we should probably expand a bit on what to do with older Ubuntu versions, we have script for 13.10, and 12.04 still seems to be popular.
            • Installation section - say few words about what INSTALL_DIR is?
            • I think in general more details in every section could help, for example in Integration tests it may not be very clear that qserv-benchmark.py --case=01 --load runs a subset of tests from qserv-test-integration.py
            • BTW, is there a particular reason why we have two scripts for integration tests (qserv-test-integration.py and qserv-benchmark.py), cannot we do everything in one script? I think that two scripts that do essentially the same may confuse people (and require more things to remember).

            Otherwise I think this moves in the right direction, we may want to think more about how to organize information on the top page, but that can be done later on.

            salnikov Andy Salnikov added a comment - Hi Fabrice, Sphinx pages look nice If people look at the new README.txt and go to http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ they may get confused by what they see there. It currently contains a directory listing with only u.fjammes. DM-649 -g3b7fe40918 folder in it. I think we should provide more details on that page and explain what that cryptic name means. Can we replace directory listing with a nicer web page with some basic information about releases and what they mean? Also it would be nice to add instructions there for how to install one of the recent production (monthly) releases - 2014_07.0 or 2014_06.0. Regarding documentation itself: quick-start-devel.rst says - "Then re-run configuration and test process, as described in README.txt." but README does not have that info anymore, you should link to configuration section in quick-start.rst instead. My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies. Prerequisites section - we should probably expand a bit on what to do with older Ubuntu versions, we have script for 13.10, and 12.04 still seems to be popular. Installation section - say few words about what INSTALL_DIR is? I think in general more details in every section could help, for example in Integration tests it may not be very clear that qserv-benchmark.py --case=01 --load runs a subset of tests from qserv-test-integration.py BTW, is there a particular reason why we have two scripts for integration tests ( qserv-test-integration.py and qserv-benchmark.py ), cannot we do everything in one script? I think that two scripts that do essentially the same may confuse people (and require more things to remember). Otherwise I think this moves in the right direction, we may want to think more about how to organize information on the top page, but that can be done later on.

            eups 1.5.0 is released and is now the default with newinstall.sh

            ktl Kian-Tat Lim added a comment - eups 1.5.0 is released and is now the default with newinstall.sh

            VERY NICE!

            One of native speakers in our group should review it, I spotted several language issues:

            • The procedure below install --> The procedure below installs
            • developpers --> developers (this is misspelled in several places)
            • Configuration data are --> Configuration data is
            • give additional informations --> provides addition information

            "# script below will ask some questions, answer 'yes' everywhere". If that is the case, why is the script even asking?

            Do we need space before every ":"? (I always noticed and always wondered)

            "as described in README.txt." - do we still have the readme.txt?

            jbecla Jacek Becla (Inactive) added a comment - VERY NICE! One of native speakers in our group should review it, I spotted several language issues: The procedure below install --> The procedure below installs developpers --> developers (this is misspelled in several places) Configuration data are --> Configuration data is give additional informations --> provides addition information "# script below will ask some questions, answer 'yes' everywhere". If that is the case, why is the script even asking? Do we need space before every ":"? (I always noticed and always wondered) "as described in README.txt." - do we still have the readme.txt?

            We should also explain that "setup qserv_testdata" needs to be followed by setting up and configuring qserv, otherwise it is not really set up

            jbecla Jacek Becla (Inactive) added a comment - We should also explain that "setup qserv_testdata" needs to be followed by setting up and configuring qserv, otherwise it is not really set up
            jammes Fabrice Jammes added a comment - - edited

            Hi,

            Thanks for your remarks, i've adressed them in my last commits.

            Two precisions :

            • About Ubuntu 12.04 : could you please provide me the dependencies list so that I integrate it in the documentation ? Nevertheless, I think that while we're in active development process, we should only support a limited number of distributions. And Ubuntu 14.04 is the new "Long Term Support" version since april.
            • About "setup qserv_testdata", add of --testdata_dir option in qserv-test-integration.py solve this, please see help about this option below.
            • Difference between qserv-test-integration.py and qserv-benchmark.py (renamed to qserv-check-integration.py as a proposal) :
              • The first one relies on unittest python framework and aims at being used in other tools like continuous integration platform (which performs and monitor many tests after each build) or cluster management tool (to check if everything is well installed). It aims at launching automatically a large panel of tests and retrieve their results in unittest module standard way.
              • The second one, whereas it runs the same algorithm, is more development and debugging oriented as it allow to run more fine grained tests (on one dataset, or, why not, on one query). Furthermore it isn't encapsulated in unittest framework. I tried to explains that in these scripts --help options.

            fjammes@clrlsst-dbmaster-vm:~/src/qserv (u/fjammes/DM-649 %) $ qserv-test-integration.py --help
            usage: qserv-test-integration.py [-h] [-v {DEBUG,INFO,WARNING,FATAL,ERROR}]
                                             [-t TESTDATA_DIR]
             
            Qserv integration tests suite. Relies on python unit testing framework,
            provide test meta-data which can be used for example in a continuous
            integration framework or by a cluster management tool. Configuration values
            are read from ~/.lsst/qserv.conf.
             
            optional arguments:
              -h, --help            show this help message and exit
              -v {DEBUG,INFO,WARNING,FATAL,ERROR}, --verbose-level {DEBUG,INFO,WARNING,FATAL,ERROR}
                                    verbosity level (default: INFO)
              -t TESTDATA_DIR, --testdata-dir TESTDATA_DIR
                                    full path to directory containing test datasets. This
                                    value is set, by precedence, by this option, then by
                                    QSERV_TESTDATA_DIR environment variable if not empty,
                                    and finally by setting testdata_dir value in
                                    ~/.lsst/qserv.conf (default:
                                    /data/fjammes/stack/Linux64/qserv_testdata/2014_06.0)

            Please also note that DM-930 needs to be integrated in these ticket in order to close this review.

            Thanks,

            jammes Fabrice Jammes added a comment - - edited Hi, Thanks for your remarks, i've adressed them in my last commits. Two precisions : About Ubuntu 12.04 : could you please provide me the dependencies list so that I integrate it in the documentation ? Nevertheless, I think that while we're in active development process, we should only support a limited number of distributions. And Ubuntu 14.04 is the new "Long Term Support" version since april. About "setup qserv_testdata", add of --testdata_dir option in qserv-test-integration.py solve this, please see help about this option below. Difference between qserv-test-integration.py and qserv-benchmark.py (renamed to qserv-check-integration.py as a proposal) : The first one relies on unittest python framework and aims at being used in other tools like continuous integration platform (which performs and monitor many tests after each build) or cluster management tool (to check if everything is well installed). It aims at launching automatically a large panel of tests and retrieve their results in unittest module standard way. The second one, whereas it runs the same algorithm, is more development and debugging oriented as it allow to run more fine grained tests (on one dataset, or, why not, on one query). Furthermore it isn't encapsulated in unittest framework. I tried to explains that in these scripts --help options. fjammes@clrlsst-dbmaster-vm:~ /src/qserv (u /fjammes/DM-649 %) $ qserv- test -integration.py --help usage: qserv- test -integration.py [-h] [- v {DEBUG,INFO,WARNING,FATAL,ERROR}] [-t TESTDATA_DIR]   Qserv integration tests suite. Relies on python unit testing framework, provide test meta-data which can be used for example in a continuous integration framework or by a cluster management tool. Configuration values are read from ~/.lsst /qserv .conf.   optional arguments: -h, --help show this help message and exit - v {DEBUG,INFO,WARNING,FATAL,ERROR}, --verbose-level {DEBUG,INFO,WARNING,FATAL,ERROR} verbosity level (default: INFO) -t TESTDATA_DIR, --testdata- dir TESTDATA_DIR full path to directory containing test datasets. This value is set , by precedence, by this option, then by QSERV_TESTDATA_DIR environment variable if not empty, and finally by setting testdata_dir value in ~/.lsst /qserv .conf (default: /data/fjammes/stack/Linux64/qserv_testdata/2014_06 .0) Please also note that DM-930 needs to be integrated in these ticket in order to close this review. Thanks,

            Hi Fabrice,

            thanks for all fixes. One minor thing, in the piece of "code":

            INSTALL_DIR = root/directory/where/qserv/stack/will/be/installed

            it's better to remove spaces around assignment, should help people who copy and paste this (they will modify it anyway but still).

            I still do not understand the reason to keep qserv-test-integration.py and qserv-check-integration.py separate, they do essentially the same job, it seems logical to me that they can be unified.

            Regarding Ubuntu 12.04 - let me try to install VM and figure out exactly what is needed.

            Anyways, I'm satisfied and remove myself from reviewers.

            Cheers,
            Andy

            salnikov Andy Salnikov added a comment - Hi Fabrice, thanks for all fixes. One minor thing, in the piece of "code": INSTALL_DIR = root/directory/where/qserv/stack/will/be/installed it's better to remove spaces around assignment, should help people who copy and paste this (they will modify it anyway but still). I still do not understand the reason to keep qserv-test-integration.py and qserv-check-integration.py separate, they do essentially the same job, it seems logical to me that they can be unified. Regarding Ubuntu 12.04 - let me try to install VM and figure out exactly what is needed. Anyways, I'm satisfied and remove myself from reviewers. Cheers, Andy

            Fabrice, I just remembered one more thing: the "-r <qserv run dir>" option is very handy, it needs to be well documented (and explained that the default is set to ~/qserv-run)

            jbecla Jacek Becla (Inactive) added a comment - Fabrice, I just remembered one more thing: the "-r <qserv run dir>" option is very handy, it needs to be well documented (and explained that the default is set to ~/qserv-run)

            Fabrice, I am pasting comments from Daniel here. I believe this completes the review

            "Welcome to Qserv's documentation!" should just be "Qserv 2014_08.0 documentation <http://lsst-web.ncsa.illinois.edu/%7Efjammes/qserv-doc/u.fjammes.DM-649-gee2bed707c/index.html#>" or just "Qserv documentation"

            "Software architecture" --> "Administration reference"
            ---------------------------
            quickstart page:

            "FOR DEVELOPERS:..." --> "This procedure is for RELEASED and PUBLISHED qserv software. Developers interested in modifying code or running unreleased versions should see Quick start guide for developers "

            Configuration
            The code block should just be:
            qserv-configure.py --all

            The alert:"Qserv commands listed below provides advanced options, use –help option to discover how to use it." is unnecessary. Just put a note below the command saying something like: "For more options, run qserv-configure.py --help". In the future, we should at least list, of not document some of the options. The whole point of this page is to "document".

            In general, the alert blocks ( that have ) should only be used to point out "gotchas" or ways in which the reader may possibly screw up. The block at the top of the page is appropriate--it just says "Hey, watch out" and is not really adding to the essentials of the page.

            Integration tests
            "Integration tests" -> "Testing"

            The "Default value for ..." alert should probably be a comment in the code block below.

            Quick start guide for developers

            You want something to say: "Using Qserv with your own custom code or arbitrary versions can be done by connecting your local git clone with an existing released qserv installation (e.g. from Quick start guide). This should be stated above "Setup current Qserv version in eups".

            Setup current Qserv version in eups:

            text should say: "These commands point the infrastructure of an existing qserv installation to work with a local Qserv git repository."

            below code block:
            Once the qserv eups stack is integrated with your local Qserv repository, you will need to re-configure and (if desired) re-test as described in the Quick start guide.

            I'll have to look at "Software architecture" later-it's a bit messy and not really user documentation-it looks like like a Trac/JIRA ticket or epic.

            Hope this helps,
            -Daniel

            jbecla Jacek Becla (Inactive) added a comment - Fabrice, I am pasting comments from Daniel here. I believe this completes the review "Welcome to Qserv's documentation!" should just be "Qserv 2014_08.0 documentation < http://lsst-web.ncsa.illinois.edu/%7Efjammes/qserv-doc/u.fjammes.DM-649-gee2bed707c/index.html# >" or just "Qserv documentation" "Software architecture" --> "Administration reference" --------------------------- quickstart page: "FOR DEVELOPERS:..." --> "This procedure is for RELEASED and PUBLISHED qserv software. Developers interested in modifying code or running unreleased versions should see Quick start guide for developers " Configuration The code block should just be: qserv-configure.py --all The alert:"Qserv commands listed below provides advanced options, use –help option to discover how to use it." is unnecessary. Just put a note below the command saying something like: "For more options, run qserv-configure.py --help". In the future, we should at least list, of not document some of the options. The whole point of this page is to "document". In general, the alert blocks ( that have ) should only be used to point out "gotchas" or ways in which the reader may possibly screw up. The block at the top of the page is appropriate--it just says "Hey, watch out" and is not really adding to the essentials of the page. Integration tests "Integration tests" -> "Testing" The "Default value for ..." alert should probably be a comment in the code block below. Quick start guide for developers You want something to say: "Using Qserv with your own custom code or arbitrary versions can be done by connecting your local git clone with an existing released qserv installation (e.g. from Quick start guide). This should be stated above "Setup current Qserv version in eups". Setup current Qserv version in eups: text should say: "These commands point the infrastructure of an existing qserv installation to work with a local Qserv git repository." below code block: Once the qserv eups stack is integrated with your local Qserv repository, you will need to re-configure and (if desired) re-test as described in the Quick start guide. I'll have to look at "Software architecture" later- it's a bit messy and not really user documentation -it looks like like a Trac/JIRA ticket or epic. Hope this helps, -Daniel

            Looking at the documentation again it occurs to me that instructions as written now are for installing the most recent qserv version. Should they have explicit release name in the "eups distrib unstall qserv" and "setup qserv"?

            salnikov Andy Salnikov added a comment - Looking at the documentation again it occurs to me that instructions as written now are for installing the most recent qserv version. Should they have explicit release name in the "eups distrib unstall qserv" and "setup qserv"?

            One more comment (sorry I continue commenting on a closed issue, I don't feel this deserves a new ticket yet). I managed to build and run qserv on Ubuntu 12.04 clean server install after I added these packages:

            make
            g++
            libz-dev
            gettext
            flex
            bison
            scons
            libreadline-dev
            libncurses5-dev
            cmake
            python-dev
            python-numpy
            swig
            libbz2-dev
            python-setuptools
            pkg-config
            libglib2.0-dev
            openjdk-7-jdk

            salnikov Andy Salnikov added a comment - One more comment (sorry I continue commenting on a closed issue, I don't feel this deserves a new ticket yet). I managed to build and run qserv on Ubuntu 12.04 clean server install after I added these packages: make g++ libz-dev gettext flex bison scons libreadline-dev libncurses5-dev cmake python-dev python-numpy swig libbz2-dev python-setuptools pkg-config libglib2.0-dev openjdk-7-jdk

            Hi, i've integrated most of your post-review comment directly into master. About Daniel comments, i've only replaced "existing released qserv installation " with "eups software stack containing Qserv dependencies". Indeed, in last developer doc updates, it is no more required to install a Qserv release prior to installing it from user git repository.

            jammes Fabrice Jammes added a comment - Hi, i've integrated most of your post-review comment directly into master. About Daniel comments, i've only replaced "existing released qserv installation " with "eups software stack containing Qserv dependencies". Indeed, in last developer doc updates, it is no more required to install a Qserv release prior to installing it from user git repository.

            People

              jammes Fabrice Jammes
              fritzm Fritz Mueller
              Daniel Wang [X] (Inactive), Jacek Becla (Inactive)
              Andy Salnikov, Daniel Wang [X] (Inactive), Fabrice Jammes, Jacek Becla (Inactive), Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.