# Improve install/configuration/tests documentation and migrate it to reST format

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.2
• Epic Link:
• Sprint:
DB_S14_06, DB_S14_07, DB_S14_08
• Team:
Data Access and Database

#### Description

This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/

Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation.

README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt
The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"?
README-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .'

#### Activity

No builds found.
Fabrice Jammes created issue -
Fabrice Jammes made changes -
Field Original Value New Value
Sprint DB_S14_06 [ 35 ]
Fabrice Jammes made changes -
 Summary Improve documentation and migrate it to RST format Improve documentation and migrate it to reST format
Fabrice Jammes made changes -
 Description This ticket propose to migrate README and README-devel to RST format. The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"? REDME-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote} This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"? REDME-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote}
Hide
Fabrice Jammes added a comment -

Hi Andy,

Some of DM-622 comments where more documentation-oriented than configuration-oriented, that's I propose to adress them here. In addition I've migrated the documentation to reST format, html output is available here :
http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/

Have a nice day,

Fabrice

Show
Fabrice Jammes added a comment - Hi Andy, Some of DM-622 comments where more documentation-oriented than configuration-oriented, that's I propose to adress them here. In addition I've migrated the documentation to reST format, html output is available here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Have a nice day, Fabrice
Fabrice Jammes made changes -
 Reviewers Andy Salnikov [ salnikov ] Status To Do [ 10001 ] In Review [ 10004 ]
Hide
Fabrice Jammes added a comment -

Andy,

Please note that configuration procedure documented in README.txt, refers to the DM-622 one, so it's not consistent with DM-930 code which is issued from current master tip (for DM-930 please use previous configuration procedure).

I don't think it's worth changin it as DM-622 merge will soon solve this minor issue.

Cheers

Show
Fabrice Jammes added a comment - Andy, Please note that configuration procedure documented in README.txt, refers to the DM-622 one, so it's not consistent with DM-930 code which is issued from current master tip (for DM-930 please use previous configuration procedure). I don't think it's worth changin it as DM-622 merge will soon solve this minor issue. Cheers
Hide
Andy Salnikov added a comment -

Hi Fabrice,

here are few things that I think worth mentioning:

• If we migrate docs to reStructuredtext (or some other markdown-like format) should this apply to all text files, is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration? I guess main utility of that would be publishing on web site or in github-like setup. Do we have an idea what do we want to do in that respect?
• My biggest issue is with dependencies. I do not think it's a good idea to include the list of dependencies into README-devel.rst, we'll end up with two lists that need to be maintained (one in ups/qserv.table and another in README-devel.rst). There should be only one list so there is no risk of inconsistent package names or version numbers. Until eups provides us a way to install package dependencies only (if that ever happens) can we do something simple just for ourselves like a small script which parses qserv.table and uses eups to install dependencies?
• I am a bit confused by two distribution servers in README-devel when installing dependencies. When you install qserv with "eups distrib install qserv -r http://lsst-web.ncsa.illinois.edu/~fjammes/qserv" do dependencies also come from two servers, how is that handled? Does your private server redirects to http://sw.lsstcorp.org/eupspkg for some packages or is it handled by eups itself?
• admin/tools/qserv-install.sh script still uses scons in admin directory instead of running qserv-confgiure.py, should it be changed as well?
• I was burned once by eupspkg doing unpredictable things (I try hard to avoid using it so there may be more interesting things that I miss about it), it may be worth adding small note to README-devel cautioning developers (below "setup qserv $VERSION"), something along the lines: Be advised that eupspkg may generate different version numbers depending on whether the code has changed after checkout. For example it may generate version which looks like "master-g86a30ec72a" for freshly checked-out code but if you change anything in your repository it will generate new version "master-g86a30ec72a-dirty". You may end up with two versions of qserv installed, be very careful and remember to run "setup qserv" with the correct version number. Show Andy Salnikov added a comment - Hi Fabrice, here are few things that I think worth mentioning: If we migrate docs to reStructuredtext (or some other markdown-like format) should this apply to all text files, is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration? I guess main utility of that would be publishing on web site or in github-like setup. Do we have an idea what do we want to do in that respect? My biggest issue is with dependencies. I do not think it's a good idea to include the list of dependencies into README-devel.rst, we'll end up with two lists that need to be maintained (one in ups/qserv.table and another in README-devel.rst). There should be only one list so there is no risk of inconsistent package names or version numbers. Until eups provides us a way to install package dependencies only (if that ever happens) can we do something simple just for ourselves like a small script which parses qserv.table and uses eups to install dependencies? I am a bit confused by two distribution servers in README-devel when installing dependencies. When you install qserv with " eups distrib install qserv -r http://lsst-web.ncsa.illinois.edu/~fjammes/qserv " do dependencies also come from two servers, how is that handled? Does your private server redirects to http://sw.lsstcorp.org/eupspkg for some packages or is it handled by eups itself? admin/tools/qserv-install.sh script still uses scons in admin directory instead of running qserv-confgiure.py , should it be changed as well? I was burned once by eupspkg doing unpredictable things (I try hard to avoid using it so there may be more interesting things that I miss about it), it may be worth adding small note to README-devel cautioning developers (below "setup qserv$VERSION"), something along the lines: Be advised that eupspkg may generate different version numbers depending on whether the code has changed after checkout. For example it may generate version which looks like "master-g86a30ec72a" for freshly checked-out code but if you change anything in your repository it will generate new version "master-g86a30ec72a-dirty". You may end up with two versions of qserv installed, be very careful and remember to run "setup qserv" with the correct version number.
Jacek Becla made changes -
 Sprint DB_S14_06 [ 35 ] DB_S14_06, DB_S14_07 [ 35, 36 ]
Jacek Becla made changes -
 Rank Ranked higher
Hide
Fabrice Jammes added a comment - - edited
• If we migrate docs to reStructuredtext (or some other markdown-like format) should this apply to all text files, is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration? I guess main utility of that would be publishing on web site or in github-like setup. Do we have an idea what do we want to do in that respect?

Not for the moment sorry, this will be adressed in DM-649.

• My biggest issue is with dependencies. I do not think it's a good idea to include the list of dependencies into README-devel.rst, we'll end up with two lists that need to be maintained (one in ups/qserv.table and another in README-devel.rst). There should be only one list so there is no risk of inconsistent package names or version numbers. Until eups provides us a way to install package dependencies only (if that ever happens) can we do something simple just for ourselves like a small script which parses qserv.table and uses eups to install dependencies?

Both of this solution are weak : indeed they don't contain information about version of each dependency. So the last version of each dependency will be installed, and this may be inconsistent with Qserv version of user git repos,and prevent Qserv build. I think the ideal solution would be to have an eups command like that :

 eups distrib install qserv $VERSION --deps-only As discussed with Daniel on the 07/02/2014, the Qserv dependencies don't move a lot, so maybe we can keep this imperfect solution which will satisfy 90% of our needs, and hope a a eups future feature which will cover this ? This is handled by eups itself, I suppose that when a dependency package (example boost) isn't found in http://lsst-web.ncsa.illinois.edu/~fjammes/qserv, eups falls back on$EUPS_PKGROOT (i.e. http://sw.lsstcorp.org/eupspkg).

• admin/tools/qserv-install.sh script still uses scons in admin directory instead of running qserv-confgiure.py, should it be changed as well?

Done in DM-622, current ticket was created from master branch, as recommended, and only adress READMEs files.

• I was burned once by eupspkg doing unpredictable things (I try hard to avoid using it so there may be more interesting things that I miss about it), it may be worth adding small note to README-devel cautioning developers (below "setup qserv $VERSION"), something along the lines:... I think that this should be done in eups documentation, and not in Qserv one, in order to avoid redundancy and ease maintenance. When eups doc will be up to date, we could provide a pointer in Qserv one. Cheers, Show Fabrice Jammes added a comment - - edited If we migrate docs to reStructuredtext (or some other markdown-like format) should this apply to all text files, is it worth to include KNOWN-ISSUES.txt and RELEASE-NOTES.txt into this migration? I guess main utility of that would be publishing on web site or in github-like setup. Do we have an idea what do we want to do in that respect? Not for the moment sorry, this will be adressed in DM-649 . My biggest issue is with dependencies. I do not think it's a good idea to include the list of dependencies into README-devel.rst, we'll end up with two lists that need to be maintained (one in ups/qserv.table and another in README-devel.rst). There should be only one list so there is no risk of inconsistent package names or version numbers. Until eups provides us a way to install package dependencies only (if that ever happens) can we do something simple just for ourselves like a small script which parses qserv.table and uses eups to install dependencies? Both of this solution are weak : indeed they don't contain information about version of each dependency. So the last version of each dependency will be installed, and this may be inconsistent with Qserv version of user git repos,and prevent Qserv build. I think the ideal solution would be to have an eups command like that : eups distrib install qserv$VERSION --deps-only As discussed with Daniel on the 07/02/2014, the Qserv dependencies don't move a lot, so maybe we can keep this imperfect solution which will satisfy 90% of our needs, and hope a a eups future feature which will cover this ? I am a bit confused by two distribution servers in README-devel when installing dependencies. When you install qserv with "eups distrib install qserv -r http://lsst-web.ncsa.illinois.edu/~fjammes/qserv " do dependencies also come from two servers, how is that handled? Does your private server redirects to http://sw.lsstcorp.org/eupspkg for some packages or is it handled by eups itself? This is handled by eups itself, I suppose that when a dependency package (example boost) isn't found in http://lsst-web.ncsa.illinois.edu/~fjammes/qserv , eups falls back on $EUPS_PKGROOT (i.e. http://sw.lsstcorp.org/eupspkg ). admin/tools/qserv-install.sh script still uses scons in admin directory instead of running qserv-confgiure.py, should it be changed as well? Done in DM-622 , current ticket was created from master branch, as recommended, and only adress READMEs files. I was burned once by eupspkg doing unpredictable things (I try hard to avoid using it so there may be more interesting things that I miss about it), it may be worth adding small note to README-devel cautioning developers (below "setup qserv$VERSION"), something along the lines:... I think that this should be done in eups documentation, and not in Qserv one, in order to avoid redundancy and ease maintenance. When eups doc will be up to date, we could provide a pointer in Qserv one. Cheers,
Fabrice Jammes made changes -
 Epic Link DM-51 [ 11299 ]
Fabrice Jammes made changes -
 Link This issue is blocked by DM-622 [ DM-622 ]
Fabrice Jammes made changes -
 Summary Improve documentation and migrate it to reST format Improve install/configuration/tests documentation and migrate it to reST format
Hide
Fabrice Jammes added a comment -

Discussed during Dev Hangout between A. Salnikov and F. Jammes on Tue July 8, 8:30am-10:30am CEST :

• write a qserv.table parser to generate eups distrib install commands
• Point "I am a bit confused by two distribution servers in README-devel when installing dependencies. When you install qserv with "eups distrib install qserv -r http://lsst-web.ncsa.illinois.edu/~fjammes/qserv" do dependencies also come from two servers, how is that handled? Does your private server redirects to http://sw.lsstcorp.org/eupspkg for some packages or is it handled by eups itself" is solved by previous feature.
• update the documentation (README-devel.rst) with :
"Be advised that eupspkg may generate different version numbers depending on whether the code has changed after checkout. For example it may generate version which looks like "master-g86a30ec72a" for freshly checked-out code but if you change anything in your repository it will generate new version "master-g86a30ec72a-dirty". You may end up with two versions of qserv installed, be very careful and remember to run "setup qserv" with the correct version number."
• "-a" option : add it in the README file
Show
Fabrice Jammes added a comment - Discussed during Dev Hangout between A. Salnikov and F. Jammes on Tue July 8, 8:30am-10:30am CEST : write a qserv.table parser to generate eups distrib install commands Point "I am a bit confused by two distribution servers in README-devel when installing dependencies. When you install qserv with "eups distrib install qserv -r http://lsst-web.ncsa.illinois.edu/~fjammes/qserv " do dependencies also come from two servers, how is that handled? Does your private server redirects to http://sw.lsstcorp.org/eupspkg for some packages or is it handled by eups itself" is solved by previous feature. update the documentation (README-devel.rst) with : "Be advised that eupspkg may generate different version numbers depending on whether the code has changed after checkout. For example it may generate version which looks like "master-g86a30ec72a" for freshly checked-out code but if you change anything in your repository it will generate new version "master-g86a30ec72a-dirty". You may end up with two versions of qserv installed, be very careful and remember to run "setup qserv" with the correct version number." "-a" option : add it in the README file
Hide
Kian-Tat Lim added a comment -

Please don't write an eups table parser.

1) Ask the eups experts if there is a way to do what you want (install the dependencies of a package you have git-cloned the source for).
2) Ask the eups experts to add this feature if it is not present.
3) Use the code already within eups to parse eups table files and generate install commands if they can't add this feature and don't want to tell you how to add it.

Show
Kian-Tat Lim added a comment - Please don't write an eups table parser. 1) Ask the eups experts if there is a way to do what you want (install the dependencies of a package you have git-cloned the source for). 2) Ask the eups experts to add this feature if it is not present. 3) Use the code already within eups to parse eups table files and generate install commands if they can't add this feature and don't want to tell you how to add it.
Hide
Fabrice Jammes added a comment - - edited

Hello K.T.,

After a few investigations, i realized this feature wasn't present in eups. That's why i implement a first proposal and send a pull request to Robert (first tests are successfull) :
https://github.com/RobertLuptonTheGood/eups/pull/19

All others review request are done, and the ticket is ready to merge.
Andy could you please complete the review ?

Thanks and have a nice day,

Fabrice

P.S. : please note that "eups distrib install --onlydepend" won't be available until it is packaged in eups stable release. For now, you have to install eups from master tip to test it.

Show
Fabrice Jammes added a comment - - edited Hello K.T., After a few investigations, i realized this feature wasn't present in eups. That's why i implement a first proposal and send a pull request to Robert (first tests are successfull) : https://github.com/RobertLuptonTheGood/eups/pull/19 All others review request are done, and the ticket is ready to merge. Andy could you please complete the review ? Thanks and have a nice day, Fabrice P.S. : please note that "eups distrib install --onlydepend" won't be available until it is packaged in eups stable release. For now, you have to install eups from master tip to test it.
Fabrice Jammes made changes -
 Description This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"? REDME-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote} This ticket propose to migrate README and README-devel to reST format (see http://sphinx-doc.org/rest.html). The output is located here : http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/ Furthermore this ticket wil integrate Andy S. DM-622 value-added remarks about Qserv embedded documentation. {quote} README.txt needs a bit of formatting, whole "NOTE FOR DEVELOPERS" is one long line which may need scrolling depending on what do you use to read the file, same applies to README-devel.txt The install procedure in README.txt implies that the whole stack has to be installed including eups. If people have some part of it installed already the it would probably be better to reuse existing stack. Shall we spit install instructions into "Install eups (if not installed already)" and "Install qserv"? README-devel.txt says "Once Qserv is installed...", I don't think that we need or want to install whole qserv before we start development (what if qserv is not available yet for the platform I'm trying to test). What probably needed is installed dependencies, and this should be covered by the comments before 'setup -r .' {quote}
Fabrice Jammes made changes -
 Story Points 1 1.2
Jacek Becla made changes -
 Team Database [ 10204 ]
Jacek Becla made changes -
 Sprint DB_S14_06, DB_S14_07 [ 35, 36 ] DB_S14_06, DB_S14_07, DB_S14_08 [ 35, 36, 37 ]
Jacek Becla made changes -
 Rank Ranked higher
Hide
Fabrice Jammes added a comment - - edited

K.T. learn us that newinstall.sh now use eups v1.5.0 which supports "eups distrib install --onlydepend" option.

Andy, could you please mark this ticket as done. Furthermore, this would allows to integrate your remarks in master branch, and then rebasing DM-649 on master tip would solve your DM-649 remark :

My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies.

Thanks

Show
Fabrice Jammes added a comment - - edited K.T. learn us that newinstall.sh now use eups v1.5.0 which supports "eups distrib install --onlydepend" option. Andy, could you please mark this ticket as done. Furthermore, this would allows to integrate your remarks in master branch, and then rebasing DM-649 on master tip would solve your DM-649 remark : My old comment still applies - developer setup should not need qserv installed already, I guess this is just waiting for new eups version which will allow us to install dependencies. Thanks
Fabrice Jammes made changes -
 Link This issue blocks DM-649 [ DM-649 ]
Andy Salnikov made changes -
 Status In Review [ 10004 ] Review Complete [ 10101 ]
Fabrice Jammes made changes -
 Resolution Done [ 10000 ] Status Review Complete [ 10101 ] Done [ 10002 ]

#### People

Assignee:
Fabrice Jammes
Reporter:
Fabrice Jammes
Reviewers:
Andy Salnikov
Watchers:
Andy Salnikov, Daniel Wang [X] (Inactive), Fabrice Jammes, Jacek Becla, Kian-Tat Lim
Votes:
0 Vote for this issue
Watchers:
5 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### CI Builds

No builds found.