Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1338

Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :

      Making your "qserv data" directory independent of where qserv is installed

      I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either:

      • get all their information about various paths from a single file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or...
      • refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS.

      The two are not mutually exclusive – e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them.

      Assuming you did that, your development loop may look something like this:

          # assuming that qserv-configure.py has already been run
          # in ../qserv-run
       
          # do something with qserv-a clone
          cd qserv-a
          setup -r .
          ... do some edits ...
          scons
       
          # now do the tests
          cd ../qserv-run
          ./bin/qserv-start.sh
          ... do tests ...
          ./bin/qserv-stop.sh
       
          # now switch to qserv-b clone
          cd ../qserv-b
          setup -r .
       
          # and do the tests again
          cd ../qserv-run
          ./bin/qserv-start.sh
          ... do tests ...
          ./bin/qserv-stop.sh
       

      that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time.

        Attachments

          Issue Links

            Activity

            Hide
            jammes Fabrice Jammes added a comment -

            At first glance, I think that making init.d scripts using eups environment isn't enough to complete this ticket.
            Indeed, eups products path are also used in xrootd, mysql-proxy, and client config file :

            fjammes@clrlsst-dbmaster-vm:~/src/qserv/admin/templates/server/etc (u/fjammes/DM-1338) $ grep QSERV_DIR *
            lsp.cf:ssi.svclib {{QSERV_DIR}}/lib/libxrdsvc.so
            lsp.cf:ofs.osslib {{QSERV_DIR}}/lib/libxrdoss.so
            my-proxy.cnf:proxy-lua-script = {{QSERV_DIR}}/proxy/mysqlProxy.lua
            qserv-client.conf:base_dir = {{QSERV_DIR}}

            Show
            jammes Fabrice Jammes added a comment - At first glance, I think that making init.d scripts using eups environment isn't enough to complete this ticket. Indeed, eups products path are also used in xrootd, mysql-proxy, and client config file : fjammes@clrlsst-dbmaster-vm:~ /src/qserv/admin/templates/server/etc (u /fjammes/DM-1338 ) $ grep QSERV_DIR * lsp.cf:ssi.svclib {{QSERV_DIR}} /lib/libxrdsvc .so lsp.cf:ofs.osslib {{QSERV_DIR}} /lib/libxrdoss .so my-proxy.cnf:proxy-lua-script = {{QSERV_DIR}} /proxy/mysqlProxy .lua qserv-client.conf:base_dir = {{QSERV_DIR}}
            Hide
            jammes Fabrice Jammes added a comment -

            xrootd, mysql-proxy and qserv-czar can now use eups environment, but integration test fails. I may use git bisect to find the issue.

            Show
            jammes Fabrice Jammes added a comment - xrootd, mysql-proxy and qserv-czar can now use eups environment, but integration test fails. I may use git bisect to find the issue.
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov,

            In init.d scripts:

            • paths management can be held by eups,
            • other services parameters, which are nor eups-managed or in services configuration files, are now sources from etc/sysconfig/service-name.

            Thanks in advance for your help,

            Fabrice

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , In init.d scripts: paths management can be held by eups, other services parameters, which are nor eups-managed or in services configuration files, are now sources from etc/sysconfig/service-name. Thanks in advance for your help, Fabrice
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov,

            Please note that this feature may allow to switch Qserv version using for example setup qserv -t qserv, and then restart Qserv with the same QSERV_RUN_DIR and different Qserv binaries. This should work also for Qserv dependencies (xrootd, mysql-proxy, mysql, scisql).

            But I think there's a few conditions for this to work (QSERV_RUN_DIR content related to a eups product has to remain compliant when product version change).
            That's why I propose to test it for a few time, and then document it, when we'll have more user feedback.

            Thanks,

            Fabrice

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , Please note that this feature may allow to switch Qserv version using for example setup qserv -t qserv , and then restart Qserv with the same QSERV_RUN_DIR and different Qserv binaries. This should work also for Qserv dependencies (xrootd, mysql-proxy, mysql, scisql). But I think there's a few conditions for this to work (QSERV_RUN_DIR content related to a eups product has to remain compliant when product version change). That's why I propose to test it for a few time, and then document it, when we'll have more user feedback. Thanks, Fabrice
            Hide
            salnikov Andy Salnikov added a comment -

            I think we need to discuss this more before we could implement something that works. I can see several issues that are not addressed in a proposed change:

            • in general init.d scripts are not supposed to use the environment (this is why recommended way to run things is via service some-service start)
            • it looks like we want to satisfy requirement from both development side and production in a single solution, I'm not sure it's the best approach
            • scripts in run directory may not be even compatible with the qserv release, there is no way to test for that now.

            Maybe we can talk about this again at Wednesday meeting.

            Show
            salnikov Andy Salnikov added a comment - I think we need to discuss this more before we could implement something that works. I can see several issues that are not addressed in a proposed change: in general init.d scripts are not supposed to use the environment (this is why recommended way to run things is via service some-service start ) it looks like we want to satisfy requirement from both development side and production in a single solution, I'm not sure it's the best approach scripts in run directory may not be even compatible with the qserv release, there is no way to test for that now. Maybe we can talk about this again at Wednesday meeting.
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy,

            You're point of view is understandable and very reasonable, but I can't see how to to make point 1 and 2 compliant with previous Mario proposal (please refer to ticket description).

            Point 3 is related to DM-895, and this un-covered feature isn't adressed by current ticket, but it would be good to solve this point soon.

            As a reminder, the current Qserv configuration tool prototype was quickly re-written in order to enable developers to run integration tests with eups-installed Qserv. So I think it would be good, now that we have some user feedback, to define a better design, compliant with our use-cases.
            Furthermore, Daniel Wang [X] has also some good ideas to improve Qserv startup scripts. So I think that your proposal to discuss this at our Wednesday meeting is very interesting.

            Andy, maybe we could meet together before so that we can already propose some interesting proposals at the Wednesday meeting? Tomorrow around 10 a.m. or 2 p.m. would be fine for me.

            Cheers,

            Fabrice

            Show
            jammes Fabrice Jammes added a comment - Hi Andy, You're point of view is understandable and very reasonable, but I can't see how to to make point 1 and 2 compliant with previous Mario proposal (please refer to ticket description). Point 3 is related to DM-895 , and this un-covered feature isn't adressed by current ticket, but it would be good to solve this point soon. As a reminder, the current Qserv configuration tool prototype was quickly re-written in order to enable developers to run integration tests with eups-installed Qserv. So I think it would be good, now that we have some user feedback, to define a better design, compliant with our use-cases. Furthermore, Daniel Wang [X] has also some good ideas to improve Qserv startup scripts. So I think that your proposal to discuss this at our Wednesday meeting is very interesting. Andy, maybe we could meet together before so that we can already propose some interesting proposals at the Wednesday meeting? Tomorrow around 10 a.m. or 2 p.m. would be fine for me. Cheers, Fabrice
            Hide
            salnikov Andy Salnikov added a comment -

            Hi Fabrice,

            I'm not sure yet I'm going to be at SLAC tomorrow, but if I will I'll let you know. Discussing that with everybody interested at Wed meeting may be even better.

            Andy.

            Show
            salnikov Andy Salnikov added a comment - Hi Fabrice, I'm not sure yet I'm going to be at SLAC tomorrow, but if I will I'll let you know. Discussing that with everybody interested at Wed meeting may be even better. Andy.
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov,

            I've removed the eups environment feature, but kept sysconfig and path.cfg.sh.
            What do you think of it please?

            Cheers,

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , I've removed the eups environment feature, but kept sysconfig and path.cfg.sh. What do you think of it please? Cheers,
            Hide
            salnikov Andy Salnikov added a comment -

            Hi Fabrice, I made few comments in a pull request: https://github.com/LSST/qserv/pull/7
            My biggest issue now is that having multiple files in sysconfig directory does not seem to simplify things a lot. I think the idea that Mario suggested was to have all variables defined in one file (paths.cfg.sh) which is sourced by the init.scripts. If someone wants to change some paths then it's easy to find it in that file instead of searching many files for location to change. Can I propose to do this instead:

            • merge all variable definitions from all files in /etc/sysconfig into one single file, it does not need to be in sysconfig, having it in /etc is OK, maybe name it differently (qserv-paths.cfg.sh?)
            • that file does not need any conditional stuff or any verification that paths exist, it only defines all variables (again, the purpose is to have all variables in one place so that it is easy to change).
            • all service-specific stuff and path validation should be in init.d scripts, those ideally should not have any hardcoded paths, only use whatever is defined in qserv-paths.cfg.sh

            There is also one issue with lua version number hardcoded into scripts (and it was hardcoded in SConscript files before), we should find a way to determine that when we run qserv-configure (this could go into separate ticket).

            Show
            salnikov Andy Salnikov added a comment - Hi Fabrice, I made few comments in a pull request: https://github.com/LSST/qserv/pull/7 My biggest issue now is that having multiple files in sysconfig directory does not seem to simplify things a lot. I think the idea that Mario suggested was to have all variables defined in one file (paths.cfg.sh) which is sourced by the init.scripts. If someone wants to change some paths then it's easy to find it in that file instead of searching many files for location to change. Can I propose to do this instead: merge all variable definitions from all files in /etc/sysconfig into one single file, it does not need to be in sysconfig, having it in /etc is OK, maybe name it differently ( qserv-paths.cfg.sh ?) that file does not need any conditional stuff or any verification that paths exist, it only defines all variables (again, the purpose is to have all variables in one place so that it is easy to change). all service-specific stuff and path validation should be in init.d scripts, those ideally should not have any hardcoded paths, only use whatever is defined in qserv-paths.cfg.sh There is also one issue with lua version number hardcoded into scripts (and it was hardcoded in SConscript files before), we should find a way to determine that when we run qserv-configure (this could go into separate ticket).
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Andy Salnikov,

            I tried to take in account all your remarks. Could you please review it again?

            Cheers,

            Show
            jammes Fabrice Jammes added a comment - Hi Andy Salnikov , I tried to take in account all your remarks. Could you please review it again? Cheers,
            Hide
            salnikov Andy Salnikov added a comment -

            Hi Fabrice,

            I made few comments in the pull request. Basically I do not like what qserv-check-config does:

            • having all checks in one place does not make much sense, they should be done where they are needed
            • you are probably doing more checks than necessary, you try to avoid the cost of mis-configuration mistake (which I think is rather small) but you introduce potential code management cost instead. I believe some of those checks can be done by the services themselves, not init scripts
            Show
            salnikov Andy Salnikov added a comment - Hi Fabrice, I made few comments in the pull request. Basically I do not like what qserv-check-config does: having all checks in one place does not make much sense, they should be done where they are needed you are probably doing more checks than necessary, you try to avoid the cost of mis-configuration mistake (which I think is rather small) but you introduce potential code management cost instead. I believe some of those checks can be done by the services themselves, not init scripts
            Hide
            jammes Fabrice Jammes added a comment - - edited

            Hi Andy Salnikov,

            I tried to stick as much as possible to your recommendations. Nevertheless I kept 2 controls related to mysql-proxy, which could be removed in DM-1470.
            Moreover, I don't like to much having such duplicated checks in each init.d scripts. Would it be possible to factorize the common ones?
            If yes, would two functions be fine: one for checking qserv_run_dir, and one for both qserv_log_dir and qserv_pid_dir?

            Thanks,

            Fabrice

            Show
            jammes Fabrice Jammes added a comment - - edited Hi Andy Salnikov , I tried to stick as much as possible to your recommendations. Nevertheless I kept 2 controls related to mysql-proxy, which could be removed in DM-1470 . Moreover, I don't like to much having such duplicated checks in each init.d scripts. Would it be possible to factorize the common ones? If yes, would two functions be fine: one for checking qserv_run_dir, and one for both qserv_log_dir and qserv_pid_dir? Thanks, Fabrice
            Hide
            salnikov Andy Salnikov added a comment -

            Fabrice, I'm afraid that we are spending too much time on this simple issue, so please go ahead, implement it in a way that you think is reasonable.

            Show
            salnikov Andy Salnikov added a comment - Fabrice, I'm afraid that we are spending too much time on this simple issue, so please go ahead, implement it in a way that you think is reasonable.
            Hide
            salnikov Andy Salnikov added a comment -

            I think it should be OK one way or another.

            Show
            salnikov Andy Salnikov added a comment - I think it should be OK one way or another.
            Hide
            jammes Fabrice Jammes added a comment -

            Thanks Andy for your help in improving these scripts.

            Show
            jammes Fabrice Jammes added a comment - Thanks Andy for your help in improving these scripts.

              People

              Assignee:
              jammes Fabrice Jammes
              Reporter:
              jammes Fabrice Jammes
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.