Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1338

Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :

      Making your "qserv data" directory independent of where qserv is installed

      I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either:

      • get all their information about various paths from a single file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or...
      • refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS.

      The two are not mutually exclusive – e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them.

      Assuming you did that, your development loop may look something like this:

          # assuming that qserv-configure.py has already been run
          # in ../qserv-run
       
          # do something with qserv-a clone
          cd qserv-a
          setup -r .
          ... do some edits ...
          scons
       
          # now do the tests
          cd ../qserv-run
          ./bin/qserv-start.sh
          ... do tests ...
          ./bin/qserv-stop.sh
       
          # now switch to qserv-b clone
          cd ../qserv-b
          setup -r .
       
          # and do the tests again
          cd ../qserv-run
          ./bin/qserv-start.sh
          ... do tests ...
          ./bin/qserv-stop.sh
       

      that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time.

        Attachments

          Issue Links

            Activity

            jammes Fabrice Jammes created issue -
            jammes Fabrice Jammes made changes -
            Field Original Value New Value
            Epic Link DM-1047 [ 13839 ]
            jammes Fabrice Jammes made changes -
            Description * Making your "qserv data" directory independent of where qserv is
              installed
             
              I think this is a big one, and largely independent of EUPS. You have
              a problem where you want to use one set of test data potentially with
              different qserv binaries (not at the same time, of course). I'd argue
              you should refactor the scripts generated by qserv-configure to
              either:
             
              * get _all_ their information about various paths from a _single_
                file, for example, from etc/paths.cfg.sh. Then you can easily
                regenerate just that file when you need to switch to a different
                qserv (or zookeper, or what not), or...
             
              * refactor the generated scripts to learn from the environment which
                binaries to run. I.e., if $QSERV_DIR is defined, use that qserv,
                etc. This will let you switch binaries by simply setup-ing the new
                one with EUPS.
             
                The two are not mutually exclusive -- e.g., all of this logic
                could be in etc/paths.cfg.sh, and depending on whether this is a
                development build or a "non-EUPS" build, it can either pick up
                the paths from the environment or hardcode them.
             
              Assuming you did that, your development loop may look something like
              this:
             
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
              that is, as qserv-start picks up the relevant products from the
              environment, there's no need to rebuild/reconfigure the qserv-run
              directory each time.
             
            Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :
            {noformat}

            * Making your "qserv data" directory independent of where qserv is
              installed
             
              I think this is a big one, and largely independent of EUPS. You have
              a problem where you want to use one set of test data potentially with
              different qserv binaries (not at the same time, of course). I'd argue
              you should refactor the scripts generated by qserv-configure to
              either:
             
              * get _all_ their information about various paths from a _single_
                file, for example, from etc/paths.cfg.sh. Then you can easily
                regenerate just that file when you need to switch to a different
                qserv (or zookeper, or what not), or...
             
              * refactor the generated scripts to learn from the environment which
                binaries to run. I.e., if $QSERV_DIR is defined, use that qserv,
                etc. This will let you switch binaries by simply setup-ing the new
                one with EUPS.
             
                The two are not mutually exclusive -- e.g., all of this logic
                could be in etc/paths.cfg.sh, and depending on whether this is a
                development build or a "non-EUPS" build, it can either pick up
                the paths from the environment or hardcode them.
             
              Assuming you did that, your development loop may look something like
              this:
             
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
              that is, as qserv-start picks up the relevant products from the
              environment, there's no need to rebuild/reconfigure the qserv-run
              directory each time.
             
            jammes Fabrice Jammes made changes -
            Description Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :
            {noformat}

            * Making your "qserv data" directory independent of where qserv is
              installed
             
              I think this is a big one, and largely independent of EUPS. You have
              a problem where you want to use one set of test data potentially with
              different qserv binaries (not at the same time, of course). I'd argue
              you should refactor the scripts generated by qserv-configure to
              either:
             
              * get _all_ their information about various paths from a _single_
                file, for example, from etc/paths.cfg.sh. Then you can easily
                regenerate just that file when you need to switch to a different
                qserv (or zookeper, or what not), or...
             
              * refactor the generated scripts to learn from the environment which
                binaries to run. I.e., if $QSERV_DIR is defined, use that qserv,
                etc. This will let you switch binaries by simply setup-ing the new
                one with EUPS.
             
                The two are not mutually exclusive -- e.g., all of this logic
                could be in etc/paths.cfg.sh, and depending on whether this is a
                development build or a "non-EUPS" build, it can either pick up
                the paths from the environment or hardcode them.
             
              Assuming you did that, your development loop may look something like
              this:
             
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
              that is, as qserv-start picks up the relevant products from the
              environment, there's no need to rebuild/reconfigure the qserv-run
              directory each time.
             
            Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :
            {noformat}

            * Making your "qserv data" directory independent of where qserv is
              installed
             
              I think this is a big one, and largely independent of EUPS. You have
              a problem where you want to use one set of test data potentially with
              different qserv binaries (not at the same time, of course). I'd argue
              you should refactor the scripts generated by qserv-configure to
              either:
             
              * get _all_ their information about various paths from a _single_
                file, for example, from etc/paths.cfg.sh. Then you can easily
                regenerate just that file when you need to switch to a different
                qserv (or zookeper, or what not), or...
             
              * refactor the generated scripts to learn from the environment which
                binaries to run. I.e., if $QSERV_DIR is defined, use that qserv,
                etc. This will let you switch binaries by simply setup-ing the new
                one with EUPS.
             
                The two are not mutually exclusive -- e.g., all of this logic
                could be in etc/paths.cfg.sh, and depending on whether this is a
                development build or a "non-EUPS" build, it can either pick up
                the paths from the environment or hardcode them.
             
              Assuming you did that, your development loop may look something like
              this:
             
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
              that is, as qserv-start picks up the relevant products from the
              environment, there's no need to rebuild/reconfigure the qserv-run
              directory each time.
             {noformat}
            jammes Fabrice Jammes made changes -
            Watchers Andy Salnikov, Fabrice Jammes, Jacek Becla [ Andy Salnikov, Fabrice Jammes, Jacek Becla ] Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric [ Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric ]
            jammes Fabrice Jammes made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbecla Jacek Becla made changes -
            Team Database [ 10204 ]
            jammes Fabrice Jammes made changes -
            Story Points 2 5
            jammes Fabrice Jammes made changes -
            Summary Make "qserv data" directory independent of where qserv is installed Make QSERV_RUN_DIR able to detect qserv install paths using eups
            jammes Fabrice Jammes made changes -
            Summary Make QSERV_RUN_DIR able to detect qserv install paths using eups Make QSERV_RUN_DIR scripts able to detect qserv install paths using eups
            jammes Fabrice Jammes made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            jammes Fabrice Jammes made changes -
            Link This issue has to be done before DM-1426 [ DM-1426 ]
            jbecla Jacek Becla made changes -
            Description Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :
            {noformat}

            * Making your "qserv data" directory independent of where qserv is
              installed
             
              I think this is a big one, and largely independent of EUPS. You have
              a problem where you want to use one set of test data potentially with
              different qserv binaries (not at the same time, of course). I'd argue
              you should refactor the scripts generated by qserv-configure to
              either:
             
              * get _all_ their information about various paths from a _single_
                file, for example, from etc/paths.cfg.sh. Then you can easily
                regenerate just that file when you need to switch to a different
                qserv (or zookeper, or what not), or...
             
              * refactor the generated scripts to learn from the environment which
                binaries to run. I.e., if $QSERV_DIR is defined, use that qserv,
                etc. This will let you switch binaries by simply setup-ing the new
                one with EUPS.
             
                The two are not mutually exclusive -- e.g., all of this logic
                could be in etc/paths.cfg.sh, and depending on whether this is a
                development build or a "non-EUPS" build, it can either pick up
                the paths from the environment or hardcode them.
             
              Assuming you did that, your development loop may look something like
              this:
             
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
              that is, as qserv-start picks up the relevant products from the
              environment, there's no need to rebuild/reconfigure the qserv-run
              directory each time.
             {noformat}
            Ticket related to Mario email (subject: [QSERV-L] Some points/actions from the discussion today) :

            Making your "qserv data" directory independent of where qserv is installed
             
            I think this is a big one, and largely independent of EUPS. You have a problem where you want to use one set of test data potentially with different qserv binaries (not at the same time, of course). I'd argue you should refactor the scripts generated by qserv-configure to either:
             
            * get _all_ their information about various paths from a _single_ file, for example, from etc/paths.cfg.sh. Then you can easily regenerate just that file when you need to switch to a different qserv (or zookeper, or what not), or...
            * refactor the generated scripts to learn from the environment which binaries to run. I.e., if $QSERV_DIR is defined, use that qserv, etc. This will let you switch binaries by simply setup-ing the new one with EUPS.
             
            The two are not mutually exclusive -- e.g., all of this logic could be in etc/paths.cfg.sh, and depending on whether this is a development build or a "non-EUPS" build, it can either pick up the paths from the environment or hardcode them.
             
            Assuming you did that, your development loop may look something like this:
             
            {code}
                # assuming that qserv-configure.py has already been run
                # in ../qserv-run
             
                # do something with qserv-a clone
                cd qserv-a
                setup -r .
                ... do some edits ...
                scons
             
                # now do the tests
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             
                # now switch to qserv-b clone
                cd ../qserv-b
                setup -r .
             
                # and do the tests again
                cd ../qserv-run
                ./bin/qserv-start.sh
                ... do tests ...
                ./bin/qserv-stop.sh
             {code}

            that is, as qserv-start picks up the relevant products from the environment, there's no need to rebuild/reconfigure the qserv-rundirectory each time.
            jbecla Jacek Becla made changes -
            Sprint DB_S14_10 [ 92 ] DB_S14_10, DB_S14_11 [ 92, 93 ]
            jbecla Jacek Becla made changes -
            Rank Ranked higher
            jbecla Jacek Becla made changes -
            Rank Ranked lower
            jammes Fabrice Jammes made changes -
            Link This issue relates to DM-212 [ DM-212 ]
            jammes Fabrice Jammes made changes -
            Link This issue relates to DM-1470 [ DM-1470 ]
            jammes Fabrice Jammes made changes -
            Story Points 5 7
            salnikov Andy Salnikov made changes -
            Status In Review [ 10004 ] Review Complete [ 10101 ]
            jammes Fabrice Jammes made changes -
            Resolution Done [ 10000 ]
            Status Review Complete [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              jammes Fabrice Jammes
              Reporter:
              jammes Fabrice Jammes
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Fabrice Jammes, Jacek Becla, Mario Juric
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.