Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-868

Audit ups table files to replace envAppend with envPrepend

    Details

    • Team:
      SQuaRE

      Description

      Tim Jenness reports problems running v8 on OSX Mavericks due to the build being made against an LSST-provided cfitsio, but attempting to run against a system-installed cfitiso. This appears to be due to some LSST packages being appended to his DYLD_LIBRARY_PATH instead of prepended:

      $ echo $DYLD_LIBRARY_PATH 
      /Users/timj/work/lsst/DarwinX86/shapelet/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/meas_extensions_multiShapelet/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/skymap/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/coadd_chisquared/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/coadd_utils/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/ip_diffim/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/ip_isr/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/astrometry_net/0.30+3/lib:/Users/timj/work/lsst/DarwinX86/meas_astrom/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/meas_algorithms/8.0.0.0+3/lib:/Users/timj/work/lsst/DarwinX86/minuit2/5.22.00+2/lib:/Users/timj/work/lsst/DarwinX86/wcslib/4.14+3/lib:/Users/timj/work/lsst/DarwinX86/fftw/3.3.2+2/lib:/Users/timj/work/lsst/DarwinX86/pex_config/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/afw/8.0.0.3/lib:/Users/timj/work/lsst/DarwinX86/base/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/pex_policy/8.0.0.0+1/lib:/star/lib:/star/starjava/lib/i386:/star/starjava/lib/x86_64:/Users/timj/work/lsst/DarwinX86/daf_base/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/utils/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/boost/1.55.0.1/lib:/Users/timj/work/lsst/DarwinX86/pex_exceptions/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/pex_logging/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/daf_persistence/8.0.0.0+1/lib:/Users/timj/work/lsst/DarwinX86/mysqlclient/5.1.65+3/lib:/Users/timj/work/lsst/DarwinX86/cfitsio/3310+2/lib:/Users/timj/work/lsst/DarwinX86/xpa/2.1.14+2/lib:/Users/timj/work/lsst/DarwinX86/gsl/1.15+2/lib:/Users/timj/work/lsst/DarwinX86/meas_deblender/8.0.0.0+3/lib

      This suggests we need a thorough audit of ups table files, removing envAppend in favour of envPrepend.

        Attachments

          Issue Links

            Activity

            Hide
            krughoff Simon Krughoff added a comment - - edited

            A quick grep through the build directory of ~lsstsw shows the following packages using envAppend:
            ap
            apr
            apr_util
            boost
            cat
            cfitsio
            ctrl_events
            ctrl_orca
            daf_base
            daf_butlerUtils
            daf_persistence
            datarel
            db
            gsl
            lsst_libs
            meas_base
            meas_deblender
            ndarray
            obs_sdss
            obs_test
            pex_config
            pex_exceptions
            pex_harness
            pex_logging
            pex_policy
            pyephem
            pyfits
            qserv_testdata
            testing_displayQA
            testing_endtoend
            utils
            xpa

            Updated 09/Oct/14 RAA

            Show
            krughoff Simon Krughoff added a comment - - edited A quick grep through the build directory of ~lsstsw shows the following packages using envAppend: ap apr apr_util boost cat cfitsio ctrl_events ctrl_orca daf_base daf_butlerUtils daf_persistence datarel db gsl lsst_libs meas_base meas_deblender ndarray obs_sdss obs_test pex_config pex_exceptions pex_harness pex_logging pex_policy pyephem pyfits qserv_testdata testing_displayQA testing_endtoend utils xpa Updated 09/Oct/14 RAA
            Hide
            price Paul Price added a comment -

            This would be good to do before the next (uprev) release.

            Show
            price Paul Price added a comment - This would be good to do before the next (uprev) release.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            I'm building a full private stack on which to test the switch from 'append' to 'prepend' in a bootstrap order. So the DM-868 branches will slowly get added to the relevant packages during this testing phase.

            Why a separate stack....I tried to 'eups list' the production stack this morning and waited what seemed like hours for the response. This work will be faster in a separate stack.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - I'm building a full private stack on which to test the switch from 'append' to 'prepend' in a bootstrap order. So the DM-868 branches will slowly get added to the relevant packages during this testing phase. Why a separate stack....I tried to 'eups list' the production stack this morning and waited what seemed like hours for the response. This work will be faster in a separate stack.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            The testing of the u/rallsman/DM-868 based branches has completed successfully.

            It was necessary to include /u/pgee/DM-1439 in the branches supplied to my stack in order to get past the afw tests filename conflict. afw itself was not involved in the DM-868 changes but was rebuilt due to them.

            The stack in question is available for a limited time only (until Review is completed):
            export LSSTSW=/lsst/home/rallsman/DM_868
            export EUPS_PATH=$LSSTSW/stack
            . $LSSTSW/bin/setup.sh

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - The testing of the u/rallsman/ DM-868 based branches has completed successfully. It was necessary to include /u/pgee/ DM-1439 in the branches supplied to my stack in order to get past the afw tests filename conflict. afw itself was not involved in the DM-868 changes but was rebuilt due to them. The stack in question is available for a limited time only (until Review is completed): export LSSTSW=/lsst/home/rallsman/DM_868 export EUPS_PATH=$LSSTSW/stack . $LSSTSW/bin/setup.sh
            Hide
            price Paul Price added a comment -

            Is this out for review then?

            Show
            price Paul Price added a comment - Is this out for review then?
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Not quite when you wrote your comment but now you may feel free to start!
            The packages which were modified were on branch u/rallsman/DM-868 — well, except for pex_exceptions which ended up on master, accidentally.
            ap daf_butlerUtils mysqlclient pyephem
            apr daf_persistence ndarray pyfits
            apr_util datarel obs_sdss qserv_testdata
            boost db obs_test
            cat gsl pex_config testing_endtoend
            cfitsio pex_exceptions utils
            ctrl_events meas_base pex_harness xpa
            ctrl_orca meas_deblender pex_logging
            daf_base mysql pex_policy

            Except for db and meas_deblender, all the changes were isolated to the <pkg>/ups/<pkg>.table file.
            Both db and meas_deblender also had the python/lsst/_init.py updated and meas_deblender also had python/lsst/meas/init_.py updated.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Not quite when you wrote your comment but now you may feel free to start! The packages which were modified were on branch u/rallsman/ DM-868 — well, except for pex_exceptions which ended up on master, accidentally. ap daf_butlerUtils mysqlclient pyephem apr daf_persistence ndarray pyfits apr_util datarel obs_sdss qserv_testdata boost db obs_test cat gsl pex_config testing_endtoend cfitsio pex_exceptions utils ctrl_events meas_base pex_harness xpa ctrl_orca meas_deblender pex_logging daf_base mysql pex_policy Except for db and meas_deblender, all the changes were isolated to the <pkg>/ups/<pkg>.table file. Both db and meas_deblender also had the python/lsst/_ init .py updated and meas_deblender also had python/lsst/meas/ init _.py updated.
            Hide
            price Paul Price added a comment -

            After reviewing a small sample of the changes, it looks fine to me.

            Show
            price Paul Price added a comment - After reviewing a small sample of the changes, it looks fine to me.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            The changes have all been merged to master and buildbot has validated their builds and integration. This Issue is now Done.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - The changes have all been merged to master and buildbot has validated their builds and integration. This Issue is now Done.

              People

              • Assignee:
                robyn Robyn Allsman [X] (Inactive)
                Reporter:
                price Paul Price
                Reviewers:
                Paul Price
                Watchers:
                Mario Juric, Paul Price, Robyn Allsman [X] (Inactive), Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel