Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7010

Builds should be optimised by default

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: sconsUtils
    • Labels:
      None
    • Story Points:
      0.5
    • Epic Link:
    • Sprint:
      DRP F16-2
    • Team:
      Data Release Production

      Description

      By default, our builds are not optimised (-O0), which requires everyone who doesn't want to wait until the heat death of the universe to set SCONSFLAGS="opt=3", but other packages that are built with scons may not recognise this. This default is also contrary to the standard practise for open-source software, which is that by default builds are optimised. I will change the default optimisation level to opt=3 from the current opt=0. I will also add support for -Og.

      This change was approved in RFC-202.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Tim Jenness, would you review this please? I've implemented what I threatened, and observed that it works (-O3 is default, can set -Og). I also fixed the annoying error: no result from the compiler classifier.

            price@price-laptop:~/LSST/sconsUtils (tickets/DM-7010=) $ git sub
            commit b5e6e6106b33172be28b07195fe5b3c63323fda7
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jul 25 10:48:14 2016 -0400
             
                set default optimisation level to 3
                
                Our C++ code needs good optimisation to run efficiently, but this
                was not set by default.
             
             python/lsst/sconsUtils/state.py | 2 +-
             1 file changed, 1 insertion(+), 1 deletion(-)
             
            commit b6cc932bcadb347e719aaa45d0d24e75e0488729
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jul 25 10:51:44 2016 -0400
             
                add support for -Og
                
                This optimisation flag is supported from gcc 4.8, and is apparently
                useful for debugging C++ code.
             
             python/lsst/sconsUtils/state.py | 6 +++---
             1 file changed, 3 insertions(+), 3 deletions(-)
             
            commit 29ad0e61099543c74249f1998e055ea5c6eab8a8
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jul 25 11:31:21 2016 -0400
             
                ClassifyCc: fix "error: no result"
                
                We weren't setting a result, so the default result ("error: no result")
                was being used for us, which can confuse the user.
             
             python/lsst/sconsUtils/state.py | 2 ++
             1 file changed, 2 insertions(+)
            

            Show
            price Paul Price added a comment - Tim Jenness , would you review this please? I've implemented what I threatened, and observed that it works ( -O3 is default, can set -Og ). I also fixed the annoying error: no result from the compiler classifier. price@price-laptop:~/LSST/sconsUtils (tickets/DM-7010=) $ git sub commit b5e6e6106b33172be28b07195fe5b3c63323fda7 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jul 25 10:48:14 2016 -0400   set default optimisation level to 3 Our C++ code needs good optimisation to run efficiently, but this was not set by default.   python/lsst/sconsUtils/state.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)   commit b6cc932bcadb347e719aaa45d0d24e75e0488729 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jul 25 10:51:44 2016 -0400   add support for -Og This optimisation flag is supported from gcc 4.8, and is apparently useful for debugging C++ code.   python/lsst/sconsUtils/state.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)   commit 29ad0e61099543c74249f1998e055ea5c6eab8a8 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jul 25 11:31:21 2016 -0400   ClassifyCc: fix "error: no result" We weren't setting a result, so the default result ("error: no result") was being used for us, which can confuse the user.   python/lsst/sconsUtils/state.py | 2 ++ 1 file changed, 2 insertions(+)
            Hide
            tjenness Tim Jenness added a comment -

            Looks good to me.

            Show
            tjenness Tim Jenness added a comment - Looks good to me.
            Hide
            price Paul Price added a comment -

            sims_utils keeps failing under Jenkins.

            ### sims_utils - testModifiedJulianDate.py.failed
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            tests/testModifiedJulianDate.py
             
            /home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-6/python/py2/lsstsw/miniconda/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
              warn(ConfigurationMissingWarning(msg))
            /home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-6/python/py2/lsstsw/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
              warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
            WARNING: CacheMissingWarning: Remote data cache could not be accessed due to OSError: Could not find unix home directory to search for astropy config dir [astropy.utils.data]
            Downloading http://maia.usno.navy.mil/ser7/finals2000A.all [Done]
            WARNING: CacheMissingWarning: (u'File downloaded to temporary location due to problem with cache directory and will not be cached.', '/tmp/tmpL9wmVf') [astropy.utils.data]
            .....F
            ======================================================================
            FAIL: test_warnings (__main__.MjdTest)
            ----------------------------------------------------------------------
            Traceback (most recent call last):
              File "tests/testModifiedJulianDate.py", line 134, in test_warnings
                self.assertIsInstance(w_list[1].message, UTCtoUT1Warning)
            AssertionError: ErfaWarning('ERFA function "utcut1" yielded 1 of "dubious year (Note 3)"',) is not an instance of <class 'lsst.sims.utils.ModifiedJulianDate.UTCtoUT1Warning'>
             
            ----------------------------------------------------------------------
            Ran 6 tests in 8.620s
             
            FAILED (failures=1)
            

            This looks like it's someone else's fault, but it's keeping me from finishing the Jenkins run.

            Show
            price Paul Price added a comment - sims_utils keeps failing under Jenkins. ### sims_utils - testModifiedJulianDate.py.failed ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tests/testModifiedJulianDate.py   /home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-6/python/py2/lsstsw/miniconda/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None warn(ConfigurationMissingWarning(msg)) /home/jenkins-slave/workspace/stack-os-matrix/compiler/gcc/label/centos-6/python/py2/lsstsw/miniconda/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment. warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.') WARNING: CacheMissingWarning: Remote data cache could not be accessed due to OSError: Could not find unix home directory to search for astropy config dir [astropy.utils.data] Downloading http://maia.usno.navy.mil/ser7/finals2000A.all [Done] WARNING: CacheMissingWarning: (u'File downloaded to temporary location due to problem with cache directory and will not be cached.', '/tmp/tmpL9wmVf') [astropy.utils.data] .....F ====================================================================== FAIL: test_warnings (__main__.MjdTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/testModifiedJulianDate.py", line 134, in test_warnings self.assertIsInstance(w_list[1].message, UTCtoUT1Warning) AssertionError: ErfaWarning('ERFA function "utcut1" yielded 1 of "dubious year (Note 3)"',) is not an instance of <class 'lsst.sims.utils.ModifiedJulianDate.UTCtoUT1Warning'>   ---------------------------------------------------------------------- Ran 6 tests in 8.620s   FAILED (failures=1) This looks like it's someone else's fault, but it's keeping me from finishing the Jenkins run.
            Hide
            price Paul Price added a comment -

            It passes for me when I run it manually (though I fiddled with the environment some to get the necessary modules). Also note that there is no C++ in that module for my optimisation changes to impact. I believe there's something funky with the system, and have asked Joshua Hoblitt to look at it.

            Show
            price Paul Price added a comment - It passes for me when I run it manually (though I fiddled with the environment some to get the necessary modules). Also note that there is no C++ in that module for my optimisation changes to impact. I believe there's something funky with the system, and have asked Joshua Hoblitt to look at it.
            Hide
            price Paul Price added a comment -

            After a build of master sims_utils failed on Jenkins, we filed SIM-2067 to address that.

            Show
            price Paul Price added a comment - After a build of master sims_utils failed on Jenkins, we filed SIM-2067 to address that.
            Hide
            price Paul Price added a comment -

            From HipChat:

            [12:00 AM] Paul Price: OK, can I merge 7010 to master without running Jenkins on the sims side?
            [12:00 AM] K-T Lim: If they're broken, and you don't break them worse, then I'd say OK.

            Merged to master.

            Show
            price Paul Price added a comment - From HipChat: [12:00 AM] Paul Price: OK, can I merge 7010 to master without running Jenkins on the sims side? [12:00 AM] K-T Lim: If they're broken, and you don't break them worse, then I'd say OK. Merged to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Tim Jenness
              Watchers:
              Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.