Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3403

Fix parallel build in generating defects

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_subaru
    • Labels:
      None
    • Team:
      Data Release Production

      Description

      Building obs_subaru can fail with errors like the following:

      'import sitecustomize' failed; use -v for traceback
      'import sitecustomize' failed; use -v for traceback
      'import sitecustomize' failed; use -v for traceback
      Traceback (most recent call last):
        File "bin/genDefectFits.py", line 12, in <module>
      Traceback (most recent call last):
        File "bin/genDefectFits.py", line 12, in <module>
          from lsst.obs.hsc import HscMapper
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/__init__.py", line 2, in <module>
      Traceback (most recent call last):
        File "bin/genDefectFits.py", line 12, in <module>
          from lsst.obs.hsc import HscMapper
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/__init__.py", line 2, in <module>
          from lsst.obs.hsc import HscMapper
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/__init__.py", line 2, in <module>
              from .transformRegistry import *
      from .transformRegistry import *
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/transformRegistry.py", line 3, in <module>
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/transformRegistry.py", line 3, in <module>
          from .transformRegistry import *
        File "/nfs/home/price/LSST/obs/subaru/python/lsst/obs/hsc/transformRegistry.py", line 3, in <module>
          from lsst.obs.subaru.subaruLib import HscDistortion, DistortionPolynomial
      ImportError: No module named subaruLib
          from lsst.obs.subaru.subaruLib import HscDistortion, DistortionPolynomial
          from lsst.obs.subaru.subaruLib import HscDistortion, DistortionPolynomial
      ImportError: No module named subaruLib
      ImportError: No module named subaruLib
      scons: *** [hsc/defects/2013-01-31/defects.dat-fits] Error 1
      scons: *** [hsc/defects/2014-06-01/defects.dat-fits] Error 1
      scons: *** [hsc/defects/2014-04-03/defects.dat-fits] Error 1
      scons: building terminated because of errors.
      

      This is due to a missing dependency for building the defects FITS files.

      This is a pull request for a simple fix.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          Lauren MacArthur, would you please approve this really simple patch?

          price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3403] $ git sub-patch
          commit 58801671c8976c62c1ce70530fb9a20f12da66b5
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Aug 5 19:00:17 2015 +0000
           
              Fix dependency in building defects FITS files
              
              They depend on the swig module.
           
          diff --git a/hsc/SConscript b/hsc/SConscript
          index 7067e69..f60decf 100644
          --- a/hsc/SConscript
          +++ b/hsc/SConscript
          @@ -17,7 +17,7 @@ dep = "#python/lsst/obs/subaru/_subaruLib.so" # Dependency required for running
           defectsDatList = glob.glob("defects/*/defects.dat")
           for defectsDat in defectsDatList:
               command = "%s bin/genDefectFits.py HSC %s --force" % (python, os.path.join("hsc", defectsDat))
          -    env.AlwaysBuild(env.Command(defectsDat + "-fits", defectsDat, command))
          +    env.AlwaysBuild(env.Command(defectsDat + "-fits", [defectsDat, dep], command))
           
           command = "%s bin/genDefectRegistry.py --root hsc/defects" % (python)
           registry = env.AlwaysBuild(env.Command("defects/defectRegistry.sqlite3",
          

          Show
          price Paul Price added a comment - Lauren MacArthur , would you please approve this really simple patch? price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3403] $ git sub-patch commit 58801671c8976c62c1ce70530fb9a20f12da66b5 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 5 19:00:17 2015 +0000   Fix dependency in building defects FITS files They depend on the swig module.   diff --git a/hsc/SConscript b/hsc/SConscript index 7067e69..f60decf 100644 --- a/hsc/SConscript +++ b/hsc/SConscript @@ -17,7 +17,7 @@ dep = "#python/lsst/obs/subaru/_subaruLib.so" # Dependency required for running defectsDatList = glob.glob("defects/*/defects.dat") for defectsDat in defectsDatList: command = "%s bin/genDefectFits.py HSC %s --force" % (python, os.path.join("hsc", defectsDat)) - env.AlwaysBuild(env.Command(defectsDat + "-fits", defectsDat, command)) + env.AlwaysBuild(env.Command(defectsDat + "-fits", [defectsDat, dep], command)) command = "%s bin/genDefectRegistry.py --root hsc/defects" % (python) registry = env.AlwaysBuild(env.Command("defects/defectRegistry.sqlite3",
          Hide
          lauren Lauren MacArthur added a comment -

          Just confirmed that the build failed without it, but passed with it. Free to merge.

          Show
          lauren Lauren MacArthur added a comment - Just confirmed that the build failed without it, but passed with it. Free to merge.
          Hide
          price Paul Price added a comment -

          Thanks!

          Merged to master.

          Show
          price Paul Price added a comment - Thanks! Merged to master.
          Hide
          lauren Lauren MacArthur added a comment -

          Paul, I just tried to build the master obs_subaru on lsst-dev and got the above errors, even though this update has been merged. Any ideas?

          Show
          lauren Lauren MacArthur added a comment - Paul, I just tried to build the master obs_subaru on lsst-dev and got the above errors, even though this update has been merged. Any ideas?
          Hide
          price Paul Price added a comment -

          As resolved on HipChat, this was due to a mismatch between anaconda and the LSST stack; resolved by using the proper anaconda.

          Show
          price Paul Price added a comment - As resolved on HipChat, this was due to a mismatch between anaconda and the LSST stack; resolved by using the proper anaconda.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Lauren MacArthur
              Watchers:
              Lauren MacArthur, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel