Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5132

obs_subaru install with eups distrib fails

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: lsstsw, obs_subaru
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      Science Pipelines DM-W16-6
    • Team:
      Data Release Production

      Description

      Thus:

      $ eups distrib install -t w_2016_06 obs_subaru
      ...
        [ 52/52 ]  obs_subaru 5.0.0.1-60-ge4efae7+2 ...
       
      ***** error: from /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/build.log:
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "tests/hscRepository.py", line 91, in setUp
          self.repoPath = createDataRepository("lsst.obs.hsc.HscMapper", rawPath)
        File "tests/hscRepository.py", line 63, in createDataRepository
          check_call([ingest_cmd, repoPath] + glob(os.path.join(inputPath, "*.fits.gz")))
        File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
          raise CalledProcessError(retcode, cmd)
      CalledProcessError: Command '['/Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/bin/hscIngestImages.py', '/var/folders/jp/lqz3n0m17nqft7bwtw3b8n380000gp/T/tmptUSKuf', '/Users/jds/Projects/Astronomy/LSST/stack/DarwinX86/testdata_subaru/master-gf9ba9abdbe/hsc/raw/HSCA90402512.fits.gz']' returned non-zero exit status 1
       
      ----------------------------------------------------------------------
      Ran 8 tests in 9.928s
       
      FAILED (errors=7)
      The following tests failed:
      /Users/jds/Projects/Astronomy/LSST/stack/EupsBuildDir/DarwinX86/obs_subaru-5.0.0.1-60-ge4efae7+2/obs_subaru-5.0.0.1-60-ge4efae7+2/tests/.tests/hscRepository.py.failed
      1 tests failed
      scons: *** [checkTestStatus] Error 1
      scons: building terminated because of errors.
      + exit -4
      

      Please fix it.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            $ eups distrib install -t fe_test4 obs_subaru
               [  1/51 ]  cfitsio 3360.lsst4 (already installed)                     done.
            ...
               [ 51/51 ]  obs_subaru 5.0.0.1-63-gfd8740e+1                           done.
            

            Thanks, everybody!

            Show
            swinbank John Swinbank added a comment - $ eups distrib install -t fe_test4 obs_subaru [ 1/51 ] cfitsio 3360.lsst4 (already installed) done. ... [ 51/51 ] obs_subaru 5.0.0.1-63-gfd8740e+1 done. Thanks, everybody!
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            It looks like there were multiple issues going. Mario Juric identified the most severe with distrib packages being unaware of git-lfs. Another, minor, issue was that git tags were not being applied to the repo – resolved by Tim Jenness. I have merged John Swinbank PR to add testsdata_subaru to the remap list. TL;DR – the w_2016_06 weekly tag is broken for obs_subaru.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited It looks like there were multiple issues going. Mario Juric identified the most severe with distrib packages being unaware of git-lfs. Another, minor, issue was that git tags were not being applied to the repo – resolved by Tim Jenness . I have merged John Swinbank PR to add testsdata_subaru to the remap list. TL;DR – the w_2016_06 weekly tag is broken for obs_subaru .
            Hide
            swinbank John Swinbank added a comment -

            Thanks, Mario Juric. It's appropriate to treat testdata_subaru as analogous to afwdata, so blacklisting package creation seems like the way to go. The wider issue of eups distrib/git lfs integration isn't a current requirement.

            Joshua Hoblitt, are you happy with the changes on lsstsw PR#83?

            Show
            swinbank John Swinbank added a comment - Thanks, Mario Juric . It's appropriate to treat testdata_subaru as analogous to afwdata , so blacklisting package creation seems like the way to go. The wider issue of eups distrib / git lfs integration isn't a current requirement. Joshua Hoblitt , are you happy with the changes on lsstsw PR#83 ?
            Hide
            mjuric Mario Juric added a comment -

            EUPS doesn't know about git-lfs; we need to teach it about it, or you can also override the create() and/or fetch() verbs in eupspkg.cfg.sh in the short term.

            To answer the original question, I think it's fine to blacklist this package in manifest.remap if you don't expect end users to want to install it using EUPS.

            Show
            mjuric Mario Juric added a comment - EUPS doesn't know about git-lfs; we need to teach it about it, or you can also override the create() and/or fetch() verbs in eupspkg.cfg.sh in the short term. To answer the original question, I think it's fine to blacklist this package in manifest.remap if you don't expect end users to want to install it using EUPS.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment - - edited

            Should we create tests for the testdata_* packages that explicitly verify that the git-lfs pull worked?
            This would prevent such packages from being setup.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - - edited Should we create tests for the testdata_* packages that explicitly verify that the git-lfs pull worked? This would prevent such packages from being setup .
            Hide
            swinbank John Swinbank added a comment - - edited

            By the way, I imagine this is breaking attempts to install lsst_distrib through eups distrib.

            Show
            swinbank John Swinbank added a comment - - edited By the way, I imagine this is breaking attempts to install lsst_distrib through eups distrib .
            Hide
            swinbank John Swinbank added a comment -

            The obs_subaru table file declares testdata_subaru as setupOptional. However, eups distrib still attempts to install it:

            $ eups distrib install -t w_2016_06 obs_subaru
            ...
               [ 11/52 ]  testdata_subaru master-gf9ba9abdbe (already installed)     done.
            ...
            

            This produces an installed version which has not checked out files from git lfs:

            $ cat ${TESTDATA_SUBARU_DIR}/hsc/calib/BIAS/2013-11-02/NONE/master/BIAS-050.fits.gz
            version https://git-lfs.github.com/spec/v1
            oid sha256:d70512df8c1fd25f5c9bcc5b95f413c62f65da752f50a3da0a8331686682ad5a
            size 31345503
            

            obs_subaru will skip tests gracefully if testdata_subaru is not set up, however here it is set up but is populated with (from the point of view of the tests, at least) garbage data.

            My suggested fix is that it should be treated in the same way as afwdata in lsstsw/etc/manifest.remap, which will suppress the creation of the package – I guess this will do the trick, as it's evidently possible to install afw through eups distrib. I'd appreciate input from eups distrib experts (Mario Juric?) on whether this is actually the correct solution, though.

            Even if that fixes the immediate failure, I'm not sure if the failure of eups distrib install to check out the contents of files from git lfs is a bug or a feature.

            Show
            swinbank John Swinbank added a comment - The obs_subaru table file declares testdata_subaru as setupOptional . However, eups distrib still attempts to install it: $ eups distrib install -t w_2016_06 obs_subaru ... [ 11/52 ] testdata_subaru master-gf9ba9abdbe (already installed) done. ... This produces an installed version which has not checked out files from git lfs : $ cat ${TESTDATA_SUBARU_DIR}/hsc/calib/BIAS/2013-11-02/NONE/master/BIAS-050.fits.gz version https://git-lfs.github.com/spec/v1 oid sha256:d70512df8c1fd25f5c9bcc5b95f413c62f65da752f50a3da0a8331686682ad5a size 31345503 obs_subaru will skip tests gracefully if testdata_subaru is not set up, however here it is set up but is populated with (from the point of view of the tests, at least) garbage data. My suggested fix is that it should be treated in the same way as afwdata in lsstsw/etc/manifest.remap , which will suppress the creation of the package – I guess this will do the trick, as it's evidently possible to install afw through eups distrib . I'd appreciate input from eups distrib experts ( Mario Juric ?) on whether this is actually the correct solution, though. Even if that fixes the immediate failure, I'm not sure if the failure of eups distrib install to check out the contents of files from git lfs is a bug or a feature.

              People

              • Assignee:
                swinbank John Swinbank
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Joshua Hoblitt
                Watchers:
                John Swinbank, Joshua Hoblitt, Mario Juric, Michael Wood-Vasey, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel