Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15415

weekly-release w.2018.32 failed

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: stack release
    • Labels:
      None
    • Story Points:
      0.1
    • Team:
      Architecture

      Description

      The build of w.2018.32 failed during tagging due to new eups products being pulled into the build which did not have the required github team membership. It appears that this was corrected and the weekly build resubmitted.

      The 2nd weekly attempt failing attempting to eups distrib install lsst/scarlet as part of the binary tarball build. The failure is clearly from the setup.py.

      Writing log to: /build/stack/miniconda3-4.5.4-10a4fa6/EupsBuildDir/Linux64/scarlet-lsst-dev-gc293e6f0ef/build.log
      [build] + eval 'VAL_=$VERSION'
      ++ VAL_=lsst-dev-gc293e6f0ef
      + '[' -z lsst-dev-gc293e6f0ef ']'
      + [[ -f SConstruct ]]
      + [[ -f configure ]]
      + [[ -f Makefile ]]
      + [[ -f makefile ]]
      + [[ -f GNUmakefile ]]
      + [[ -f setup.py ]]
      + python setup.py build
      fatal: Not a git repository (or any of the parent directories): .git
      Traceback (most recent call last):
        File "setup.py", line 21, in <module>
          __version__ = '0.3.'+subprocess.check_output(['git', 'rev-parse', 'HEAD'])[:7].decode("utf-8")
        File "/build/python/miniconda3-4.5.4/envs/lsst-scipipe-10a4fa6/lib/python3.6/subprocess.py", line 336, in check_output
          **kwargs).stdout
        File "/build/python/miniconda3-4.5.4/envs/lsst-scipipe-10a4fa6/lib/python3.6/subprocess.py", line 418, in run
          output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.
      + exit -4
      

      https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball/detail/tarball/3173/pipeline

      The failure appears to be caused by shelling out to run git from setup.py, resulting in an eups product that can be build under lsst-build but not by eups distrib install – which means that binary tarballs can not be built. In order for the weekly to run either lsst/scarlet needs to be updated to not assume the build is running in a git repo or be converted to use standard python package tooling such as setuptools_scm or DM-15104 should be reverted until the packaging is repaired.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I'm pretty sure that DM-15104 is needed for the deblending workshop this week at LSST2018 so I don't think we can revert that. I've put in a quick fix on the lsst-dev branch of scarlet which should trap for git not being available. Obviously the fix is not perfect (regardless, setuptools complains about the version formatting but that's for Fred Moolekamp to fix) but I hope it fixes the immediate problem.

            Show
            tjenness Tim Jenness added a comment - I'm pretty sure that DM-15104 is needed for the deblending workshop this week at LSST2018 so I don't think we can revert that. I've put in a quick fix on the lsst-dev branch of scarlet which should trap for git not being available. Obviously the fix is not perfect (regardless, setuptools complains about the version formatting but that's for Fred Moolekamp to fix) but I hope it fixes the immediate problem.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Tim Jenness It is unfortunate that end-to-end testing in advance wasn't considered a blocker for the workshop. Also, is scarlet a true 3rd party package? It isn't install-able from pypi in its current state, so it seems likely that LSST pipelines is the only consumer. There don't appear to be any unit tests, which is a bit troubling for a package considered a critical dependency.

            Show
            jhoblitt Joshua Hoblitt added a comment - Tim Jenness It is unfortunate that end-to-end testing in advance wasn't considered a blocker for the workshop. Also, is scarlet a true 3rd party package? It isn't install-able from pypi in its current state, so it seems likely that LSST pipelines is the only consumer. There don't appear to be any unit tests, which is a bit troubling for a package considered a critical dependency.
            Hide
            fred3m Fred Moolekamp added a comment -

            scarlet is not considered a critical dependency yet. Currently the default deblender is still the HSC-SDSS deblender that has been running in meas_deblender. The work this week was to integrate scarlet so that it can be setup to run in the stack in parallel with the current deblender for future testing, and so that it can be demonstrated during the workshop.

             I admit that the lack of testing is very unfortunate, but in order to have scarlet integrated into the stack before I leave the project at the end of next week we prioritized refactoring the task structure so that scarlet can be run, since Peter Melchior and I still plan on developing scarlet (including writing test code in the near future) but neither of us will be responsible for maintaining the stack implementation.

            So it was important to finish DM-15104 and DM-15246 this week. I apologize for all of the problems that this caused, I wasn't aware that there was any danger once all of the Jenkins tests passed and ci_hsc built and ran successfully.

            Thank you Tim Jenness for the quick fix.

            Show
            fred3m Fred Moolekamp added a comment - scarlet is not considered a critical dependency yet. Currently the default deblender is still the HSC-SDSS deblender that has been running in meas_deblender . The work this week was to integrate scarlet so that it can be setup to run in the stack in parallel with the current deblender for future testing, and so that it can be demonstrated during the workshop.  I admit that the lack of testing is very unfortunate, but in order to have scarlet integrated into the stack before I leave the project at the end of next week we prioritized refactoring the task structure so that scarlet can be run, since Peter Melchior and I still plan on developing scarlet (including writing test code in the near future) but neither of us will be responsible for maintaining the stack implementation. So it was important to finish DM-15104 and DM-15246 this week. I apologize for all of the problems that this caused, I wasn't aware that there was any danger once all of the Jenkins tests passed and ci_hsc built and ran successfully. Thank you Tim Jenness for the quick fix.
            Hide
            gcomoretto Gabriele Comoretto added a comment -

            Tim Jenness change:

            commit fdfbe1c24037acfb2a9f7f87f3f505e4df92f041 (HEAD -> lsst-dev, tag: w.2018.32, origin/lsst-dev)

            Author: Tim Jenness <tjenness@lsst.org>

            Date:   Sat Aug 11 18:30:20 2018 -0700

             

            seems to have solved the problem, since the scarlet binary is now available in EUPS:

            https://eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.5.4-10a4fa6/scarlet-lsst-dev-gfdfbe1c240@Linux64.tar.gz

            We can just wait for confirmation next weekly to be build correctly before closing this issue.

            Show
            gcomoretto Gabriele Comoretto added a comment - Tim Jenness change: commit fdfbe1c24037acfb2a9f7f87f3f505e4df92f041 ( HEAD -> lsst-dev , tag: w.2018.32 , origin/lsst-dev ) Author: Tim Jenness <tjenness@lsst.org> Date:   Sat Aug 11 18:30:20 2018 -0700   seems to have solved the problem, since the scarlet binary is now available in EUPS: https://eups.lsst.codes/stack/redhat/el7/devtoolset-6/miniconda3-4.5.4-10a4fa6/scarlet-lsst-dev-gfdfbe1c240@Linux64.tar.gz We can just wait for confirmation next weekly to be build correctly before closing this issue.
            Hide
            tjenness Tim Jenness added a comment -

            The new weekly did come out. Fred Moolekamp can you please ensure that you make a similar fix upstream (and consider creating proper development release version numbers to prevent the other warning).

            Show
            tjenness Tim Jenness added a comment - The new weekly did come out. Fred Moolekamp can you please ensure that you make a similar fix upstream (and consider creating proper development release version numbers to prevent the other warning).
            Hide
            fred3m Fred Moolekamp added a comment -

            Sure, I'll make the same fix upstream so that this will not happen in the future.

            Show
            fred3m Fred Moolekamp added a comment - Sure, I'll make the same fix upstream so that this will not happen in the future.

              People

              • Assignee:
                tjenness Tim Jenness
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Adam Thornton, Fred Moolekamp, Frossie Economou, Gabriele Comoretto, Joshua Hoblitt, Kian-Tat Lim, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel