Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12516

weekly release w_2017_44 (and d_2017_11_0[5678]) failed

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      The canonical build and eupspkg publishing succeeded but the tarball construction failed. I strongly suspect this is the result of changes merged from DM-12510 on Friday, in that multiple products are now being listed for the `eups distrib install`.

      eups distrib: Product lsst_distrib validate_drp not found in any package repository
      

      I've confirmed that eupspkgs for validate_drp do exist. My current guess is thatis a shell quoting issue.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            It appears, that after 5 attempts, the the tarball issue is resolved. The cause was eups distrib install will not operate on multiple product names. There was then a comedy of shell logic errors and groovy heredoc interpolation problems attempt to put a trivial loop around eups distrib install.

            However, the jenkins stack-build (docker image) job has a similar problem which I will also fix on this ticket.

            Show
            jhoblitt Joshua Hoblitt added a comment - It appears, that after 5 attempts, the the tarball issue is resolved. The cause was eups distrib install will not operate on multiple product names. There was then a comedy of shell logic errors and groovy heredoc interpolation problems attempt to put a trivial loop around eups distrib install . However, the jenkins stack-build (docker image) job has a similar problem which I will also fix on this ticket.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The distrib install issue appears to be resolved but validate_drp is still failing to distrib install on osx due to a test failure that does not occur under lsstsw. This looks like a path length issue.

            E           OSError: [Errno 63] File name too long: 'Users_square_jenkins_workspace_release_tarball_osx_10.9_clang-800.0.42.1_miniconda3-4.3.21-10a4fa6_build_stack_miniconda3-4.3.21-10a4fa6_EupsBuildDir_DarwinX86_validate_drp-master-g960648a5d8+11_validate_drp-master-g960648a5d8+11_tests_CfhtQuick_output_r_AM1_D_5_arcmin_17.0_21.5_mag.png'
            

            I'm unsure of how to resolve that as the majority of that path is constructed by eups distrib install. However, this is good reason/opportunity to implement junit reporting and log artifacting for tarball builds.

            Show
            jhoblitt Joshua Hoblitt added a comment - The distrib install issue appears to be resolved but validate_drp is still failing to distrib install on osx due to a test failure that does not occur under lsstsw . This looks like a path length issue. E OSError: [Errno 63 ] File name too long : 'Users_square_jenkins_workspace_release_tarball_osx_10.9_clang-800.0.42.1_miniconda3-4.3.21-10a4fa6_build_stack_miniconda3-4.3.21-10a4fa6_EupsBuildDir_DarwinX86_validate_drp-master-g960648a5d8+11_validate_drp-master-g960648a5d8+11_tests_CfhtQuick_output_r_AM1_D_5_arcmin_17.0_21.5_mag.png' I'm unsure of how to resolve that as the majority of that path is constructed by eups distrib install . However, this is good reason/opportunity to implement junit reporting and log artifacting for tarball builds.
            Hide
            jhoblitt Joshua Hoblitt added a comment -
            Show
            jhoblitt Joshua Hoblitt added a comment - Testing a possible fix for the osx test failure: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27018/pipeline
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The fix passed on stack-os-matrix, this was excepted either way as that job builds with lsstsw. That branch was built and published jhoblitt-validate_drp-osx and is being test build as tarballs (that will not be published):

            https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball-matrix/detail/tarball-matrix/28/pipeline/31

            One of the OSX configurations has completed, so it looks probably that this is a working fix.

            Show
            jhoblitt Joshua Hoblitt added a comment - The fix passed on stack-os-matrix , this was excepted either way as that job builds with lsstsw . That branch was built and published jhoblitt-validate_drp-osx and is being test build as tarballs (that will not be published): https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball-matrix/detail/tarball-matrix/28/pipeline/31 One of the OSX configurations has completed, so it looks probably that this is a working fix.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            This issue was resolved by fixing several jobs to loop over eups distrb install when invoked with multiple (space separated) products and fixing the validate_drp test failure on OSX due to generating extremely long filenames.

            Log recording + junit report generation were added to the tarball job and appears to be working (only for failed products due to eups distrib install not saving the build dir for successful products) – see attached screenshots.

            Show
            jhoblitt Joshua Hoblitt added a comment - This issue was resolved by fixing several jobs to loop over eups distrb install when invoked with multiple (space separated) products and fixing the validate_drp test failure on OSX due to generating extremely long filenames. Log recording + junit report generation were added to the tarball job and appears to be working (only for failed products due to eups distrib install not saving the build dir for successful products) – see attached screenshots.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Also worthy of mention is that the w_2017_44 release was manually walked through published tarballs for lsst_distrib on all platforms. validate_drp was only published for Linux due to the test failure on OSX.

            Show
            jhoblitt Joshua Hoblitt added a comment - Also worthy of mention is that the w_2017_44 release was manually walked through published tarballs for lsst_distrib on all platforms. validate_drp was only published for Linux due to the test failure on OSX.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Adam Thornton, Frossie Economou, John Swinbank, Joshua Hoblitt, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel