weekly release w_2017_44 (and d_2017_11_0[5678]) failed

XMLWordPrintable

Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.75
• Epic Link:
• Team:
SQuaRE

Description

The canonical build and eupspkg publishing succeeded but the tarball construction failed. I strongly suspect this is the result of changes merged from DM-12510 on Friday, in that multiple products are now being listed for the eups distrib install.

 eups distrib: Product lsst_distrib validate_drp not found in any package repository 

I've confirmed that eupspkgs for validate_drp do exist. My current guess is thatis a shell quoting issue.

Attachments

1. artifacts.png
101 kB
2. screenshot-1.png
90 kB

Activity

Hide
Joshua Hoblitt added a comment -

It appears, that after 5 attempts, the the tarball issue is resolved. The cause was eups distrib install will not operate on multiple product names. There was then a comedy of shell logic errors and groovy heredoc interpolation problems attempt to put a trivial loop around eups distrib install.

However, the jenkins stack-build (docker image) job has a similar problem which I will also fix on this ticket.

Show
Joshua Hoblitt added a comment - It appears, that after 5 attempts, the the tarball issue is resolved. The cause was eups distrib install will not operate on multiple product names. There was then a comedy of shell logic errors and groovy heredoc interpolation problems attempt to put a trivial loop around eups distrib install . However, the jenkins stack-build (docker image) job has a similar problem which I will also fix on this ticket.
Hide
Joshua Hoblitt added a comment -

The distrib install issue appears to be resolved but validate_drp is still failing to distrib install on osx due to a test failure that does not occur under lsstsw. This looks like a path length issue.

 E OSError: [Errno 63] File name too long: 'Users_square_jenkins_workspace_release_tarball_osx_10.9_clang-800.0.42.1_miniconda3-4.3.21-10a4fa6_build_stack_miniconda3-4.3.21-10a4fa6_EupsBuildDir_DarwinX86_validate_drp-master-g960648a5d8+11_validate_drp-master-g960648a5d8+11_tests_CfhtQuick_output_r_AM1_D_5_arcmin_17.0_21.5_mag.png' 

I'm unsure of how to resolve that as the majority of that path is constructed by eups distrib install. However, this is good reason/opportunity to implement junit reporting and log artifacting for tarball builds.

Show
Joshua Hoblitt added a comment - The distrib install issue appears to be resolved but validate_drp is still failing to distrib install on osx due to a test failure that does not occur under lsstsw . This looks like a path length issue. E OSError: [Errno 63 ] File name too long : 'Users_square_jenkins_workspace_release_tarball_osx_10.9_clang-800.0.42.1_miniconda3-4.3.21-10a4fa6_build_stack_miniconda3-4.3.21-10a4fa6_EupsBuildDir_DarwinX86_validate_drp-master-g960648a5d8+11_validate_drp-master-g960648a5d8+11_tests_CfhtQuick_output_r_AM1_D_5_arcmin_17.0_21.5_mag.png' I'm unsure of how to resolve that as the majority of that path is constructed by eups distrib install . However, this is good reason/opportunity to implement junit reporting and log artifacting for tarball builds.
Hide
Joshua Hoblitt added a comment -
Show
Joshua Hoblitt added a comment - Testing a possible fix for the osx test failure: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27018/pipeline
Hide
Joshua Hoblitt added a comment -

The fix passed on stack-os-matrix, this was excepted either way as that job builds with lsstsw. That branch was built and published jhoblitt-validate_drp-osx and is being test build as tarballs (that will not be published):

https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball-matrix/detail/tarball-matrix/28/pipeline/31

One of the OSX configurations has completed, so it looks probably that this is a working fix.

Show
Joshua Hoblitt added a comment - The fix passed on stack-os-matrix , this was excepted either way as that job builds with lsstsw . That branch was built and published jhoblitt-validate_drp-osx and is being test build as tarballs (that will not be published): https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball-matrix/detail/tarball-matrix/28/pipeline/31 One of the OSX configurations has completed, so it looks probably that this is a working fix.
Hide
Joshua Hoblitt added a comment -

This issue was resolved by fixing several jobs to loop over eups distrb install when invoked with multiple (space separated) products and fixing the validate_drp test failure on OSX due to generating extremely long filenames.

Log recording + junit report generation were added to the tarball job and appears to be working (only for failed products due to eups distrib install not saving the build dir for successful products) – see attached screenshots.

Show
Joshua Hoblitt added a comment - This issue was resolved by fixing several jobs to loop over eups distrb install when invoked with multiple (space separated) products and fixing the validate_drp test failure on OSX due to generating extremely long filenames. Log recording + junit report generation were added to the tarball job and appears to be working (only for failed products due to eups distrib install not saving the build dir for successful products) – see attached screenshots.
Hide
Joshua Hoblitt added a comment -

Also worthy of mention is that the w_2017_44 release was manually walked through published tarballs for lsst_distrib on all platforms. validate_drp was only published for Linux due to the test failure on OSX.

Show
Joshua Hoblitt added a comment - Also worthy of mention is that the w_2017_44 release was manually walked through published tarballs for lsst_distrib on all platforms. validate_drp was only published for Linux due to the test failure on OSX.

People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Watchers:
Adam Thornton, Frossie Economou, John Swinbank, Joshua Hoblitt, Simon Krughoff
• Votes:
0 Vote for this issue
Watchers:
5 Start watching this issue

Dates

• Created:
Updated:
Resolved: