Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2303

Odd and intermittent eups failure when expanding tables

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: eups
    • Labels:
      None
    • Team:
      SQuaRE

      Description

      Build 4581 http://lsst-buildx.ncsa.illinois.edu:8010/builders/DM_stack/builds/4581 failed on obs_sdss with an error I’ve never seen before:

      eups expandtable: Processing /lsst/home/lsstsw/stack/Linux64/obs_sdss/tickets.DM-1576-g4aab34585a+1/ups/obs_sdss.table: could not find MARK

      K-T searched for it and it appears to be an unpickling error. I resubmitted the job and it ran the next time.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Ready for review on branch tickets/DM-2303 of sconsUtils.

            I have no idea if this will solve the problem, but it seems possible.

            Show
            jbosch Jim Bosch added a comment - Ready for review on branch tickets/ DM-2303 of sconsUtils. I have no idea if this will solve the problem, but it seems possible.
            Hide
            ktl Kian-Tat Lim added a comment -

            Looks OK to me, although it's kind of tricky.

            Show
            ktl Kian-Tat Lim added a comment - Looks OK to me, although it's kind of tricky.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've had a test build of v10_1_rc2 + tickets/DM-2303 running on el7 for about an hour without a build failure. It's too early to confirm that the issue is resolved based on it's observed rate of occurrence. I've gone ahead and merged the fix as I'd like it to be included in v10_1.

            I'd like to leave this ticket open in the in review status for a few days until I'm confident this is resolved.

            /cc Frossie Economou

            Show
            jhoblitt Joshua Hoblitt added a comment - I've had a test build of v10_1_rc2 + tickets/ DM-2303 running on el7 for about an hour without a build failure. It's too early to confirm that the issue is resolved based on it's observed rate of occurrence. I've gone ahead and merged the fix as I'd like it to be included in v10_1. I'd like to leave this ticket open in the in review status for a few days until I'm confident this is resolved. /cc Frossie Economou
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've completed 5 clean builds of "tickets/DM-2303 v10_1_rc2" el6, el7, fedora 21, ubuntu 12.04, and 14.04 on SSD backed VMs without seeing the expand table issue. Prior to Jim's patch, I would have expected to see the pickle fault multiple times.

            There were 3 or 4 packages that had test failures on fedora 21 that were non-reproducible (they all passed on a second build attempt) but I doubt that was scons related. I'm going to mark this ticket as resolved.

            Jim Bosch is the □ hero for April.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've completed 5 clean builds of "tickets/ DM-2303 v10_1_rc2" el6, el7, fedora 21, ubuntu 12.04, and 14.04 on SSD backed VMs without seeing the expand table issue. Prior to Jim's patch, I would have expected to see the pickle fault multiple times. There were 3 or 4 packages that had test failures on fedora 21 that were non-reproducible (they all passed on a second build attempt) but I doubt that was scons related. I'm going to mark this ticket as resolved. Jim Bosch is the □ hero for April.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Delivery is delayed to these recipients or groups:

            rowen@uw.edu<rowen@uw.edu>

            Subject: [JIRA] (DM-2303) Odd and intermittent eups failure when expanding tables

            This message hasn't been delivered yet. Delivery will continue to be attempted.

            The server will keep trying to deliver this message for the next 1 days, 19 hours and 58 minutes. You'll be notified if the message can't be delivered by that time.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Delivery is delayed to these recipients or groups: rowen@uw.edu< rowen@uw.edu > Subject: [JIRA] ( DM-2303 ) Odd and intermittent eups failure when expanding tables This message hasn't been delivered yet. Delivery will continue to be attempted. The server will keep trying to deliver this message for the next 1 days, 19 hours and 58 minutes. You'll be notified if the message can't be delivered by that time.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                rowen Russell Owen
                Reviewers:
                Joshua Hoblitt
                Watchers:
                Bill Glick, Jim Bosch, Joshua Hoblitt, Kian-Tat Lim, Mario Juric, Paul Price, Robert Lupton, Robyn Allsman [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel