Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9315

MarkCompleted threw an exception when a Tracked job could not be located.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Investigate why the job could not be found and take appropriate action.

      [2017-02-06T19:14:21.430-0600] [LWP:20631] DEBUG qdisp.QueryRequest (core/modules/qdisp/QueryRequest.cc:194) - QI=1943:33; ProcessResponseData with buflen=118 (more)
      [2017-02-06T19:14:21.561-0600] [LWP:20631] DEBUG rproc.InfileMerger (core/modules/rproc/InfileMerger.cc:155) - Executing InfileMerger::merge(QI=1943:33; largeResult=0 sizes=45, 118, rowCount=0, row_size=0, errCode=0 hasErMsg=1)
      [2017-02-06T19:14:21.726-0600] [LWP:20631] DEBUG qdisp.QueryRequest (core/modules/qdisp/QueryRequest.cc:235) - QI=1943:33; ProcessResponse data flush failed
      [2017-02-06T19:14:21.745-0600] [LWP:20631] DEBUG qdisp.QueryRequest (core/modules/qdisp/QueryRequest.cc:306) - QI=1943:33; QueryRequest::_errorFinish() shouldCancel=0
      [2017-02-06T19:14:25.279-0600] [LWP:20705] DEBUG qdisp.Executive (core/modules/qdisp/Executive.cc:377) - Executive reaped requester for QI=1943:33;
      [2017-02-06T19:14:26.692-0600] [LWP:20631] DEBUG qdisp.QueryRequest (core/modules/qdisp/QueryRequest.cc:324) - QI=1943:33; QueryRequest::_errorFinish ok
      [2017-02-06T19:14:26.713-0600] [LWP:20631] DEBUG qdisp.Executive (core/modules/qdisp/Executive.cc:215) - Executive::markCompleted QI=1943:33; 0
      [2017-02-06T19:14:26.724-0600] [LWP:20631] DEBUG qdisp.Executive (core/modules/qdisp/Executive.cc:225) - Executive::markCompleted failed to find tracked QI=1943:33; size=1077
      [2017-02-06T19:14:26.748-0600] [LWP:20631] ERROR lsst.qserv (core/modules/global/Bug.cc:49) - Bug::Bug:Executive::markCompleted failed to find tracked QI=1943:33; size=1077
        what():  Executive::markCompleted failed to find tracked QI=1943:33; size=1077
      

        Attachments

          Activity

          Hide
          jgates John Gates added a comment - - edited

          The error( indicated by InfileMerger::merge(..., hasErMsg=1) ) caused _reapRequester to run, which removed the entry for QI=1943:33; from the _incompleteJobs map. When markCompleted could not locate the job in _incompleteJobs map, it threw a Bug exception.

          It looks like _reapRequester may be unnecessary, but there could be problems with the job retry logic.

          Show
          jgates John Gates added a comment - - edited The error( indicated by InfileMerger::merge(..., hasErMsg=1) ) caused _reapRequester to run, which removed the entry for QI=1943:33; from the _incompleteJobs map. When markCompleted could not locate the job in _incompleteJobs map, it threw a Bug exception. It looks like _reapRequester may be unnecessary, but there could be problems with the job retry logic.
          Hide
          jgates John Gates added a comment -

          The job retry logic appears sound as long as _reapRequesters is removed. Jobs should only be removed from the _incompleteJobs map by calls to markCompleted. markCompleted is only called if the job succeed or all retries failed.

          Show
          jgates John Gates added a comment - The job retry logic appears sound as long as _reapRequesters is removed. Jobs should only be removed from the _incompleteJobs map by calls to markCompleted. markCompleted is only called if the job succeed or all retries failed.
          Hide
          gapon Igor Gaponenko added a comment -

          I've reviewed the code, and the changes make a sense to me.

          Show
          gapon Igor Gaponenko added a comment - I've reviewed the code, and the changes make a sense to me.

            People

            Assignee:
            jgates John Gates
            Reporter:
            jgates John Gates
            Reviewers:
            Fabrice Jammes, Igor Gaponenko
            Watchers:
            Fabrice Jammes, Igor Gaponenko, John Gates
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: