# Provide a way for RawIngestTask to report successes and failures programmatically

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

I have a DM-28636 branch that does this by appending to a list, but on the middleware call we realized that a callback or two would be much better. I'll put that here.

#### Activity

Hide
Jim Bosch added a comment -

Here's a basic version of what we talked about at the meeting, ready for review. I've asked Tim Jenness to be the official reviewer, but I'd like to hear from Michelle Gower and Steve Pietrowicz on whether this is helpful/neutral/harmful for DBBBM and OODS (I'm guessing "neutral" and "helpful", respectively).

There may also be ways to replace existing logging (e.g. failure logging) by using these callbacks, but I'm ambivalent on whether that would actually be better, and more importantly I'd like to keep this ticket to pure additions and no subtractions so I don't have to worry about breaking anything on it.

Show
Jim Bosch added a comment - Here's a basic version of what we talked about at the meeting, ready for review. I've asked Tim Jenness to be the official reviewer, but I'd like to hear from Michelle Gower and Steve Pietrowicz on whether this is helpful/neutral/harmful for DBBBM and OODS (I'm guessing "neutral" and "helpful", respectively). There may also be ways to replace existing logging (e.g. failure logging) by using these callbacks, but I'm ambivalent on whether that would actually be better, and more importantly I'd like to keep this ticket to pure additions and no subtractions so I don't have to worry about breaking anything on it.
Hide
Jim Bosch added a comment - - edited

Here's (roughly) how I'm using it in my driver scripts, as an example:

 ingested = [] task = RawIngestTask(config, butler=butler, on_success=ingested.extend) # Open a file that, if present, means ingest may have died without # reliably reporting its progress, and human intervention may be # needed just to understand where it left off. file = open(self._in_progress_filename, "wt") try:  file.flush()  # Actual run ingest.  task.run(todo) finally:  # Record what we successfully ingested, regardless of  # whether there were some failures, or even a ctrl+c interrupt.  file.writelines(line + "\n" for line in sorted(ingested))  file.close()  # If all that worked, rename the file to indicate that it's reliable.  os.replace(self._in_progress_filename, self._completed_filename) 

Show
Jim Bosch added a comment - - edited Here's (roughly) how I'm using it in my driver scripts, as an example: ingested = [] task = RawIngestTask(config, butler=butler, on_success=ingested.extend) # Open a file that, if present, means ingest may have died without # reliably reporting its progress, and human intervention may be # needed just to understand where it left off. file = open(self._in_progress_filename, "wt") try: file.flush() # Actual run ingest. task.run(todo) finally: # Record what we successfully ingested, regardless of # whether there were some failures, or even a ctrl+c interrupt. file.writelines(line + "\n" for line in sorted(ingested)) file.close() # If all that worked, rename the file to indicate that it's reliable. os.replace(self._in_progress_filename, self._completed_filename)
Hide
Tim Jenness added a comment -

Looks great. One minor thing needs fixing I think. Also we need to think whether this replaces failFast in the future.

Show
Tim Jenness added a comment - Looks great. One minor thing needs fixing I think. Also we need to think whether this replaces failFast in the future.

#### People

Assignee:
Jim Bosch
Reporter:
Jim Bosch
Reviewers:
Tim Jenness
Watchers:
Jim Bosch, Michelle Gower, Steve Pietrowicz, Tim Jenness