Jim Bosch added a comment - 05/Mar/21 3:16 AM Here's a basic version of what we talked about at the meeting, ready for review. I've asked Tim Jenness to be the official reviewer, but I'd like to hear from Michelle Gower and Steve Pietrowicz on whether this is helpful/neutral/harmful for DBBBM and OODS (I'm guessing "neutral" and "helpful", respectively). There may also be ways to replace existing logging (e.g. failure logging) by using these callbacks, but I'm ambivalent on whether that would actually be better, and more importantly I'd like to keep this ticket to pure additions and no subtractions so I don't have to worry about breaking anything on it.

Jim Bosch added a comment - 05/Mar/21 3:44 AM - edited Here's (roughly) how I'm using it in my driver scripts, as an example: ingested = [] task = RawIngestTask(config, butler=butler, on_success=ingested.extend) # Open a file that, if present, means ingest may have died without # reliably reporting its progress, and human intervention may be # needed just to understand where it left off. file = open(self._in_progress_filename, "wt") try: file.flush() # Actual run ingest. task.run(todo) finally: # Record what we successfully ingested, regardless of # whether there were some failures, or even a ctrl+c interrupt. file.writelines(line + "\n" for line in sorted(ingested)) file.close() # If all that worked, rename the file to indicate that it's reliable. os.replace(self._in_progress_filename, self._completed_filename)

Tim Jenness added a comment - 05/Mar/21 5:51 PM Looks great. One minor thing needs fixing I think. Also we need to think whether this replaces failFast in the future.