Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29071

Provide a way for RawIngestTask to report successes and failures programmatically

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      I have a DM-28636 branch that does this by appending to a list, but on the middleware call we realized that a callback or two would be much better. I'll put that here.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Here's a basic version of what we talked about at the meeting, ready for review. I've asked Tim Jenness to be the official reviewer, but I'd like to hear from Michelle Gower and Steve Pietrowicz on whether this is helpful/neutral/harmful for DBBBM and OODS (I'm guessing "neutral" and "helpful", respectively).

          There may also be ways to replace existing logging (e.g. failure logging) by using these callbacks, but I'm ambivalent on whether that would actually be better, and more importantly I'd like to keep this ticket to pure additions and no subtractions so I don't have to worry about breaking anything on it.

          Show
          jbosch Jim Bosch added a comment - Here's a basic version of what we talked about at the meeting, ready for review. I've asked Tim Jenness to be the official reviewer, but I'd like to hear from Michelle Gower and Steve Pietrowicz on whether this is helpful/neutral/harmful for DBBBM and OODS (I'm guessing "neutral" and "helpful", respectively). There may also be ways to replace existing logging (e.g. failure logging) by using these callbacks, but I'm ambivalent on whether that would actually be better, and more importantly I'd like to keep this ticket to pure additions and no subtractions so I don't have to worry about breaking anything on it.
          Hide
          jbosch Jim Bosch added a comment - - edited

          Here's (roughly) how I'm using it in my driver scripts, as an example:

          ingested = []
          task = RawIngestTask(config, butler=butler, on_success=ingested.extend)
          # Open a file that, if present, means ingest may have died without
          # reliably reporting its progress, and human intervention may be
          # needed just to understand where it left off.
          file = open(self._in_progress_filename, "wt")
          try:
              file.flush()
              # Actual run ingest.
              task.run(todo)
          finally:
              # Record what we successfully ingested, regardless of
              # whether there were some failures, or even a ctrl+c interrupt.
              file.writelines(line + "\n" for line in sorted(ingested))
              file.close()
              # If all that worked, rename the file to indicate that it's reliable.
              os.replace(self._in_progress_filename, self._completed_filename)
          

          Show
          jbosch Jim Bosch added a comment - - edited Here's (roughly) how I'm using it in my driver scripts, as an example: ingested = [] task = RawIngestTask(config, butler=butler, on_success=ingested.extend) # Open a file that, if present, means ingest may have died without # reliably reporting its progress, and human intervention may be # needed just to understand where it left off. file = open(self._in_progress_filename, "wt") try: file.flush() # Actual run ingest. task.run(todo) finally: # Record what we successfully ingested, regardless of # whether there were some failures, or even a ctrl+c interrupt. file.writelines(line + "\n" for line in sorted(ingested)) file.close() # If all that worked, rename the file to indicate that it's reliable. os.replace(self._in_progress_filename, self._completed_filename)
          Hide
          tjenness Tim Jenness added a comment -

          Looks great. One minor thing needs fixing I think. Also we need to think whether this replaces failFast in the future.

          Show
          tjenness Tim Jenness added a comment - Looks great. One minor thing needs fixing I think. Also we need to think whether this replaces failFast in the future.

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            Tim Jenness
            Watchers:
            Jim Bosch, Michelle Gower, Steve Pietrowicz, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.