Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33853

Consider adding logging when reading in many datasets

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      In DM-33820 it seems that we are getting a situation on the IDF where hundreds of input datasets are being read by ButlerQuantumContext.get() and it taking longer than two hours to read them all from the object store.

      Granted that this probably means the algorithm should be rewritten to use deferred gets, but it does suggest that we should consider a heartbeat verbose log message where every ten minutes or so we report "Read N out of M requested datasets". Presumably there is a risk that put() also needs this.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I've added the logging for .get(). I'm inclined to not add more logging until we come across a situation where things are taking a long time and a log message every 10 minutes or so would help people.

            Show
            tjenness Tim Jenness added a comment - I've added the logging for .get(). I'm inclined to not add more logging until we come across a situation where things are taking a long time and a log message every 10 minutes or so would help people.
            Hide
            jbosch Jim Bosch added a comment -

            Looks good; sad that it's necessary, but I concede that it is. Only comment is a very minor style thing.

            Show
            jbosch Jim Bosch added a comment - Looks good; sad that it's necessary, but I concede that it is. Only comment is a very minor style thing.
            Hide
            tjenness Tim Jenness added a comment -

            I won't merge until DM-32250 comes in (just in case I messed up the type annotations)

            Show
            tjenness Tim Jenness added a comment - I won't merge until DM-32250 comes in (just in case I messed up the type annotations)

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.