Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33853

Consider adding logging when reading in many datasets

    XMLWordPrintable

Details

    • 1
    • Architecture
    • No

    Description

      In DM-33820 it seems that we are getting a situation on the IDF where hundreds of input datasets are being read by ButlerQuantumContext.get() and it taking longer than two hours to read them all from the object store.

      Granted that this probably means the algorithm should be rewritten to use deferred gets, but it does suggest that we should consider a heartbeat verbose log message where every ten minutes or so we report "Read N out of M requested datasets". Presumably there is a risk that put() also needs this.

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            I've added the logging for .get(). I'm inclined to not add more logging until we come across a situation where things are taking a long time and a log message every 10 minutes or so would help people.

            tjenness Tim Jenness added a comment - I've added the logging for .get(). I'm inclined to not add more logging until we come across a situation where things are taking a long time and a log message every 10 minutes or so would help people.
            jbosch Jim Bosch added a comment -

            Looks good; sad that it's necessary, but I concede that it is. Only comment is a very minor style thing.

            jbosch Jim Bosch added a comment - Looks good; sad that it's necessary, but I concede that it is. Only comment is a very minor style thing.
            tjenness Tim Jenness added a comment -

            I won't merge until DM-32250 comes in (just in case I messed up the type annotations)

            tjenness Tim Jenness added a comment - I won't merge until DM-32250 comes in (just in case I messed up the type annotations)

            People

              tjenness Tim Jenness
              tjenness Tim Jenness
              Jim Bosch
              Jim Bosch, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.