Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33854

Need periodic log messages for forcedPhotCoadd during aperture corrections

    XMLWordPrintable

Details

    • 3
    • Data Release Production
    • No

    Description

      The PanDA batch system needs tasks to write periodic log messages at least every 2 hours, to ensure that the task is still running and not in some stuck state.  forcedPhotCoadd already does write VERBOSE log messages every 10 minutes, but stops doing this once it starts applying aperture corrections.  Because the aperture corrections can take several hours, the lack of log messages can lead to PanDA killing the job, requiring additional re-attempts that may eventually fail again for the same reason.

      So additional periodic logging, at VERBOSE level, are needed during aperture correction and any subsequent steps.

      Attachments

        Issue Links

          Activity

            The looping_limit_default timeout in panda has been raised from 2 hours to 20 hours for the moment, so we are able to run successfully, but once this fix and DM-33820 are merged we should reset the limit back to its normal 2 hours.

            ctslater Colin Slater added a comment - The looping_limit_default timeout in panda has been raised from 2 hours to 20 hours for the moment, so we are able to run successfully, but once this fix and DM-33820 are merged we should reset the limit back to its normal 2 hours.

            The fix is ready to be reviewed. The ci_hsc datasets are too small to trigger the heartbeat logs.

            Jenkins build: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36014/artifacts

            kannawad Arun Kannawadi added a comment - The fix is ready to be reviewed. The ci_hsc datasets are too small to trigger the heartbeat logs. Jenkins build:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36014/artifacts

            Thanks for the review, Tim

            kannawad Arun Kannawadi added a comment - Thanks for the review, Tim

            Merged into the `main` branch. Waiting for `backport-approved` tag from DM-CCB.

            kannawad Arun Kannawadi added a comment - Merged into the `main` branch. Waiting for `backport-approved` tag from DM-CCB.
            tjenness Tim Jenness added a comment -

            Backport is approved.

            tjenness Tim Jenness added a comment - Backport is approved.
            kannawad Arun Kannawadi added a comment - Successful Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36041/pipeline

            People

              kannawad Arun Kannawadi
              hlin Huan Lin
              Tim Jenness
              Arun Kannawadi, Colin Slater, Huan Lin, Shuwei Ye, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.