Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16519

posting metrics to squash api is extremely slow

    XMLWordPrintable

    Details

      Description

      Uploading metric results via dispatch_verify.py to squash has been taking ~6 minutes, per invocation, in production.

      Eg., a 23m 59s for 3 invocations:

      https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Fvalidate_drp/detail/validate_drp/1465/pipeline

      This is extremely painful for a job like ap_verify which normally computes results in ~10minutes but produces 6 metric files.

        Attachments

          Issue Links

          has to be done before

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-16017 Prototype a metrics-handling Task

          • Done
          is triggered by

          Improvement - An improvement or enhancement to an existing feature or task. DM-16333 push ap_verify results to squash

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-16835 scipipe/ap_verify job timing out

          • Done

            Activity

            Ascending order - Click to sort in descending order
            6 older comments
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Yeah that solved one of the problems. The code that uploads the data blobs and register the URIs will need more refactoring though. Working on it.

            Show
            afausti Angelo Fausti added a comment - Yeah that solved one of the problems. The code that uploads the data blobs and register the URIs will need more refactoring though. Working on it.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            I've tested the above fix in a test deployment of SQuaSH with a clone of the current production database. It improves the POST /job by a factor of ~10 already i.e from 10m54s to 58s for the upload of the a test job on the validation_data_cfht dataset, a smaller fraction of the pipeline runtime.

            I'll update production with this fix now and check the POST /job for the other datasets, and specially for the ap_verify pipeline from tonight's run.

            Show
            afausti Angelo Fausti added a comment - I've tested the above fix in a test deployment of SQuaSH with a clone of the current production database. It improves the POST /job by a factor of ~10 already i.e from 10m54s to 58s for the upload of the a test job on the validation_data_cfht dataset, a smaller fraction of the pipeline runtime. I'll update production with this fix now and check the POST /job for the other datasets, and specially for the ap_verify pipeline from tonight's run.
            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            Hey Joshua Hoblitt that silly relationship made all the difference, specially for ap_verify jobs that do not contain data blobs:

            https://ci.lsst.codes/blue/organizations/jenkins/scipipe%2Fap_verify/detail/ap_verify/78/pipeline

            I'm claiming victory.

            Here is the PR for a super quick review: https://github.com/lsst-sqre/squash-restful-api/pull/41

            Show
            afausti Angelo Fausti added a comment - - edited Hey Joshua Hoblitt that silly relationship made all the difference, specially for ap_verify jobs that do not contain data blobs: https://ci.lsst.codes/blue/organizations/jenkins/scipipe%2Fap_verify/detail/ap_verify/78/pipeline I'm claiming victory. Here is the PR for a super quick review: https://github.com/lsst-sqre/squash-restful-api/pull/41
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            PR appoved on GH with a suggestion for possible method of DRYing up the code.

            Show
            jhoblitt Joshua Hoblitt added a comment - PR appoved on GH with a suggestion for possible method of DRYing up the code.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Addressed review suggestions and merged.

            Show
            afausti Angelo Fausti added a comment - Addressed review suggestions and merged.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Angelo Fausti, Jonathan Sick, Joshua Hoblitt, Krzysztof Findeisen, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.