Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13394

Increase default http timeout in lsst.verify

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Default lsst.verify http timeout for SQuaSH API requests was set initially to 30s, a larger timeout is required to upload verification jobs of 500M-1G. It will be set to 900s.

      Also, allow 202 (Accepted) status code for these long requests.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Also reviewed the nginx and uwsgi configuration as an attempt to avoid this time out error:

             
            File "/Users/afausti/Projects/lsstsw/lsstsw/stack/DarwinX86/requests/2.9.1.lsst1+2/lib/python/requests-2.9.1-py3.6.egg/requests/models.py", line 840, in raise_for_status
                raise HTTPError(http_error_msg, response=self)
            requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://squash-restful-api-demo.lsst.codes/job
            

            see the current solution at:

            https://github.com/lsst-sqre/squash-rest-api/blob/master/uwsgi.ini

            and

            https://github.com/lsst-sqre/squash-rest-api/blob/master/kubernetes/nginx/nginx.conf

            It is quite impressive the difference in memory usage and request time uploading a ~400M file if one configures nginx with proxy_pass which uses HTTP protocol (see the higher peaks in the plot) compared to uwsgi_pass which uses WSGI protocol (see the last two peaks in the plot)

            See also the discussion here:
            https://stackoverflow.com/questions/34562730/difference-between-uwsgi-pass-and-proxy-pass-in-nginx

            Show
            afausti Angelo Fausti added a comment - - edited Also reviewed the nginx and uwsgi configuration as an attempt to avoid this time out error:   File "/Users/afausti/Projects/lsstsw/lsstsw/stack/DarwinX86/requests/2.9.1.lsst1+2/lib/python/requests-2.9.1-py3.6.egg/requests/models.py" , line 840 , in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https: //squash-restful-api-demo.lsst.codes/job see the current solution at: https://github.com/lsst-sqre/squash-rest-api/blob/master/uwsgi.ini and https://github.com/lsst-sqre/squash-rest-api/blob/master/kubernetes/nginx/nginx.conf It is quite impressive the difference in memory usage and request time uploading a ~400M file if one configures nginx with proxy_pass which uses HTTP protocol (see the higher peaks in the plot) compared to uwsgi_pass which uses WSGI protocol (see the last two peaks in the plot) See also the discussion here: https://stackoverflow.com/questions/34562730/difference-between-uwsgi-pass-and-proxy-pass-in-nginx
            Show
            afausti Angelo Fausti added a comment - - edited Branch passing in Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27439/pipeline
            Hide
            afausti Angelo Fausti added a comment -
            Show
            afausti Angelo Fausti added a comment - Merged PR https://github.com/lsst/verify/pull/23

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel