Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33769

Improve timeout, retry and connexion persistence of webDAV client

    XMLWordPrintable

    Details

    • Team:
      External
    • Urgent?:
      No

      Description

      Some scalability tests against dCache (webDAV) showed that better handling of timeouts and retries were necessary as well as improvements in the management of persistent HTTP connexions to avoid overwhelming the servers when thousands of clients execute simultaneously.

      In this issue I propose modifications to improve the situation and some refactoring for making the code simpler when possible including renaming of environment variables previously named LSST_BUTLER_*.

        Attachments

          Issue Links

            Activity

            No builds found.
            FabioHernandez Fabio Hernandez created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Team External [ 12117 ]
            FabioHernandez Fabio Hernandez made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            I think I am done with the modifications and they are ready to review. Your comments are welcome.

            Show
            FabioHernandez Fabio Hernandez added a comment - I think I am done with the modifications and they are ready to review. Your comments are welcome.
            FabioHernandez Fabio Hernandez made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            tjenness Tim Jenness added a comment -

            Thanks for doing this. I have mostly minor comments. The one big comment is that I'm concerned that you are relying on a process only using one URL root per process since it seems that a per-class session cache is being used for all HTTP resource path instances even if they have differing base URLs.

            Show
            tjenness Tim Jenness added a comment - Thanks for doing this. I have mostly minor comments. The one big comment is that I'm concerned that you are relying on a process only using one URL root per process since it seems that a per-class session cache is being used for all HTTP resource path instances even if they have differing base URLs.
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            FabioHernandez Fabio Hernandez added a comment - - edited

            Thanks for your feedback, Tim. You are right. The situation is now improved. I commented directly on github on how I fixed this issue. There are still some further improvements that I think could be addressed in a separate ticket.

            Show
            FabioHernandez Fabio Hernandez added a comment - - edited Thanks for your feedback, Tim. You are right. The situation is now improved. I commented directly on github on how I fixed this issue. There are still some further improvements that I think could be addressed in a separate ticket.
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            The mypy github action is failing and I don't quite see how to fix it. My impressions is that that failure is not specific to this pull request, but I don't know enough of the system to be certain.

            Show
            FabioHernandez Fabio Hernandez added a comment - The mypy github action is failing and I don't quite see how to fix it. My impressions is that that failure is not specific to this pull request, but I don't know enough of the system to be certain.
            Hide
            tjenness Tim Jenness added a comment -

            In the mypy action change the git+git to git+https. I have the same fix on my branch but you will merge first.

            Show
            tjenness Tim Jenness added a comment - In the mypy action change the git+git to git+https. I have the same fix on my branch but you will merge first.
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            I merged branch tickets/DM-33769 of resources. Branch tickets/DM-33769 of daf_butler  can also be merged, but before merging I want to make sure that you reviewed that one too.

            Show
            FabioHernandez Fabio Hernandez added a comment - I merged branch tickets/ DM-33769 of resources . Branch tickets/DM-33769 of daf_butler   can also be merged, but before merging I want to make sure that you reviewed that one too.
            Hide
            tjenness Tim Jenness added a comment -

            I think the butler one needs to merge for everything to work so I’m happy for you to merge that as well.

            Show
            tjenness Tim Jenness added a comment - I think the butler one needs to merge for everything to work so I’m happy for you to merge that as well.
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            Thanks ! Merged. I am closing this issue.

            Show
            FabioHernandez Fabio Hernandez added a comment - Thanks ! Merged. I am closing this issue.
            FabioHernandez Fabio Hernandez made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            tjenness Tim Jenness added a comment -

            Fabio Hernandez will you need to backport all this to v23?

            Show
            tjenness Tim Jenness added a comment - Fabio Hernandez will you need to backport all this to v23?
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            For the needs of DP0.2 processing at FrDF we have patched the Singularity image of v23.0.1.rc4 to include the relevant modifications and that is the image we are currently using.

            If there are plans to issue another release in the v23 series I would be happy to backport this modification to avoid us local patching. Otherwise, we can live with our patch for the time being. So, from my perspective, it would not be necessary to issue another v23 release which would only include this back port.

            Show
            FabioHernandez Fabio Hernandez added a comment - For the needs of DP0.2 processing at FrDF we have patched the Singularity image of v23.0.1.rc4 to include the relevant modifications and that is the image we are currently using. If there are plans to issue another release in the v23 series I would be happy to backport this modification to avoid us local patching. Otherwise, we can live with our patch for the time being. So, from my perspective, it would not be necessary to issue another v23 release which would only include this back port.
            Hide
            tjenness Tim Jenness added a comment -

            Yusra AlSayyad is planning to do a v23.0.2 for step 4 so backporting is going to be needed if you want to avoid local patching in the future. Add the backport requested tag if you decide that backporting is something you want to do.

            Show
            tjenness Tim Jenness added a comment - Yusra AlSayyad is planning to do a v23.0.2 for step 4 so backporting is going to be needed if you want to avoid local patching in the future. Add the backport requested tag if you decide that backporting is something you want to do.
            tjenness Tim Jenness made changes -
            Labels gen3-middleware backport-approved backport-v23 gen3-middleware
            Hide
            tjenness Tim Jenness added a comment -

            We have approved this for backporting if you want to do it. We think that v23.0.2rc1 will be made around the end of March.

            Show
            tjenness Tim Jenness added a comment - We have approved this for backporting if you want to do it. We think that v23.0.2rc1 will be made around the end of March.
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            Thank you. I think I am done with the bulk of the back port work. I will push tomorrow.

            Show
            FabioHernandez Fabio Hernandez added a comment - Thank you. I think I am done with the bulk of the back port work. I will push tomorrow.
            FabioHernandez Fabio Hernandez made changes -
            Resolution Done [ 10000 ]
            Status Done [ 10002 ] To Do [ 10001 ]
            FabioHernandez Fabio Hernandez made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            This back port is ready for review.
            Pull request submitted: https://github.com/lsst/daf_butler/pull/666

            Show
            FabioHernandez Fabio Hernandez added a comment - This back port is ready for review. Pull request submitted: https://github.com/lsst/daf_butler/pull/666
            FabioHernandez Fabio Hernandez made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-34169 [ DM-34169 ]
            Hide
            FabioHernandez Fabio Hernandez added a comment -

            Pull request merged, thanks a lot for your help.

            I am closing this issue.

            Show
            FabioHernandez Fabio Hernandez added a comment - Pull request merged, thanks a lot for your help. I am closing this issue.
            FabioHernandez Fabio Hernandez made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            FabioHernandez Fabio Hernandez made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Labels backport-approved backport-v23 gen3-middleware backport-approved backport-done backport-v23 gen3-middleware

              People

              Assignee:
              FabioHernandez Fabio Hernandez
              Reporter:
              FabioHernandez Fabio Hernandez
              Reviewers:
              Tim Jenness
              Watchers:
              Fabio Hernandez, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.