Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: resources
-
Team:External
-
Urgent?:No
Description
Some scalability tests against dCache (webDAV) showed that better handling of timeouts and retries were necessary as well as improvements in the management of persistent HTTP connexions to avoid overwhelming the servers when thousands of clients execute simultaneously.
In this issue I propose modifications to improve the situation and some refactoring for making the code simpler when possible including renaming of environment variables previously named LSST_BUTLER_*.
Attachments
Issue Links
- relates to
-
- Done
Activity
Thanks for doing this. I have mostly minor comments. The one big comment is that I'm concerned that you are relying on a process only using one URL root per process since it seems that a per-class session cache is being used for all HTTP resource path instances even if they have differing base URLs.
Thanks for your feedback, Tim. You are right. The situation is now improved. I commented directly on github on how I fixed this issue. There are still some further improvements that I think could be addressed in a separate ticket.
The mypy github action is failing and I don't quite see how to fix it. My impressions is that that failure is not specific to this pull request, but I don't know enough of the system to be certain.
In the mypy action change the git+git to git+https. I have the same fix on my branch but you will merge first.
I merged branch tickets/DM-33769 of resources. Branch tickets/DM-33769 of daf_butler can also be merged, but before merging I want to make sure that you reviewed that one too.
I think the butler one needs to merge for everything to work so I’m happy for you to merge that as well.
Thanks ! Merged. I am closing this issue.
Fabio Hernandez will you need to backport all this to v23?
For the needs of DP0.2 processing at FrDF we have patched the Singularity image of v23.0.1.rc4 to include the relevant modifications and that is the image we are currently using.
If there are plans to issue another release in the v23 series I would be happy to backport this modification to avoid us local patching. Otherwise, we can live with our patch for the time being. So, from my perspective, it would not be necessary to issue another v23 release which would only include this back port.
Yusra AlSayyad is planning to do a v23.0.2 for step 4 so backporting is going to be needed if you want to avoid local patching in the future. Add the backport requested tag if you decide that backporting is something you want to do.
We have approved this for backporting if you want to do it. We think that v23.0.2rc1 will be made around the end of March.
Thank you. I think I am done with the bulk of the back port work. I will push tomorrow.
This back port is ready for review.
Pull request submitted: https://github.com/lsst/daf_butler/pull/666
Pull request merged, thanks a lot for your help.
I am closing this issue.
I think I am done with the modifications and they are ready to review. Your comments are welcome.