Profiling is done, and the answer is very simple: all of the time is going into S3 existence checks inside Datastore.ingest. That's good news, because I bet that's something we can fix with a relatively small patch that could be backported to the v23 branch. I'll take a look at that next. Aside from the fact that we don't need to do existence checks here at all, it looks from the profile like we might be doing them all twice: once in _prepIngest / _standardizeIngestPath, and once in _finishIngest / _extractIngestInfo. Tim Jenness, feel free to ping me on Slack today (despite it being Friday) if you want to weigh in on how to approach this.
I've attached the profile as slow-eb.profile.
As an aside, I also took the opportunity to profile QG generation for this step (it took 30 minutes, which is ok only from the perspective that it's in the "expected" range due to known issues). But it turns out the database queries (what those known issues involve) are only 24% (7min), and the rest looks a lot like it's basically the kind of Python logic we could only speed up by moving things to a compiled language. I think it might be time to start thinking seriously about how we might approach that; I think the scope of what we'd want to move to a compiled extension module is basically everything in a QuantumGraph, which means DatasetRef, some kind of container of DatasetRef and everything that goes into that. The QG generation profile is also attached as slow-eb-qg.profile.