Fix Version/s: None
The internal parallelization in these obs_base tasks ranges from non-functional to not useful. It's time to just remove it.
We decided on Slack that the parallelization in raw ingest is still a reasonable thing to do since it parallelizes file metadata extraction and does not parallelize the butler registry side of things.
Ok, done, with parallelization removed from DefineVisitsTask and its scripts only. I did the absolute minimal things to update gen2to3 and ap_verify, just not passing "processes" or "pool" on to DefineVisitsTask.run. Krzysztof Findeisen , I'd be happy to do more in ap_verify if you'd like, but I think what I've done matches your recommendation.
Looks okay although I think the coding standard prefers we don't use map().
I've removed the map usage, and that led to some nice other cleanups. I also added a trivial PR for ci_builder (machinery behind ci_imsim), following a Jenkins failure:
I plan to merge after Jenkins is green.
Note that parallelization is used in ap_verify ingestion:
Since the parallelization in ap_verify goes all the way up to the shared CLI between ap_verify.py and ingest_dataset.py, I'd recommend just removing it from these two run calls instead of trying to make everything self-consistent. The ap_verify CLI reference does not explicitly say that ingestion is parallelized, after all...