Fix Version/s: None
Sprint:AP S22-5 (April)
DM-27117 created a pipeline ($AP_VERIFY_DIR/tests/MockApPipe.yaml) that does no processing but inputs and outputs mock datasets, typically the smallest objects that can pass their types' validity checks. Running this pipeline on the one image in ap_verify_testdata takes 20 seconds; more precisely, this is the wall-clock time spent calling pipetask run from within single-process pytest.
To better pin down the run time and how much of it comes from Persistable/Click/Middleware, run MockApPipe.yaml from the command line and examine the results with a profiler. It should be possible to set up a compatible test repository using ingest_dataset.py --dataset ap_verify_testdata; that's essentially what the new
DM-27117 tests do.
- is blocked by
DM-27117 Add dummy pipeline to ap_verify testing
- links to
The profiles are now readable; thanks.
I wonder if the difference between last week and this was how much standard HSC/DC2 DRP reprocessing was going on (by affecting overall GPFS load).
I'd still be concerned about reading FITS files, because at least in some cases I think a significant part of the time is spent unpacking the data into afw objects, not in the actual reading from disk.
You can turn on in memory datastore if you want and see what happens. You will have to configure the datastore as a chained datastore and probably tell the file datastore to only accept dataset types that you really need to write out.
Some follow-up possibilities were discussed at the AP group meeting on Monday; attaching link for reference.
Jim Bosch, I think I found why the profiles got corrupted. I've re-uploaded all of them; let me know if they work.