I've generated a profile and am sifting through the results.

I've attached profiles for each detector, as well as a profile which combines them all. These are from doing the same run as described in DM-31652, but with `--profile` each for each detector.

Here's the full snakeviz for detector 55.

And here's just `imageDifference.py`'s subtree:

Total timings per CCD (we need "time" to be <60s here) :

{{
ccd time peaks sources template coverage
----- ------- ------- --------- -------------------
24 208.685 3964 85 73.5%
25 222.415 3352 145 100.0%
26 220.112 3043 109 100.0%
54 206.47 4012 104 64.8%
55 213.181 3495 112 100.0%
56 210.468 3346 98 100.0%
57 208.791 3751 75 56.4%
58 227.843 3884 81 100.0%
59 228.365 3433 139 100.0%
60 219.037 4022 44 47.8%
61 232.226 3958 77 100.0%
62 216.831 3872 83 100.0%
63 216.16 3508 145 100.0%
64 209.726 3360 131 100.0%
65 230.07 3660 139 100.0%
66 219.15 3485 119 100.0%
67 220.535 3256 151 100.0%
68 221.017 3494 120 100.0%
69 216.132 3329 65 100.0%
70 225.792 3702 107 100.0%
71 228.86 3592 111 100.0%
99 200.749 3865 20 39.1%
100 214.815 3939 78 100.0%
101 218.667 4108 133 100.0%
102 198.854 4267 20 27.7%
103 229.007 4029 70 82.1%
104 207.243 3575 95 73.4%
108 220.909 3803 87 100.0%
109 208.137 4008 79 100.0%
110 214.698 3812 175 99.8%
111 192.182 3580 80 64.4%
112 198.977 3753 37 55.7%
113 198.513 3486 59 47.0%}}

Could you please post the scripts you used to generate this output? I'll be following up on this with some attempts at trimming some of that time.

John Parejko Sure, here they are: https://gist.github.com/spenczar/fbd3e8cd4a1d3a748ebbe7e59b3cd6d2

The first file in the gist is what I did to run each CCD sequentially, generating a profile for each. These are the CCDs for exposure 254379.

The second file is the pipeline.yaml I used.

The third file is a big gnarly script for pulling timings out of log files, which I used for that "total timings" table.

https://github.com/lsst-dm/ap_pipe-notebooks/blob/main/notebooks/DM-31652-process_one_exposure_of_DC2.ipynb shows what I did to make templates ahead of time.