I found 2003 SS69 in the COSMOS field! At least, I have a bright source that's moving in the right direction at just under the right speed (expected 2.5" trail (30.58"/h over 300 s), got 2.1") and doesn't show up in DSS/SDSS. It's 6-8 arcsec away from where it should be, consistent with the ephemeris being generated 14 minutes too late.

Final update for tonight: the detection statistics with DM-31934 are:

HiTS

Image 419802-10: 1/2 detected		2007 TC479, possible faint source

					2017 OY31, off-center

Image 419802-5: 0/1 detected		2013 TZ136, no obvious source

Image 411371-56: 1/1 detected		2007 KX1, off-center

Image 411371-60: 0/0 detected

Image 411420-10: 0/0 detected

Image 411420-5: 1/1 detected		2003 HH10, off-center

COSMOS

Image 59160-51: 0/1 detected		2003 SS69, far off-center

Image 59150-50: 0/0 detected

All the asteroids that I can identify in cutouts (now up to 200×200) are "behind" the predicted position and proper motion vector, so I think the next step is to look for a systematic error in the epoch.

I've got it! The problem is parallax: by default, SkyBotEphemerisQuery computes ephemerides for Cerro Pachon. This gives a modest error for DECam data (Cerro Tololo) and a huge one for HSC data (Mauna Kea). So the solution is to use a custom config before we call SkyBotEphemerisQuery when maintaining the ap_verify datasets (geocentric observer isn't good enough; that's Obs=500 in the attached image). It's not a bug in the task itself.

FTR, the final results are:

HiTS

Image 419802-10: 1/2 detected		2007 TC479, not detected

					2017 OY31

Image 419802-5: 0/0 detected

Image 411371-56: 1/1 detected		2007 KX1

Image 411371-60: 0/0 detected

Image 411420-10: 0/0 detected

Image 411420-5: 1/1 detected		2003 HH10

COSMOS

Image 59160-51: 1/1 detected		2003 SS69

Image 59150-50: 0/0 detected

It's not clear why 2007 TC479 is missing, since it's not supposed to be particularly faint.

There's still a systematic offset between the predicted and observed positions that's smaller than the 2" matching radius. Fixing that has been deferred to DM-34906 (possibly with a larger data set).

Thanks for agreeing to do this, Ari Heinze, and sorry for the mix-up!

A brief summary, since I understand you're still getting oriented with the Stack:

• ap_association#150 has the code changes done to fix this issue. It's about 80 lines between the bugfixes and the unit tests.
• ap_pipe-notebooks#70 is a scratch notebook used for debugging. You can see a render of the notebook if you select "View File" from the upper right corner of the diff display. Notebook reviews tend to be fairly informal, since it's hard to make line comments and they aren't held to the DM coding standards anyway.
• obs_subaru#419 and obs_decam#222 add centralized config files that are automatically detected and loaded by the pipeline framework (which is why you won't find them explicitly mentioned in the code or pipelines). The task being configured is one of the two being fixed in ap_association.
• ap_verify_ci_hits2015#40, ap_verify_hits2015#45, and ap_verify_ci_cosmos_pdr2#29 are special, self-contained datasets for ap_verify. The primary change is to the input data (which needed to have their coordinates fixed), but the supporting pipelines are updated so that they actually use the configs added to obs_subaru or obs_decam, as appropriate.

For ap_association, there are a large number of unrelated but partially overlapping changes, so I recommend looking at the changes one commit at a time to keep them straight.