The five objects in the HiTS field are:

       # Num          Name        Class    Mv  Err(arcsec)    Dg(ua)    Dh(ua)  Phase(deg)         ssObjectId  visitId

77   227477    2005 WV185    MB>Middle   19.6        0.220  1.917858  2.869382        6.26  53569918513453402   411371

9         -    2015 DL180     MB>Outer   21.4        0.740  2.118523  3.069327        5.82  43707620656988304   411420

11    93419    2000 SS305    MB>Middle   20.1        0.226  2.151946  3.102871        5.74  44866635817748191   411420

12   503735     2016 LG24     MB>Outer   21.3        1.491  2.169487  3.120478        5.69  15168997349454932   411420

26   192000     2005 XW67    MB>Middle   19.1        0.173  1.583128  2.542704        6.93  26697415027480166   419802

The object in the COSMOS field is:

      # Num         Name       Class    Mv  Err(arcsec)    Dg(ua)    Dh(ua)  Phase(deg)        ssObjectId  visitId

7   276537    2003 SS69    MB>Outer   20.1        0.266  1.993244  2.949244        6.07  4362913742843557    59160

At first glance, all of these should be detectable and matchable.

We looked at this in pair coding with Eric Bellm, Kenneth Herner, and Clare Saunders. We didn't see an obvious calexp source for any of the three predicted objects in HiTS visit 411420, so we suspect that the bug is in SkyBotEphemerisQueryTask. It's not obvious at first glance how to verify its outputs, though.

I found a bug: _decdms2decdeg does not work correctly for negative angles (e.g., -06 14 41.102 gets converted to -5.755249).

However, this does not explain the COSMOS failure, since that one is at +2 degrees, and I checked the coordinate conversion by hand.

We should replace both of _decdms2decdeg and _rahms2radeg with calls to astropy.coordinates or similar. (Ah, this is already ticketed: DM-31934)

Pushed a notebook covering today's pair coding session here: https://github.com/lsst-dm/ap_pipe-notebooks/tree/tickets/DM-34693/notebooks

I found 2003 SS69 in the COSMOS field! At least, I have a bright source that's moving in the right direction at just under the right speed (expected 2.5" trail (30.58"/h over 300 s), got 2.1") and doesn't show up in DSS/SDSS. It's 6-8 arcsec away from where it should be, consistent with the ephemeris being generated 14 minutes too late.

Final update for tonight: the detection statistics with DM-31934 are:

HiTS

Image 419802-10: 1/2 detected		2007 TC479, possible faint source

					2017 OY31, off-center

Image 419802-5: 0/1 detected		2013 TZ136, no obvious source

Image 411371-56: 1/1 detected		2007 KX1, off-center

Image 411371-60: 0/0 detected

Image 411420-10: 0/0 detected

Image 411420-5: 1/1 detected		2003 HH10, off-center

COSMOS

Image 59160-51: 0/1 detected		2003 SS69, far off-center

Image 59150-50: 0/0 detected

All the asteroids that I can identify in cutouts (now up to 200×200) are "behind" the predicted position and proper motion vector, so I think the next step is to look for a systematic error in the epoch.

I've got it! The problem is parallax: by default, SkyBotEphemerisQuery computes ephemerides for Cerro Pachon. This gives a modest error for DECam data (Cerro Tololo) and a huge one for HSC data (Mauna Kea). So the solution is to use a custom config before we call SkyBotEphemerisQuery when maintaining the ap_verify datasets (geocentric observer isn't good enough; that's Obs=500 in the attached image). It's not a bug in the task itself.

FTR, the final results are:

HiTS

Image 419802-10: 1/2 detected		2007 TC479, not detected

					2017 OY31

Image 419802-5: 0/0 detected

Image 411371-56: 1/1 detected		2007 KX1

Image 411371-60: 0/0 detected

Image 411420-10: 0/0 detected

Image 411420-5: 1/1 detected		2003 HH10

COSMOS

Image 59160-51: 1/1 detected		2003 SS69

Image 59150-50: 0/0 detected

It's not clear why 2007 TC479 is missing, since it's not supposed to be particularly faint.

There's still a systematic offset between the predicted and observed positions that's smaller than the 2" matching radius. Fixing that has been deferred to DM-34906 (possibly with a larger data set).

Thanks for agreeing to do this, Ari Heinze, and sorry for the mix-up!

A brief summary, since I understand you're still getting oriented with the Stack:

• ap_association#150 has the code changes done to fix this issue. It's about 80 lines between the bugfixes and the unit tests.
• ap_pipe-notebooks#70 is a scratch notebook used for debugging. You can see a render of the notebook if you select "View File" from the upper right corner of the diff display. Notebook reviews tend to be fairly informal, since it's hard to make line comments and they aren't held to the DM coding standards anyway.
• obs_subaru#419 and obs_decam#222 add centralized config files that are automatically detected and loaded by the pipeline framework (which is why you won't find them explicitly mentioned in the code or pipelines). The task being configured is one of the two being fixed in ap_association.
• ap_verify_ci_hits2015#40, ap_verify_hits2015#45, and ap_verify_ci_cosmos_pdr2#29 are special, self-contained datasets for ap_verify. The primary change is to the input data (which needed to have their coordinates fixed), but the supporting pipelines are updated so that they actually use the configs added to obs_subaru or obs_decam, as appropriate.

For ap_association, there are a large number of unrelated but partially overlapping changes, so I recommend looking at the changes one commit at a time to keep them straight.