I'd like to RFC any DM input to this change request before I submit it to change control.
For the fastest-evolving explosive transients, such as young supernovae and gamma-ray bursts, having knowledge of the history of past nondetections is vital to taking appropriate real-time action. A new transient in a nearby galaxy is interesting if the last nondetection was >3 days ago, exciting if it was last night, and urgent if it was not seen an hour ago! Practical experience from current-generation transient surveys shows that the history of past nondetections is routinely used to make realtime followup decisions when the number of potential targets outstrips followup capacity.
In the current DPDD, however, the L1 alert packet only contains a record of previous detections (via the previous DIASource records). There is no information in the alert packet about when LSST has recently observed this position and found no sources. This information becomes available (along with a record of any sub-threshold detections) only after completion of the precovery forced photometry, which is required to occur within 24 hours--far too slow for followup of rapidly evolving events.
We propose that the alert packet definition DPDD be amended to include "median noise in the difference image from previous LSST visits in the last 12 months where this DIAObject was not detected." This will require maintaining a database table of the difference image noise on at least a per-visit basis. We would require that these limits be stored during the 60-second L1 processing cycle so that they would be accessible to later observations at that position the same night.
We use noise in the difference image in order to allow the user to construct the relevant limit. (The calculation differs depending on whether the astrophysical object in question is present in the template image (e.g., a variable star) or not (e.g., a supernova).)
We chose 12 months for consistency with the lookback interval provided for DIASources—users will have a uniform record of all observations in that time, whether detection or upper limit. While the time since last nondetection is the most critical quantity for the science cases above, it can be useful to know that an object was not previously detected many times (as opposed to simply not observed).
Provision of nondetections in the alert packet will not replace the higher-fidelity precovery alert processing, but will enable real-time filtering of the alert packets by downstream brokers that incorporates this information.
The noise-based upper limits provided will necessarily be estimates for any specific source; we do not envision taking into account masked pixels or extended background emission, for instance. Users requiring more precise limits may examine or process the earlier images themselves, or simply wait for the completion of precovery photometry a few hours later.
- alert packet size increase: alerts will have up to Nvisit/12 months (~80) additional records added, although these will be more substantially more compact than a full DIASource record (potentially just ccdVisitId, midPointTai, filter, noise). The exact increase will depend on the average number of previous detections per DIASource. We could exclude the nondetection lookback for DIASources associated with known solar system objects to provide some bandwidth savings, as moving objects will have generally have only nondetections at any given position, and those nondetections are unlikely to be scientifically useful.
- new database table: A new table storing difference image noise per visit will be required, with ~275,000 records per year. If noise is stored per CCD, as is likely due to L1 pipeline parallelism, the number of records is multiplied by 189.
- processing time: As the difference image noise database table is small relative to the L1 database, the additional queries should have a minimal impact on the time to generate alert packets.
- schedule: minimal schedule impact is expected.