An incomplete summary of the wide-ranging discussion:
Meredith Rawls presented an overview of the work to date. Her slides are available here (must log in).
Re-arranging my notes to be topical rather than chronological:
There were a range of suggestions for improvements for some of the example metrics the L1 group had brainstormed. Among them:
using rho statistics: 2-point correlation function on holdout fraction
Michael Wood-Vasey suggested doing forced photometry at a fixed blank sky location; Robert Lupton says this is implemented on direct imaging—SkyObjects.
measuring chi-squared of the residuals at the positions of known stars in an appropriate magnitude range (propagating the correlation matrix to get sigma right)
There was general agreement that ultimately ap_verify needs to use stack-built (rather than Community Pipeline) calibrations, for self-consistency.
Robert Lupton asked whether ingestion should be part of ap_verify. There was also discussion of whether we should be testing if calibration products are appropriate (which is really the job of the calibration team) or are simply being applied correctly.
The DRP team (Robert Lupton, Lauren MacArthur, and others) discussed the HSC QA workflows. The HSC pipeline stores relevant metadata in the production database for traceability. QA workflows are handled by annotating catalog products, which are then separately processed for QA.
lsst.verify, SQUASH, etc.:
Robert Lupton suggested that common metrics code could live in afw; Michael Wood-Vasey wondered if SRD-verifying KPM code should live somewhere distinct.
We discussed whether metrics should be put in the metadata. Currently PropertySet doesn’t support lsst.verify.Measurements classes, but there was some discussion that it might be worth extending Task to enable this. It was unclear whether it made more sense to wait for SuperTask.
There was suggestion that the SQUASH system should focused on continuous integration on small datasets to watch for regressions, and larger verification and/or QA/drill-down workflows should be handled differently.