As noted in the AP pipeline meeting, the timing metric is a bit vague about what it's actually timing (in particular, the documentation for TimingMetricTask.run says wall-clock time, but the documentation for TimingMetricTask itself or individual metrics does not). Since this ticket must deal with metric presentation issues anyway, modify the descriptions to be explicit. Likewise, check that other metrics' descriptions are as precise and unambiguous as possible.

Hi Eric Bellm and Jonathan Sick, would you be willing to review this ticket? Unfortunately, the work got a bit out of hand, so the changes are large and a bit tangled up, particularly in ap_verify. I apologize for that.

Jonathan Sick, please review the changes to verify and ap_verify, which are partly improvements to MetricTask infrastructure and partly fixes and workarounds I couldn't decouple from them.

Eric Bellm, please review the changes to verify_metrics, ip_diffim, and ap_association, which deal more with metric/measurement definitions.

Hi Krzysztof Findeisen--see comments on github.

My only large comment is that I'm worried now about the name "science sources" (which you probably got from me!).  It doesn't really match the rest of LSST nomenclature (are DIASources not used for science?).  Should we consider "direct imaging sources" or "calexp sources" or "PVI sources"?  Ideas welcome.

Eric Bellm, all names (including "science sources") were taken directly from the previous versions of the metrics. I don't mind changing the name, but I think if we do then we should change it everywhere (notebooks, Confluence, etc.). Maybe we should discuss the name in #dm-alert-prod or group meeting?

Sounds good--I'll add it to the Monday meeting agenda.

On Fri, Mar 8, 2019 at 3:35 PM Krzysztof Findeisen (JIRA) <jira-dm@lsst.org>

This looks good from my perspective. Let me know if there's something I should have been looking at in more detail.

Ok, I'll pick up threads here from both the verify and ap_verify PR discussions. I have a better appreciation of where this is going now, and the potential issues. To be completely honest and transparent, I haven't worked on lsst.verify or QA in general for about 1.5 years, so I don't have much context with what's going on and hence why I've been generally unopinonated.

I see now specifically that these task bases are going into the root of lsst.verify. As I said in https://jira.lsstcorp.org/browse/DM-17543?focusedCommentId=185663&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-185663 it's up to you, but it might be a good idea to collect tasks task baseclasses in a lsst.verify.tasks subpackage. As I said before, this doesn't mean creating a new EUPS package (which I don't want), but it does organize the APIs). This specifically addresses your concern in https://github.com/lsst/ap_verify/pull/63#issuecomment-473024245 that adding TimingMetricTask to lsst.verify might pollute that namespace.

(But again, my level of thinking here is probably pretty shallow. You're the lsst.verify expert right now!)

Thanks for the feedback!