The output repo is linked here, where one can access the logs and view the validate_drp plots, e.g. PA1.

A slightly overdue update on this ticket.

The metrics were re-dispatched using the following environment variable settings:

export DATASET="DC2_test-med-1"

export DATASET_REPO_URL="https://jira.lsstcorp.org/browse/DM-30730"

export RUN_ID="DM-30730"

export RUN_ID_URL="https://jira.lsstcorp.org/browse/DM-30730"

export VERSION_TAG="w_2021_24"

I've just setup an initial attempt at a chronograf dashboard (an edited clone of the existing "DRP metrics monthly DC2 test-med-1") called:
DRP monthly metrics for DC2 (Gen2)
I'm not sure why the PF1 measurements are not showing up (I may have messed up the query, but can't see where),
nor why I seem to be picking up extra TE1 values (it seems they are present for two previous runs, whereas none of the others are?), nor why no i band data seem to be showing up (despite the json files being present and dispatched). But otherwise all looks ok

Side note: should we be concerned that the matchedVisit_band.json files are of order 6.1G?

I have been updating the pipe_analysis scripts so they all run on DC2 data (some plots can be seen here). I will finish that up and include the PRs on a separate ticket.

Please let me know if you'd like to see anything else on this ticket. Otherwise, I think it's ready to be closed out.

The PF1 query looks fine; it looks like the measurements just didn't succeed.

grep -C 2 'validate_drp.PA1"' /datasets/DC2/repoRun2.2i/rerun/w_2021_24/DM-30730/validateDrp/matchedVisitMetrics/3828/g/matchedVisit_g.json

... outputs:

      "identifier": "784f5c4e871c457eab61af8c2b0985db",

      "metric": "validate_drp.PA1",

      "unit": "mmag",

      "value": 9.469147892027074

Meanwhile, the same grep for validate_drp.PF1_design_gri gets nothing. So it's probably a validate_drp bug that nobody is ever going to fix.

I think the jsons are giant because validate_drp saves blobs with entire columns worth of data in ASCII.

Ah, right...thanks Dan.  I just checked some older logs for the PF1_design_gri entries...missing there too, so probably a long-standing (and, indeed, never-to-be-fixed) bug in validate_drp.