Thanks for starting this!

Regarding the questions about the env info,

• DATASET_REPO_URL: No, there is no git lfs repo corresponding to the dataset. Generally speaking, I don't envision future datasets we process would have git lfs repo either, because most are likely larger than we want to store at a git lfs repo. However, for the HSC-RC2 dataset, we do have a Jira ticket DM–11345 so that URL may be an option? 

• RUN_ID: In the Gen3 middleware there is the concept of RUN_ID as well (see https://dmtn-056.lsst.io/grouping-and-provenance.html#run ) and it looks like the concept maps to yours perfectly. That's great. Before that, using the JIRA ticket sounds like a good idea to me.

About the last paragraph, HSC-RC2 covers 3 tracts, including 241 patches in total (in w_2018_17). I ran matchedVisitMetrics.py for each tract x filter combinations, and that summed up to 16 matchedVisitMetrics.py jobs so 16 JSON files. Should I group them differently? We can see each tract as a different sub-dataset of HSC-RC2 so I guess we want 3 at least.

Great, let us use DM-11345 as reference to the HSC-RC2 dataset and the JIRA ticket associated to the run as RUN_ID for now. Simon Krughoff I think while SQuaSH does not handle the spatial information (i.e. multiple tracks) we could combine the filters for one track into a single verification JSON and send that to SQuaSH. I have no idea of the size of the resulting JSON at this point, but the SQuaSH API should be able to handle requests up to ~1GB https://github.com/lsst-sqre/squash-restful-api/blob/master/kubernetes/nginx/nginx.conf#L16

So, before creating a new environment in dispatch_verify.py I just tried to use the "jenkins" enviroment variables to give it a try:

dispatch_verify.py --url https://squash-restful-api-demo.lsst.codes --user <squash user> --password <squash password> --env jenkins --ignore-lsstsw /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/validateDrp/matchedVisitMetrics/9615/HSC-G/matchedVisit_HSC-G.json

verify.bin.dispatchverify.main INFO: Loading /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/validateDrp/matchedVisitMetrics/9615/HSC-G/matchedVisit_HSC-G.json

verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics

verify.bin.dispatchverify.main INFO: Inserting Jenkins CI environment metadata.

verify.bin.dispatchverify.main INFO: Uploading Job JSON to https://squash-restful-api-demo.lsst.codes.

verify.squash.get INFO: GET https://squash-restful-api-demo.lsst.codes status: 200

verify.squash.post INFO: POST https://squash-restful-api-demo.lsst.codes/auth status: 200

verify.squash.post INFO: POST https://squash-restful-api-demo.lsst.codes/job status: 202

you can see the validate_drp results for HSC-RC2 dataset at https://squash-demo.lsst.codes/dash/code_changes/

Some findings:

• I would not combine all the filters for a given patch as previously suggested, sending one tract and one filter each time seems ok and we do the same in CI.

• The drill down plot for the AM1 metric takes a while to load, but it eventually loads (one tract has enough data points to make the app irresponsive...).

• lsst.very relies on repos.yaml to get the information about the package versions. I run it with --ignore-lsstsw because repos.yaml was not found in the shared stack installation. We have to figure out how to obtain that information so that we have the "Code changes" column filled.

FileNotFoundError: [Errno 2] No such file or directory: '/ssd/lsstsw/stack/etc/repos.yaml'

• It is important to have in mind that the time displayed in SQuaSH correspond to the timestamp of the job when it was registered (not when it was executed). In CI this is not an issue since dispatch_verify runs right after the job is finished. Hsin-Fang Chiang do you think we can add the dispatch_verify step as part of the pipeline so that it is also automated?

• The CmdLineTask framework stores a packages.pickle file with version information of the stack packages. It's the datasetType of "packages" in the Gen1/2 Butler. An example is at /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/config/packages.pickle. Is this useful for SQuaSH to gather the package version information?

• Yes, I think I can add an additional dispatch_verify step after generating the JSON file.

In DM-14357 we added the LDF execution environment option in dispatch_verify.py.  Another change in progress is DM-14550 that implements a new option --ignore-blobs to skip the upload of data blobs to SQuaSH.

Hsin-Fang Chiang we are ready to test the submission of validate_drp results on HSC RC2 reprocessing to SQuaSH.

I've added instructions on how to run dispatch_verify on the LDF environment here: https://confluence.lsstcorp.org/display/~hchiang2/validateDrp

Thanks a lot Angelo Fausti.   I've submitted one tract of w_2018_22's validate_drp results of HSC RC2 reprocessing to SQuaSH, and the 5 data points (for 5 filters) can be seen on the dashboard now. I'll add this to each biweekly run.

I removed the child tasks related to visualization and interface changes in order to close this ticket since we have data flowing from LDF to SQuaSH. See results on https://squash.lsst.codes for the HSC RC2 dataset.