The version of pipe_analysis was master at the time of the w_2018_28 release, at commit ebd48cf.
The comparison scripts were run against RC/w_2018_26/DM-14689

For validateDrp.py, ignored filter=NB0921 of COSMOS because running validateDrp.py /datasets/hsc/repo/rerun/RC/w_2018_28/DM-14988/validateDrp/matchedVisitMetrics/9813/NB0921/matchedVisit_NB0921.json --noplot gave the following error like before (e.g. DM-14341)

Traceback (most recent call last):

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_drp/16.0-4-g0b0e9de+6/bin/validateDrp.py", line 109, in <module>

    validate.run(args.repo, **kwargs)

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_drp/16.0-4-g0b0e9de+6/python/lsst/validate/drp/validate.py", line 157, in run

    print_metrics(job)

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_drp/16.0-4-g0b0e9de+6/python/lsst/validate/drp/validate.py", line 482, in print_metrics

    metric = job.metrics[metric_set[0]] # Pick the first one for the description

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/verify/16.0+1/python/lsst/verify/metricset.py", line 207, in __getitem__

    return self._metrics[key]

KeyError: Name('release', 'PF1')

DM-15127  has been filed.

compareVisitAnalysis.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_28/DM-14988 --rerun2 RC/w_2018_26/DM-14689 --tract=9697 --id visit=36234 failed with TaskError: No matches found.

I guess this is related to DM-15117 the fact that w_2018_28 failed in one more CCD at visit=36234 compared to w_2018_26. So I'll just ignore this visit this time. Lauren MacArthur please let me know if that's not okay.

Sounds good to me.  I will pursue this a bit further (but not in the very near future) as that particular CCD/visit does not actually overlap with tract 9697 (so doesn't get included in the plots...see CCD overlays on sky plots), so I'm not sure if it being missing in one of the repos should bring compareVisitAnalysis to a halt.  I suppose its presence could affect the meas_mosaic fitting, so perhaps I should let this situation bring it down, but perhaps try to add a more informative error message.

Indeed, it's weird that a non-overlapping CCD could affect the matches. Maybe other CCDs have changed enough as well? From the logs, the "number of sources in catalogs" are quite smaller than before.

coaddAnalysis.py failed in the COSMOS tract=9813 filters HSC-G, HSC-R, HSC-Z, and HSC-Y with matches problem; DM-15133 has been filed.

The number of sources is indeed smaller as a result of DM-14874 (some duplication of sources was occurring because I was not selecting on "isPrimary").

 It's really puzzling that HSC-I and NB0921 worked but the other four filters didn't!

 Thanks for tracking this all...hopefully I'll get to the bottom of it soon-ish!