The pipe_analysis version is 10a03bd with the w_2018_04 stack. fastparquet has been pip-intalled.

Following the scripts in /scratch/lauren/DM-11345/slurm/, some example commands are:

visitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --tract=9615 --id visit=26024

coaddAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=NB0921 -c doWriteParquetTables=True

colorAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y^NB0921  -c doWriteParquetTables=True

compareVisitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/UD_COSMOS_9813  --tract=9813 --id visit=25816

compareCoaddAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/WIDE_GAMA15H_9615  --id tract=9615 filter=HSC-Z 

As the HSC RC2 dataset was not included in w_2018_02 DM-13066, the comparison is against Lauren MacArthur's repos from w_2017_51.

The visit ones need mosaic ouputs and can be executed per visit; the coadd ones need multiband outputs and can be executed per filter/tract/patch. colorAnalysis is per tract/patch.

Also starting my own notes at https://confluence.lsstcorp.org/display/~hchiang2/Notes+on+pipe_analysis

The outputs are added to /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/
and logs are at /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs

visitAnalysis was run for each visit; 432 logs for 432 visits are at /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs/visitAnalysis/. I utilized slurm --multi-prog option to execute multiple visits on each worker node. The visit IDs are not in the log filenames because I didn't find a way (yet) to directly name the logs with the visit IDs.

coaddAnalysis and compareCoaddAnalysis were run one job for each tract+filter. So in total 16 jobs/logs for coaddAnalysis and 16 for compareCoaddAnalysis.

colorAnalysis was run for each tract; 3 logs available.

compareVisitAnalysis was done in a mixed way; some were grouped but some were done per visit. Logs are copied to /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs/compareVisitAnalysis/. Sorry for the nonuniform logs; this will be improved next time DM-13523.

For compareCoaddAnalysis.py at the COSMOS tract, I had to specify the available patches in the command line. Otherwise, process failed at not being able to find non-existent patches. For example

Traceback (most recent call last):

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+33/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__

    result = task.run(dataRef, **kwargs)

  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 1059, in run

    repoInfo1 = getRepoInfo(patchRefList1[0], coaddName=self.config.coaddName, coaddDataset=dataset)

  File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/utils.py", line 950, in getRepoInfo

    metadata = butler.get(metaStr, dataId)

  File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+8/python/lsst/daf/persistence/butler.py", line 1410, in get

    raise NoResults("No locations for get:", datasetType, dataId)

lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:deepCoadd_forced_src dataId:DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '0,0'}, tag=set())

srun: error: lsst-verify-worker29: task 0: Exited with exit code 1

Thanks for noting this.  I’ll try to fix that soon.

All runs have finished and logs have been copied.

Many calculateSysError were seen in the logs of compareVisitAnalysis and compareCoaddAnalysis. In compareVisitAnalysis logs, there are many warnings of

Warning: sysErr calculation failed: The iteration is not making good progress, as measured by the

  improvement from the last ten iterations.

Those warnings are actually indicating close-to-perfect agreement in the matched source measurement being fit (no progress on the iterative fitting can be made on zero variance).  I will think about trying to get rid of them in this case (but I think they are coming from scipy...)