Lauren MacArthur added a comment - 09/Jul/16 1:47 PM In his review of DM-2674 , John Swinbank asks: Have you compared the performance of this code vs. the HSC version? Yasuda-san was worried that the changes to catalogue loading would cause issues. Perhaps we won't see that until we try running it at scale, though. Would it be straightforward to address this in the context of this ticket (I suppose it would require a profiling to have been tracked in the HSC run)?

John Swinbank added a comment - 31/Aug/16 9:29 AM Reassigning to Lauren, as discussed.

Lauren MacArthur added a comment - 19/Sep/16 9:51 AM I just noticed that more data has been acquired since the RC processing of HSC-1382 . In HSC-1320 the RC dataset is defined as followes: New test definition, following discussion with the pipeline team: Cosmos in all bands to full available depth Two sets of Cosmos in one band to wide depth, with interleaved exposures for testing repeatability Two wide-field tracts Wide depth is 600 sec in gr, 1200 sec in izy. Should I take the above description as the guide here (i.e. include all available data), or would it be preferable to process the exact same dataset that was processed on HSC-1382 for direct comparisons?

Paul Price added a comment - 19/Sep/16 10:12 AM To make the test easier and comparable with the HSC RC tests, I suggest going with the same list of visits.

Lauren MacArthur added a comment - 19/Sep/16 10:13 AM Will do, thanks.

Lauren MacArthur added a comment - 06/Oct/16 10:21 AM - edited I have run the RC dataset through singleFrameDriver.py. While both the HSC and LSST runs have a number of failures in common (mostly for the amp-deprived CCD 9), the following is a list of the differences in failures. The left lists successes on LSST that were failures on HSC, while the right lists the HSC successes that were failures on LSST. Filenames ( updated to reflect newer no junk suppression run, see DM-8044 ): are: pointing/filter/corr/CORR-visit-ccd.fits COSMOS: HSC-G : LSST has/HSC lacks LSST lacks/HSC has 01052/HSC-G/corr/CORR-0011690-009.fits   01052/HSC-G/corr/CORR-0011702-009.fits   01052/HSC-G/corr/CORR-0011706-009.fits   01052/HSC-G/corr/CORR-0011712-009.fits   01232/HSC-G/corr/CORR-0029336-009.fits   HSC-R : LSST has/HSC lacks LSST lacks/HSC has HSC-I : LSST has/HSC lacks LSST lacks/HSC has HSC-Z : LSST has/HSC lacks LSST lacks/HSC has   01111/HSC-Z/corr/CORR-0017928-009.fits 01228/HSC-Z/corr/CORR-0028358-009.fits     01228/HSC-Z/corr/CORR-0028372-069.fits   01228/HSC-Z/corr/CORR-0028372-103.fits   01228/HSC-Z/corr/CORR-0028376-069.fits 01228/HSC-Z/corr/CORR-0028382-009.fits     01228/HSC-Z/corr/CORR-0028382-077.fits   01228/HSC-Z/corr/CORR-0028392-102.fits 01228/HSC-Z/corr/CORR-0028398-029.fits   01228/HSC-Z/corr/CORR-0028398-095.fits   01228/HSC-Z/corr/CORR-0028398-101.fits     01228/HSC-Z/corr/CORR-0028400-053.fits 01228/HSC-Z/corr/CORR-0028400-077.fits   01228/HSC-Z/corr/CORR-0028400-102.fits   01228/HSC-Z/corr/CORR-0028400-103.fits     01228/HSC-Z/corr/CORR-0028402-095.fits HSC-Y : LSST has/HSC lacks LSST lacks/HSC has 00814/HSC-Y/corr/CORR-0000276-030.fits   00814/HSC-Y/corr/CORR-0000278-009.fits   00814/HSC-Y/corr/CORR-0000278-023.fits   00814/HSC-Y/corr/CORR-0000278-070.fits   00814/HSC-Y/corr/CORR-0000278-071.fits   00814/HSC-Y/corr/CORR-0000278-087.fits   00814/HSC-Y/corr/CORR-0000278-100.fits     00814/HSC-Y/corr/CORR-0000280-022.fits 00814/HSC-Y/corr/CORR-0000280-052.fits   00814/HSC-Y/corr/CORR-0000280-054.fits   00814/HSC-Y/corr/CORR-0000280-070.fits     00814/HSC-Y/corr/CORR-0000280-069.fits 00814/HSC-Y/corr/CORR-0000280-096.fits   00814/HSC-Y/corr/CORR-0000282-029.fits   00814/HSC-Y/corr/CORR-0000282-044.fits   00814/HSC-Y/corr/CORR-0000282-091.fits     00814/HSC-Y/corr/CORR-0000284-061.fits 00823/HSC-Y/corr/CORR-0001858-005.fits   00823/HSC-Y/corr/CORR-0001868-041.fits   00823/HSC-Y/corr/CORR-0001876-026.fits   00823/HSC-Y/corr/CORR-0001876-027.fits   00823/HSC-Y/corr/CORR-0001876-028.fits   00823/HSC-Y/corr/CORR-0001876-029.fits   00823/HSC-Y/corr/CORR-0001876-034.fits   00823/HSC-Y/corr/CORR-0001876-035.fits   00823/HSC-Y/corr/CORR-0001876-036.fits   00823/HSC-Y/corr/CORR-0001876-037.fits   00823/HSC-Y/corr/CORR-0001876-041.fits   00823/HSC-Y/corr/CORR-0001876-046.fits   00823/HSC-Y/corr/CORR-0001876-048.fits   00823/HSC-Y/corr/CORR-0001876-049.fits   NB0921 : LSST has/HSC lacks LSST lacks/HSC has   01170/NB0921/corr/CORR-0023052-101.fits   01172/NB0921/corr/CORR-0023602-101.fits   01172/NB0921/corr/CORR-0023606-101.fits   01174/NB0921/corr/CORR-0024310-029.fits   01174/NB0921/corr/CORR-0024310-101.fits   01179/NB0921/corr/CORR-0025810-101.fits WIDE: HSC-G : LSST has/HSC lacks LSST lacks/HSC has 01004/HSC-G/corr/CORR-0009900-009.fits     01052/HSC-G/corr/CORR-0011572-009.fits   01052/HSC-G/corr/CORR-0011582-009.fits 01052/HSC-G/corr/CORR-0011596-009.fits   HSC-R : LSST has/HSC lacks LSST lacks/HSC has   01052/HSC-R/corr/CORR-0011446-009.fits 01052/HSC-R/corr/CORR-0011470-009.fits     01052/HSC-R/corr/CORR-0011478-009.fits   01052/HSC-R/corr/CORR-0011506-009.fits 01052/HSC-R/corr/CORR-0011508-009.fits   01052/HSC-R/corr/CORR-0011534-009.fits   HSC-I : LSST has/HSC lacks LSST lacks/HSC has 00995/HSC-I/corr/CORR-0007300-009.fits   01116/HSC-I/corr/CORR-0019468-069.fits   01116/HSC-I/corr/CORR-0019482-058.fits   HSC-Z : LSST has/HSC lacks LSST lacks/HSC has 01111/HSC-Z/corr/CORR-0017738-009.fits     01111/HSC-Z/corr/CORR-0017754-009.fits HSC-Y : LSST has/HSC lacks LSST lacks/HSC has 00991/HSC-Y/corr/CORR-0006528-066.fits   00991/HSC-Y/corr/CORR-0006528-075.fits   The most typical failure modes were of following: 15328 FATAL singleFrameDriver: Failed on dataId={'taiObs': '2015-03-16', 'pointing': 1170, 'visit': 23038, 'dateObs': '2015-03-16', 'filter': 'NB0921', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 9, 'expTime': 900.0}: File "src/math/LeastSquares.cc", line 435, in void lsst::afw::math::LeastSquares::_factor(bool) Number of columns of design matrix (1) must be smaller than number of data points (0) {0} lsst::pex::exceptions::InvalidParameterError: 'Number of columns of design matrix (1) must be smaller than number of data points (0)'   22178 FATAL singleFrameDriver: Failed on dataId={'taiObs': '2015-03-20', 'pointing': 1174, 'visit': 24310, 'dateObs': '2015-03-20', 'filter': 'NB0921', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 29, 'expTime': 900.0}: Unable to match sources   28709 FATAL singleFrameDriver: Failed on dataId={'taiObs': '2015-05-13', 'pointing': 1228, 'visit': 28400, 'dateObs': '2015-05-13', 'filter': 'HSC-Z', 'field': 'SSP_UDEEP_COSMOS', 'ccd': 53, 'expTime': 300.0}: File "src/PsfexPsf.cc", line 192, in virtual std::shared_ptr<lsst::afw::image::Image<double> > lsst::meas::extensions::psfex::PsfexPsf::_doComputeImage(const Point2D&, const lsst::afw::image::Color&, const Point2D&) const Only spatial variation (ndim == 2) is supported; saw 0 {0}

Lauren MacArthur added a comment - 12/Oct/16 5:00 PM - edited ( Updated to reflect newer no junk suppression run, see DM-8044 ) The cosmos and wide field RC datasets have been run through meas_mosaic . The following lists a comparison of the scatter in the astrometry and photometry (as measured by meas_mosaic and printed in the output diagnostic plots) from both stacks. Note that differences are expected given the slightly different data going into the analysis (see above comment on single frame processing successes vs. failures). In particular, since the LSST pipeline often had a few more successes, it is perhaps not surprising for the scatter to increase somewhat given that these are CCDs that the HSC stack had difficulty getting an initial WCS fit for. Note: ext == external catalog matched objects int == internal objects (i.e. those with no matching external catalog object) Astrometry (just listing eta sigmas) cosmos: LSST [HSC] Filter sig_all (arcsec) sig_ext sig_int HSC-G 0.008 [0.007] 0.025 [0.024] 0.005 [0.005] HSC-R 0.009 [0.006] 0.025 [0.023] 0.005 [0.004] HSC-I 0.011 [0.007] 0.026 [0.025] 0.005 [0.004] HSC-Z 0.014 [0.009] 0.030 [0.028] 0.004 [0.004] HSC-Y 0.015 [0.013] 0.026 [0.025] 0.007 [0.007] NB0921 0.009 [0.007] 0.027 [0.028] 0.004 [0.004] Astrometry wide tract 8766: LSST [HSC] Filter sig_all (arcsec) sig_ext sig_int HSC-G 0.013 [0.011] 0.026 [0.025] 0.005 [0.007] HSC-R 0.018 [0.014] 0.024 [0.024] 0.005 [0.007] HSC-I 0.015 [0.009] 0.028 [0.027] 0.005 [0.005] HSC-Z 0.016 [0.010] 0.030 [0.029] 0.005 [0.006] HSC-Y 0.015 [0.010] 0.026 [0.027] 0.004 [0.005] Astrometry wide tract 8767: LSST [HSC] Filter sig_all (arcsec) sig_ext sig_int HSC-G 0.013 [0.011] 0.026 [0.025] 0.005 [0.007] HSC-R 0.018 [0.014] 0.025 [0.024] 0.005 [0.007] HSC-I 0.015 [0.009] 0.029 [0.028] 0.005 [0.005] HSC-Z 0.016 [0.011] 0.029 [0.028] 0.005 [0.006] HSC-Y 0.015 [0.010] 0.027 [0.027] 0.004 [0.005] Photometry (sigma of Delta(mag)) cosmos: LSST [HSC] Filter sig_all (mag) sig_ext sig_int HSC-G 0.019 [0.017] 0.020 [0.014] 0.019 [0.019] HSC-R 0.027 [0.020] 0.030 [0.017] 0.026 [0.021] HSC-I 0.025 [0.019] 0.028 [0.018] 0.023 [0.019] HSC-Z 0.019 [0.014] 0.019 [0.012] 0.020 [0.015] HSC-Y 0.027 [0.020] 0.031 [0.019] 0.023 [0.020] NB0921 0.021 [0.016] 0.023 [0.016] 0.020 [0.016] Photometry wide tract 8766: LSST [HSC] Filter sig_all (arcsec) sig_ext sig_int HSC-G 0.025 [0.025] 0.030 [0.021] 0.021 [0.027] HSC-R 0.034 [0.024] 0.039 [0.021] 0.022 [0.028] HSC-I 0.026 [0.021] 0.022 [0.015] 0.028 [0.030] HSC-Z 0.021 [0.029] 0.017 [0.011] 0.025 [0.030] HSC-Y 0.036 [0.032] 0.038 [0.025] 0.035 [0.043] Photometry wide tract 8767: LSST [HSC] Filter sig_all (arcsec) sig_ext sig_int HSC-G 0.025 [0.024] 0.030 [0.020] 0.021 [0.027] HSC-R 0.032 [0.024] 0.039 [0.021] 0.020 [0.028] HSC-I 0.024 [0.020] 0.021 [0.014] 0.026 [0.027] HSC-Z 0.019 [0.017] 0.016 [0.011] 0.021 [0.026] HSC-Y 0.034 [0.031] 0.036 [0.024] 0.032 [0.040] My conclusion from the above is that the comparisons after the meas_mosaic processing stage look very good.