Lauren MacArthur added a comment - 18/Nov/19 11:03 PM - edited Dataset Type Dataset Name Where Used Comments catalog src vA Provided by parquet obj table catalog srcMatch vA Will parquet versions be created for normalized matches? camera camera vA & cA How will camera-specific info be retrieved in Gen3? image calexp vA getDetector()*: footprintNpix (note: calls {{getFootprint().getArea() on each src), addFpPoint (focalPlane coord), nQuarter, photoCalib : fluxMag0, buildVisitImage in plotUtils.py image coaddName + "Coadd calexp" cA/pU to create fullTractImage in buildTractImage() in plotUtils.py photoCalib “jointcal” or “fgcmcal [_tract] ” vA This is used to apply calibration to src cats. Will those already be included in parquet obj tables? wcs “jointcal_wcs” va This is used to apply calibration to src cats, but also for visualizing calibrated CCD outlines. How will WCSs be retrieved in Gen3? skymap coaddName + "Coadd_skyMap" vA & cA Needed to get tractInfo (e.g. wcs). How are skyMaps retrieved in Gen3? meta src.getTable().getMetadata() vA & cA Used by afw’s matchesToCatalog How will md be made available in Gen3? catalog coaddName + "Coadd_forced_src" cA colorA Provided by parquet obj table catalog coaddName + "Coadd_meas” cA colorA Provided by parquet obj table catalog coaddName + “measMatch” cA colorA Will parquet versions be created for normalized matches? catalog deepCoadd_fakeSourceCat vA & cA Will parquet versions be created? catalog fakes_src vA Will parquet versions be created? metric json colorAnalysis_verify_job_filename colorA Just for filename path, so should not be required in Gen3 NOTE: there is one function (not yet merged) called buildVisitImage that makes use of functions afw.CameraGeom.utils . These functions themselves make butler.get() files (mostly to get raw or calexp images, but may also get camera, photoCalib, wcs). If my assumptions above about what will be provided from the parquet tables is correct (and modulo the note), it seems that we need the following provided up-front for Gen3 (i.e. to get rid of all butler usage): visitAnalysis.py: srcMatch catalog calexp camera photoCalib wcs skyMap metadata from src catalog coaddAnalysis: measMatch catalog coaddName + "Coadd_calexp" camera skyMap catalog metadata (meas/forced_src)

Lauren MacArthur added a comment - 19/Nov/19 8:13 PM Would you mind giving this a look?  Yusra/Sophie have confirmed my assumptions about what will be provided in parquet tables.  While you do not need to answer the Gen3 questions in my comments, I wanted someone deeply involved in Gen3 to be "aware" of the list of objects we would like to be able to "get" up-front when this becomes a pipeline task.

Nate Lust added a comment - 20/Nov/19 9:49 PM I think most of the datasets (all the ones we care about for this will be just fine for gen3

Lauren MacArthur added a comment - 20/Nov/19 10:02 PM Great, thanks Nate!