Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22253

Audit usage of butler.get() in pipe_analysis scripts

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None

      Description

      As an initial setup towards Gen3-ification of the pipe_analysis scripts, audit all usage of butler.get() to get a feeling for how easy it will be to convert them to Gen3-land.

        Attachments

          Activity

          Hide
          lauren Lauren MacArthur added a comment - - edited
          Dataset Type Dataset Name Where Used Comments
          catalog src vA Provided by parquet obj table
          catalog srcMatch vA Will parquet versions be created for normalized matches?
          camera camera vA & cA How will camera-specific info be retrieved in Gen3?
          image calexp vA getDetector()*: footprintNpix (note: calls {{getFootprint().getArea() on each src), addFpPoint (focalPlane coord), nQuarter, photoCalib: fluxMag0, buildVisitImage in plotUtils.py
          image coaddName + "Coadd calexp" cA/pU to create fullTractImage in buildTractImage() in plotUtils.py
          photoCalib “jointcal” or “fgcmcal[_tract] vA This is used to apply calibration to src cats. Will those already be included in parquet obj tables?
          wcs “jointcal_wcs” va This is used to apply calibration to src cats, but also for visualizing calibrated CCD outlines. How will WCSs be retrieved in Gen3?
          skymap coaddName + "Coadd_skyMap" vA & cA Needed to get tractInfo (e.g. wcs). How are skyMaps retrieved in Gen3?
          meta src.getTable().getMetadata() vA & cA Used by afw’s matchesToCatalog How will md be made available in Gen3?
          catalog coaddName + "Coadd_forced_src" cA colorA Provided by parquet obj table
          catalog coaddName + "Coadd_meas” cA colorA Provided by parquet obj table
          catalog coaddName + “measMatch” cA colorA Will parquet versions be created for normalized matches?
          catalog deepCoadd_fakeSourceCat vA & cA Will parquet versions be created?
          catalog fakes_src vA Will parquet versions be created?
          metric json colorAnalysis_verify_job_filename colorA Just for filename path, so should not be required in Gen3

          NOTE: there is one function (not yet merged) called buildVisitImage that makes use of functions afw.CameraGeom.utils. These functions themselves make butler.get() files (mostly to get raw or calexp images, but may also get camera, photoCalib, wcs).

          If my assumptions above about what will be provided from the parquet tables is correct (and modulo the note), it seems that we need the following provided up-front for Gen3 (i.e. to get rid of all butler usage):

          visitAnalysis.py:

          • srcMatch catalog
          • calexp
          • camera
          • photoCalib
          • wcs
          • skyMap
          • metadata from src catalog

          coaddAnalysis:

          • measMatch catalog
          • coaddName + "Coadd_calexp"
          • camera
          • skyMap
          • catalog metadata (meas/forced_src)
          Show
          lauren Lauren MacArthur added a comment - - edited Dataset Type Dataset Name Where Used Comments catalog src vA Provided by parquet obj table catalog srcMatch vA Will parquet versions be created for normalized matches? camera camera vA & cA How will camera-specific info be retrieved in Gen3? image calexp vA getDetector()*: footprintNpix (note: calls {{getFootprint().getArea() on each src), addFpPoint (focalPlane coord), nQuarter, photoCalib : fluxMag0, buildVisitImage in plotUtils.py image coaddName + "Coadd calexp" cA/pU to create fullTractImage in buildTractImage() in plotUtils.py photoCalib “jointcal” or “fgcmcal [_tract] ” vA This is used to apply calibration to src cats. Will those already be included in parquet obj tables? wcs “jointcal_wcs” va This is used to apply calibration to src cats, but also for visualizing calibrated CCD outlines. How will WCSs be retrieved in Gen3? skymap coaddName + "Coadd_skyMap" vA & cA Needed to get tractInfo (e.g. wcs). How are skyMaps retrieved in Gen3? meta src.getTable().getMetadata() vA & cA Used by afw’s matchesToCatalog How will md be made available in Gen3? catalog coaddName + "Coadd_forced_src" cA colorA Provided by parquet obj table catalog coaddName + "Coadd_meas” cA colorA Provided by parquet obj table catalog coaddName + “measMatch” cA colorA Will parquet versions be created for normalized matches? catalog deepCoadd_fakeSourceCat vA & cA Will parquet versions be created? catalog fakes_src vA Will parquet versions be created? metric json colorAnalysis_verify_job_filename colorA Just for filename path, so should not be required in Gen3 NOTE: there is one function (not yet merged) called buildVisitImage that makes use of functions afw.CameraGeom.utils . These functions themselves make butler.get() files (mostly to get raw or calexp images, but may also get camera, photoCalib, wcs). If my assumptions above about what will be provided from the parquet tables is correct (and modulo the note), it seems that we need the following provided up-front for Gen3 (i.e. to get rid of all butler usage): visitAnalysis.py: srcMatch catalog calexp camera photoCalib wcs skyMap metadata from src catalog coaddAnalysis: measMatch catalog coaddName + "Coadd_calexp" camera skyMap catalog metadata (meas/forced_src)
          Hide
          lauren Lauren MacArthur added a comment -

          Would you mind giving this a look?  Yusra/Sophie have confirmed my assumptions about what will be provided in parquet tables.  While you do not need to answer the Gen3 questions in my comments, I wanted someone deeply involved in Gen3 to be "aware" of the list of objects we would like to be able to "get" up-front when this becomes a pipeline task.

          Show
          lauren Lauren MacArthur added a comment - Would you mind giving this a look?  Yusra/Sophie have confirmed my assumptions about what will be provided in parquet tables.  While you do not need to answer the Gen3 questions in my comments, I wanted someone deeply involved in Gen3 to be "aware" of the list of objects we would like to be able to "get" up-front when this becomes a pipeline task.
          Hide
          nlust Nate Lust added a comment -

          I think most of the datasets (all the ones we care about for this will be just fine for gen3

          Show
          nlust Nate Lust added a comment - I think most of the datasets (all the ones we care about for this will be just fine for gen3
          Hide
          lauren Lauren MacArthur added a comment -

          Great, thanks Nate!

          Show
          lauren Lauren MacArthur added a comment - Great, thanks Nate!

            People

            Assignee:
            lauren Lauren MacArthur
            Reporter:
            lauren Lauren MacArthur
            Reviewers:
            Nate Lust
            Watchers:
            Jim Bosch, Lauren MacArthur, Nate Lust, Sophie Reed, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.