Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31459

Update faro to use parquet tables for patch and tract-level metric calculation

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • faro

    Description

      Use the `objectTable` parquet files instead of "deepCoadd_forced_src" FITS files to calculate metrics in `faro` on tract and patch scales.

      Attachments

        Issue Links

          Activity

            Fly-by comment here (just some food for thought...feel free to pay it no attention!) When I added this ability (i.e. to read the parquet catalog tables) in pipe_analysis, I did maintain the ability to read in the afw SourceCatalogs. This has proven very useful for folks doing quick testing runs where the parquet table writing tasks were not run. The point may be moot, however, for a few reasons:

            • it may be that it would just not be worth running faro on small test runs and/or it may be considered fair to make the parquet table creation a pre-requisite for running faro (so, just perhaps be sure to note this in your docs/tutorials)
            • if, as indicated in the description (and I believe is the preferred route for production), you are converting to reading in the DPDD-ified objectTable tables – which have different column names for ~everything – as opposed to the *Coadd_obj tables (whose column names match the afw SourceCatalogs...I read in these versions in pipe_analysis for just this reason!), then it would be onerous (and potentially error-prone) to try to keep both schemas in play.
            lauren Lauren MacArthur added a comment - Fly-by comment here (just some food for thought...feel free to pay it no attention!) When I added this ability (i.e. to read the parquet catalog tables) in pipe_analysis, I did maintain the ability to read in the afw SourceCatalogs. This has proven very useful for folks doing quick testing runs where the parquet table writing tasks were not run. The point may be moot, however, for a few reasons: it may be that it would just not be worth running  faro  on small test runs and/or it may be considered fair to make the parquet table creation a pre-requisite for running faro (so, just perhaps be sure to note this in your docs/tutorials) if, as indicated in the description (and I believe is the preferred route for production), you are converting to reading in the DPDD-ified  objectTable  tables – which have different column names for ~everything – as opposed to the *Coadd_obj tables (whose column names match the afw SourceCatalogs...I read in these versions in pipe_analysis for just this reason!), then it would be onerous (and potentially error-prone) to try to keep both schemas in play.

            People

              edennihy Erik Dennihy
              jcarlin Jeffrey Carlin
              Brock Brendal [X] (Inactive), Jeffrey Carlin, Lauren MacArthur, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.