Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-40061

Add parquet output to new CalibrateImageTask

    XMLWordPrintable

Details

    Description

      In the interest of getting things merged so we can start testing, I'm not writing parquet tables in the new CalibrateImageTask. The end goal is for the SourceCatalogs to just contain the Footprints, while the parquet tables contain all the rest of the catalog info. The SourceCatalog outputs are initial_stars_footprints and initial_stars_psf_footprints, and the parquet versions will not include _footprints, and instead have _detector (following DMTN-172, which says that per-detector outputs should have a suffix so that the concatenated versions don't have to).

      This won't be an SDM-style catalog, as these initial catalogs aren't necessarily external data products.

      Attachments

        Issue Links

          Activity

            jbosch: I hope you're free to do this short (~50 lines) review? This makes use of the new StorageClasses you added on RFC-844, so you might have thoughts on my choice related to that (it wasn't clear what the difference between the three Arrow* types was).

            I'm not entirely happy about the length of initial_psf_stars_footprints_detector and initial_psf_stars_detector, but those are optional outputs. initial_stars_footprints_detector and initial_stars_detector are also a bit much, but they follow the naming given on the "great refactor" document.

            It'll be a while before we switch later code to use the astropy outputs instead of the afw catalogs, but at least they will exist now.

            Parejkoj John Parejko added a comment - jbosch : I hope you're free to do this short (~50 lines) review? This makes use of the new StorageClasses you added on RFC-844 , so you might have thoughts on my choice related to that (it wasn't clear what the difference between the three Arrow* types was). I'm not entirely happy about the length of initial_psf_stars_footprints_detector and initial_psf_stars_detector , but those are optional outputs. initial_stars_footprints_detector and initial_stars_detector are also a bit much, but they follow the naming given on the "great refactor" document. It'll be a while before we switch later code to use the astropy outputs instead of the afw catalogs, but at least they will exist now.
            Parejkoj John Parejko added a comment - Jenkins: https://rubin-ci.slac.stanford.edu/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/687/pipeline
            jbosch Jim Bosch added a comment -

            Looks good, and apologies for the delay.

            jbosch Jim Bosch added a comment - Looks good, and apologies for the delay.

            People

              Parejkoj John Parejko
              Parejkoj John Parejko
              Jim Bosch
              Ian Sullivan, Jim Bosch, John Parejko
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.