Lauren MacArthur added a comment - 06/Aug/21 9:44 PM The tasks were added in DM-28389 . It lives in the analysis_drp repo and are called make[Forced/Unforced]QaTractTables . There is a pipeline to run them. The gist of it is that I read in the deepCoadd_obj parquet tables (one per patch) and load only a subsection of the columns based on a few search criteria which boil down to: include columns whose names start with a prefix in the list defined in the baseColStrList ** config, while excluding this containing any string in the notInColStrList . One can choose to make consolidated tables using either the "meas" (a.k.a unforced) or "forced_src" (a.k.a. forced) values. When considering forced_src parameters, I copy over some columns from the "meas" and "ref" catalogs, as defined in columnsToCopyFromMeas *** and columnsToCopyFromRef , respectively. I will note that I didn't necessarily use every single column that gets included with these live searches in the pipe_analysis QA scripts (but the ease and extensibility of this kind of search was worth ending up with a few "extras"). The example tables referred to in DM-28389 do still exist, so you can feel free to have a look at them. But for your convenience, here are the column listings (note that since the search happens live, a given run could end up with a slightly different list. Say, e.g., if the HSM algorithm was not run all *HSM* columns will be absent): In [ 1 ]: from lsst.daf.butler import Butler In [ 2 ]: rootDir = "/project/hsc/gen3repo/rc2w02_ssw03" In [ 3 ]: butler = Butler(rootDir, collections = "u/lauren/DM-28389" ) In [ 4 ]: gQaTable_unforced = butler.get( "qaTractTable_unforced" , tract = 9813 , band = "g" ) In [ 5 ]: list (gQaTable_unforced.columns) Out[ 5 ]: [ 'base_Blendedness_abs' , 'base_Blendedness_flag' , 'base_CircularApertureFlux_12_0_flag' , 'base_CircularApertureFlux_12_0_instFlux' , 'base_CircularApertureFlux_12_0_instFluxErr' , 'base_CircularApertureFlux_25_0_flag' , 'base_CircularApertureFlux_25_0_instFlux' , 'base_CircularApertureFlux_25_0_instFluxErr' , 'base_CircularApertureFlux_9_0_instFlux' , 'base_CircularApertureFlux_9_0_instFluxErr' , 'base_ClassificationExtendedness_flag' , 'base_ClassificationExtendedness_value' , 'base_FootprintArea_value' , 'base_GaussianFlux_apCorr' , 'base_GaussianFlux_apCorrErr' , 'base_GaussianFlux_flag' , 'base_GaussianFlux_flag_apCorr' , 'base_GaussianFlux_instFlux' , 'base_GaussianFlux_instFluxErr' , 'base_InputCount_flag' , 'base_InputCount_value' , 'base_PixelFlags_flag' , 'base_PixelFlags_flag_bright_object' , 'base_PixelFlags_flag_bright_objectCenter' , 'base_PixelFlags_flag_clipped' , 'base_PixelFlags_flag_clippedCenter' , 'base_PixelFlags_flag_cr' , 'base_PixelFlags_flag_crCenter' , 'base_PixelFlags_flag_edge' , 'base_PixelFlags_flag_inexact_psf' , 'base_PixelFlags_flag_inexact_psfCenter' , 'base_PixelFlags_flag_interpolated' , 'base_PixelFlags_flag_interpolatedCenter' , 'base_PixelFlags_flag_offimage' , 'base_PixelFlags_flag_saturated' , 'base_PixelFlags_flag_saturatedCenter' , 'base_PixelFlags_flag_sensor_edge' , 'base_PixelFlags_flag_sensor_edgeCenter' , 'base_PixelFlags_flag_suspect' , 'base_PixelFlags_flag_suspectCenter' , 'base_PsfFlux_apCorr' , 'base_PsfFlux_apCorrErr' , 'base_PsfFlux_area' , 'base_PsfFlux_flag' , 'base_PsfFlux_flag_apCorr' , 'base_PsfFlux_flag_edge' , 'base_PsfFlux_instFlux' , 'base_PsfFlux_instFluxErr' , 'base_SdssCentroid_flag' , 'base_SdssCentroid_x' , 'base_SdssCentroid_xErr' , 'base_SdssCentroid_y' , 'base_SdssCentroid_yErr' , 'base_SdssShape_flag' , 'base_SdssShape_flag_psf' , 'base_SdssShape_flag_shift' , 'base_SdssShape_instFlux' , 'base_SdssShape_instFluxErr' , 'base_SdssShape_psf_xx' , 'base_SdssShape_psf_xy' , 'base_SdssShape_psf_yy' , 'base_SdssShape_x' , 'base_SdssShape_xx' , 'base_SdssShape_xxErr' , 'base_SdssShape_xy' , 'base_SdssShape_xyErr' , 'base_SdssShape_y' , 'base_SdssShape_yy' , 'base_SdssShape_yyErr' , 'calib_astrometry_used' , 'calib_photometry_reserved' , 'calib_photometry_used' , 'calib_psf_candidate' , 'calib_psf_reserved' , 'calib_psf_used' , 'coord_dec' , 'coord_ra' , 'deblend_nChild' , 'detect_isPatchInner' , 'detect_isPrimary' , 'detect_isTractInner' , 'ext_photometryKron_KronFlux_apCorr' , 'ext_photometryKron_KronFlux_apCorrErr' , 'ext_photometryKron_KronFlux_flag' , 'ext_photometryKron_KronFlux_flag_apCorr' , 'ext_photometryKron_KronFlux_flag_edge' , 'ext_photometryKron_KronFlux_instFlux' , 'ext_photometryKron_KronFlux_instFluxErr' , 'ext_shapeHSM_HsmPsfMoments_flag' , 'ext_shapeHSM_HsmPsfMoments_x' , 'ext_shapeHSM_HsmPsfMoments_xx' , 'ext_shapeHSM_HsmPsfMoments_xy' , 'ext_shapeHSM_HsmPsfMoments_y' , 'ext_shapeHSM_HsmPsfMoments_yy' , 'ext_shapeHSM_HsmShapeRegauss_e1' , 'ext_shapeHSM_HsmShapeRegauss_e2' , 'ext_shapeHSM_HsmShapeRegauss_flag' , 'ext_shapeHSM_HsmShapeRegauss_flag_galsim' , 'ext_shapeHSM_HsmShapeRegauss_resolution' , 'ext_shapeHSM_HsmShapeRegauss_sigma' , 'ext_shapeHSM_HsmSourceMoments_flag' , 'ext_shapeHSM_HsmSourceMoments_x' , 'ext_shapeHSM_HsmSourceMoments_xx' , 'ext_shapeHSM_HsmSourceMoments_xy' , 'ext_shapeHSM_HsmSourceMoments_y' , 'ext_shapeHSM_HsmSourceMoments_yy' , 'merge_peak_sky' , 'modelfit_CModel_apCorr' , 'modelfit_CModel_apCorrErr' , 'modelfit_CModel_ellipse_xx' , 'modelfit_CModel_ellipse_xy' , 'modelfit_CModel_ellipse_yy' , 'modelfit_CModel_flag' , 'modelfit_CModel_flag_apCorr' , 'modelfit_CModel_flags_smallShape' , 'modelfit_CModel_instFlux' , 'modelfit_CModel_instFluxErr' , 'modelfit_CModel_instFlux_inner' , 'parent' , 'slot_Centroid_flag' , 'slot_Centroid_flag_almostNoSecondDerivative' , 'slot_Centroid_flag_edge' , 'slot_Centroid_flag_resetToPeak' , 'slot_Centroid_x' , 'slot_Centroid_xErr' , 'slot_Centroid_y' , 'slot_Centroid_yErr' , 'slot_ShapeRound_Flux' , 'slot_ShapeRound_flag' , 'slot_ShapeRound_x' , 'slot_ShapeRound_xx' , 'slot_ShapeRound_xy' , 'slot_ShapeRound_y' , 'slot_ShapeRound_yy' , 'slot_Shape_flag' , 'slot_Shape_x' , 'slot_Shape_xx' , 'slot_Shape_xy' , 'slot_Shape_y' , 'slot_Shape_yy' , 'tractId' , 'patchId' , 'id' ]   In [ 6 ]: gQaTable_forced = butler.get( "qaTractTable_forced" , tract = 9813 , band = "g" )   In [ 7 ]: list (gQaTable_forced.columns) Out[ 7 ]: [ 'base_CircularApertureFlux_12_0_flag' , 'base_CircularApertureFlux_12_0_instFlux' , 'base_CircularApertureFlux_12_0_instFluxErr' , 'base_CircularApertureFlux_25_0_flag' , 'base_CircularApertureFlux_25_0_instFlux' , 'base_CircularApertureFlux_25_0_instFluxErr' , 'base_CircularApertureFlux_9_0_instFlux' , 'base_CircularApertureFlux_9_0_instFluxErr' , 'base_ClassificationExtendedness_flag' , 'base_ClassificationExtendedness_value' , 'base_GaussianFlux_apCorr' , 'base_GaussianFlux_apCorrErr' , 'base_GaussianFlux_flag' , 'base_GaussianFlux_flag_apCorr' , 'base_GaussianFlux_instFlux' , 'base_GaussianFlux_instFluxErr' , 'base_InputCount_flag' , 'base_InputCount_value' , 'base_PixelFlags_flag' , 'base_PixelFlags_flag_bright_object' , 'base_PixelFlags_flag_bright_objectCenter' , 'base_PixelFlags_flag_clipped' , 'base_PixelFlags_flag_clippedCenter' , 'base_PixelFlags_flag_cr' , 'base_PixelFlags_flag_crCenter' , 'base_PixelFlags_flag_edge' , 'base_PixelFlags_flag_inexact_psf' , 'base_PixelFlags_flag_inexact_psfCenter' , 'base_PixelFlags_flag_interpolated' , 'base_PixelFlags_flag_interpolatedCenter' , 'base_PixelFlags_flag_offimage' , 'base_PixelFlags_flag_rejected' , 'base_PixelFlags_flag_rejectedCenter' , 'base_PixelFlags_flag_saturated' , 'base_PixelFlags_flag_saturatedCenter' , 'base_PixelFlags_flag_sensor_edge' , 'base_PixelFlags_flag_sensor_edgeCenter' , 'base_PixelFlags_flag_suspect' , 'base_PixelFlags_flag_suspectCenter' , 'base_PsfFlux_apCorr' , 'base_PsfFlux_apCorrErr' , 'base_PsfFlux_area' , 'base_PsfFlux_flag' , 'base_PsfFlux_flag_apCorr' , 'base_PsfFlux_flag_edge' , 'base_PsfFlux_instFlux' , 'base_PsfFlux_instFluxErr' , 'base_SdssCentroid_flag' , 'base_SdssCentroid_x' , 'base_SdssCentroid_xErr' , 'base_SdssCentroid_y' , 'base_SdssCentroid_yErr' , 'base_SdssShape_flag' , 'base_SdssShape_flag_psf' , 'base_SdssShape_flag_shift' , 'base_SdssShape_instFlux' , 'base_SdssShape_instFluxErr' , 'base_SdssShape_psf_xx' , 'base_SdssShape_psf_xy' , 'base_SdssShape_psf_yy' , 'base_SdssShape_x' , 'base_SdssShape_xx' , 'base_SdssShape_xxErr' , 'base_SdssShape_xy' , 'base_SdssShape_xyErr' , 'base_SdssShape_y' , 'base_SdssShape_yy' , 'base_SdssShape_yyErr' , 'coord_dec' , 'coord_ra' , 'deblend_nChild' , 'ext_photometryKron_KronFlux_apCorr' , 'ext_photometryKron_KronFlux_apCorrErr' , 'ext_photometryKron_KronFlux_flag' , 'ext_photometryKron_KronFlux_flag_apCorr' , 'ext_photometryKron_KronFlux_flag_edge' , 'ext_photometryKron_KronFlux_instFlux' , 'ext_photometryKron_KronFlux_instFluxErr' , 'modelfit_CModel_apCorr' , 'modelfit_CModel_apCorrErr' , 'modelfit_CModel_flag' , 'modelfit_CModel_flag_apCorr' , 'modelfit_CModel_instFlux' , 'modelfit_CModel_instFluxErr' , 'modelfit_CModel_instFlux_inner' , 'parent' , 'slot_Centroid_flag' , 'slot_Centroid_x' , 'slot_Centroid_y' , 'slot_Shape_flag' , 'slot_Shape_xx' , 'slot_Shape_xy' , 'slot_Shape_yy' , 'tractId' , 'patchId' , 'detect_isPatchInner' , 'detect_isPrimary' , 'detect_isTractInner' , 'merge_measurement_N1010' , 'merge_measurement_N387' , 'merge_measurement_N515' , 'merge_measurement_N816' , 'merge_measurement_N921' , 'merge_measurement_g' , 'merge_measurement_i' , 'merge_measurement_i2' , 'merge_measurement_r' , 'merge_measurement_r2' , 'merge_measurement_y' , 'merge_measurement_z' , 'merge_peak_sky' , 'calib_astrometry_used' , 'calib_photometry_reserved' , 'calib_photometry_used' , 'calib_psf_candidate' , 'calib_psf_reserved' , 'calib_psf_used' , 'id' ] ** Note: I have since added the following deblender prefixes to this list in pipe_analysis : "deblend_parentNPeaks" , "deblend_nPeaks" , "deblend_scarletFlux" , "deblend_skipped" and you may want check with Arun Kannawadi as to which of the GAaP columns should be included. So far I have added "ext_gaap_GaapFlux_1_15x_Optimal_" to this list in pipe_analysis . *** Note: I have since added the following deblender prefixes to this list in pipe_analysis : "deblend_parentNPeaks" , "deblend_nPeaks" , "deblend_scarletFlux" , "deblend_skipped"

Lee Kelvin added a comment - 20/Sep/21 3:40 PM A few thoughts about circular aperture fluxes. Following our group meeting this morning, I've also added the circular aperture fluxes for all available radii to Lauren's list (specifically: 3.0, 4.5, 6.0, 17.0, 25.0, 35.0, 50.0, 70.0 pixels). The 9 pixel flag column was missing too, so I also requested this to be added at the same time. These should only be required in the case of forced tables (not meas). Following the existing naming conventions, columns have been named ApXFlux, where X is the radius value in pixels. This presents problems for the 4.5 pixel radius column. Would it instead be acceptable to use the lower-case letter 'p' to symbolize the decimal point, i.e., Ap3p0Flux, Ap4p5Flux, etc...? The 12 pixel radius aperture is treated differently, named ApFlux (no radius value given). This reflects the fact that this is the only aperture flux value that is treated differently, and calibrated to infinity. Considering I'm requesting adding a large number of additional radius columns into the object table, would it make sense to add Ap12Flux (or Ap12p0Flux) also as an uncalibrated column, for consistency?