Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_analysis
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Sprint:DRP F20-2 (July), DRP F20-3 (Aug)
-
Team:Data Release Production
-
Urgent?:No
Description
In DM-12030, the persistence to parquet tables of the compiled per-tract(visit) catalogs assembled when running the pipe_analysis scripts was added to facilitate QA efforts external to pipe_analysis (these were originally intended for use with the interactive QA dashboard under development – see, e.g. DM-25395 – but, e.g., they were also used in the analysis of DM-25709 as a quicker full-tract input than the now-persisted per-patch obj_ parquet files). The (denormalized) catalogs created by matching the detected sources to an external reference catalog would also be of use for external QA analyses and validations, so they will be added as (optional) outputs on this ticket.
Attachments
Issue Links
- relates to
-
- Done
Activity
Happy for them to get more use! I do believe the will be available in the in-progress w_2020_36 reprocessing on DM-26637 (unless they locked in the pipe_analysis version before this got merged). If not, they are trivial to create for the coadds (as it's only 3 commands). Getting all the visits would require some more effort.
Are Jeffrey Carlin and Simon Krughoff aware of these new tables? Will you all be computing metrics based on the reference catalogs? And will the creation of these tables be separated from pipe_analysis at some point? Yusra AlSayyad and Jim Bosch encouraged me to move the dashboard away from using the pipe_analysis tables, but it would be useful to still have tables like this and other derived-quantity tables. There's probably a better forum to ask this question than this ticket; is there a ticket yet for creating tables like this (to replace the current specific functionality of the pipe_analysis parquet tables)?
We've been thinking about inputs to analysis as their own pipelines in the Gen3 sense. I.e. there are PipelineTask classes that produce products like this and persist them back to the butler repository. We currently have single band and multi-band matched catalog tasks, but catalogs matched to external catalogs are definitely something we would like to explore. It would be very interesting to see what it would take to make a PipelineTask that creates tables like this.
Yes, and do note the persistence of the tables on this ticket were added because of an immediate use-case for Eli and his FGCM fine-tuning validations. They are by no means meant to reflect any design strategies going forward (this was a very path-of-least-resistance way to get Eli quick & easy access to this info).
Hi guys- sorry I'm late to this party, but I just saw this go through; exciting! I'll try to work these tables into what I'm working on. Will they be available in the next RC2 run?