Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_subaru, pipe_analysis
-
Labels:
-
Story Points:1
-
Epic Link:
-
Sprint:DRP F17-6
-
Team:Data Release Production
Description
In order to make interactive QA plots such as DM-11682, we need to persist parquet files containing the compiled data that are used to make the static matplotlib plots. Admittedly, this should be a temporary step until all the source info is easily loadable from a column-store database, but for now it is necessary. At the visit level it might be less so, but I'll do it anyway. This is probably technically part of DM-10859, but I thought it might be useful to get it in its own ticket so it can be merged quickly.
Attachments
Issue Links
- blocks
-
DM-10859 Interface prototype Bokeh QA system with HSC data
- Done
-
DM-12043 Run pipe_analysis scripts on PDR1 data
- Done
- is duplicated by
-
DM-12084 A unit test is broken in obs_test
- Invalid
- is triggering
-
DM-12084 A unit test is broken in obs_test
- Invalid
- relates to
-
DM-26082 Persist source-to-external reference matched catalogs in pipe_analysis to parquet
- Done
-
DM-18353 Add pre-computed values to parquet tables output by pipe_analysis scripts
- Done
Just remembered that in order to write these tables, you need to have the fastparquet python library. I did a pip install --user fastparquet, but perhaps this should be conda install'ed to the stack before merging? Or should I hide the fastparquet import behind a warning? Thoughts, John Swinbank?