Tim Morton [X] (Inactive) added a comment - 17/Apr/18 10:29 PM - edited The following works for me now: writeObjectTable.py /datasets/hsc/repo/rerun/RC/w_2018_14/DM-13890/ --output /project/tmorton/DM-13877 --id tract=9615 filter=HSC-G^HSC-R --no-versions -j 18 writeQATable.py /project/tmorton/DM-13877 --output /project/tmorton/DM-13877 --id tract=9615 --no-versions -j 20 consolidateQATable.py /project/tmorton/DM-13877 --output /project/tmorton/DM-13877 --id tract=9615 --no-versions These tasks all use the new ParquetStorage to write and read (sometimes partial) dataframes, mediated by the ParquetTable and/or MultilevelParquetTable objects. WriteQATableTask also uses a YAML specification of what functors to calculate for the QA table. In principle this can be made configurable, but this is just a demo for now. Up next will be to connect this new tract-level dataset to the QADataset object to enable visualizations on top of these calculations.

Tim Morton [X] (Inactive) added a comment - 17/Apr/18 10:31 PM - edited For interest, the following is what the functor specfication looks like. The functor objects are imported from (and implemented in) functors.py . It is currently living next to the code in the package, which may not be the best place; I'm open to suggestions. x: functor: Mag args: base_PsfFlux label: functor: StarGalaxyLabeller funcs: cmodel_magDiff: functor: MagDiff args: - modelfit_CModel - base_PsfFlux gauss_magDiff: functor: MagDiff args: - base_GaussianFlux - base_PsfFlux count: functor: Column args: - base_InputCount_value seeing: functor: Seeing   deconvolved_moments: functor: DeconvolvedMoments flags: ref: - calib_psfUsed - merge_measurement_i - merge_measurement_r - merge_measurement_z - merge_measurement_y - merge_measurement_g - base_PixelFlags_flag_inexact_psfCenter meas: forced_src:

Tim Morton [X] (Inactive) added a comment - 17/Apr/18 11:36 PM Kian-Tat Lim , let me know if you can also review this one. I thought it would make sense for you to review both DM-13876 and this at once, since this is the first use case of the stuff in the first ticket. Sorry about the qa_explorer PR; it looks a bit messy probably because I rebased it off the other ticket branch.

Tim Morton [X] (Inactive) added a comment - 18/Apr/18 8:27 AM Kian-Tat Lim I've switched Jim Bosch to be the reviewer for this one, since he wants to look at it anyway. Jim Bosch , you should be good with these two ticket branches and with the DM-13876 ticket branch in daf_persistence.

Jim Bosch added a comment - 19/Apr/18 10:06 AM Review complete, comments on the PR. One thing I didn't mention there is that I would like to get a lot of this code out of qa_explorer  and into a new package - how does pipe_postprocessing sound?  That doesn't need to be done on this ticket, or even be done by you if you have other priorities before the end of the sprint, but I'd appreciate if you could open a ticket for it and identify there what you think should be moved vs. what is actually specific to the QA tools.