Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: afw
-
Labels:
-
Story Points:4
-
Epic Link:
-
Sprint:AP S19-5
-
Team:Alert Production
Description
The two `countsTo*` methods that operate in-place on SourceCatalogs in PhotoCalib are marked "not implemented" due RFC-322. Once that RFC is completed and the necessary key name changes are in place, we can finally implement those methods.
We might want to consolidate them into a single countsToFluxAndMagnitude, since if we're iterating over the catalog anyway, might as well do all the math at once.
Attachments
Issue Links
- is blocked by
-
DM-10153 Replace Calib with finished PhotoCalib
- Done
- is duplicated by
-
- Invalid
- is triggering
-
DM-19456 fix docstring for PhotoCalib.calibrateCatalog
- Done
- relates to
-
- Won't Fix
-
- Won't Fix
-
- Done
Activity
I also ran into this when writing a “calibrate catalog using PhotoCalib” function for pipe_analysis. I ended up doing the kind of hack-y thing for any *instFlux field that did not have an associated *instFluxErr described here: https://github.com/lsst-dm/pipe_analysis/blob/master/python/lsst/pipe/analysis/utils.py#L941-L957
I think we want to be able to just handle instFlux fields without errors by calibrating them without magnitude/calibrated-flux errors.
As part of this work, I'm also greatly accelerating the other PhotoCalib SourceCatalog operations by vectorizing them to make AST-backed PhotoCalibs much faster. Looks like roughly a factor of >15 for TransformBoundedField, and little change for constant PhotoCalibs.
Jim Bosch: can you please review this (about 250 lines)? I took your advice above re: fields without errors.
It isn't as fast as I'd hoped: it takes about 30ms to calibrate every field on my test catalog (photoCalib.calibrateCatalog(src)) and 22ms to do just one field (photoCalib.calibrateCatalog(src)), which tells me the "find the fields" part isn't taking much time. The `evaluateCatalog` step should take about 1ms, given the performance of e.g. photoCalib.instFluxToMagnitude(src, 'slot_PsfFlux'). I've attached my profiling script, if you have any ideas.
Looks good!
My best guess at where the time is going is that it's in the pybind11 layer, but that really is just a guess.
It looks like most of that time is going into BaseTable::copyRecord(record, schemaMapper), as we guessed on the phone. No real way to avoid that, I don't think.
Thank you for the review suggestions: I think the result is much cleaner.
Merged and done.
This is close, but I just tested it on an stack-produced afw table, and the "do all the columns" version fails because of base_Blendedness_abs_child_instFlux, which doesn't have a corresponding *_instFluxErr field. I wonder if I should add a check for the word "Blendedness" in the field name, or just ignore fields that don't have error, or what?