Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Labels:None
-
Story Points:3
-
Team:Data Release Production
Description
We need to include fields added by other tasks (e.g. the deblender) or the source minimal schema (e.g. parent) in transformed catalogs. I believe it should be safe to assume any such fields can simply be copied (i.e. they do not require any actual transformation - they're mostly flags), but we do need to make the list of fields to be copied by this mechanism dynamic.
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
copyFields can do what we need, but using it in this role leads to very fragile configuration, because we have to list a number of fields in config that we'd ideally just get from the original task configuration or schema somehow. I do think that makes this a lower priority, though, and I've adjusted it accordingly.
I agree that this configuration is inelegant, but I'm not immediately sure how we could do better without significant work.
I'm reluctant to add anything explicit to the configuration of the task that generated the catalog, since this isn't configuration for that task but for the transformation.
Nicer would be to manipulate the schema. Ideally it would be possible to add some sort of indication to individual fields that they should be copied, but I don't think afw::table supports that sort of annotation (other than by overloading the doc strings). I may have missed something though, so I'll dig into that a bit further.
A third option might be to add a static method to the measurement task itself which returns a list of fields to copy (in much the same way as we do for the measurement plugins at the moment). Again, this feels quite invasive on the source tasks, but maybe it's the best option.
Suggestions welcome; I'll give this some more thought when I have time.
I don't have any good ideas either. One half-formed thought is that we may want to approach this from the perspective of copying all fields that aren't explicitly handled by measurement transformers; that would "just" require adding a way for those transformers to claim the fields they're handling.
Yes, sounds like that approach may well be the most feasible.
Catalog transformation is obsolete given ongoing work on the SDM standardization system (see also DM-3348).
TransformTask already supports copying a configurable list of fields from the input to the output – see the copyFields configuration parameter. Does that do what you need, or am I misunderstanding?
(I note that copyFields is documented in terms of having a doc parameter, but that doesn't seem to appear anywhere obvious on Doxygen. I should look at fixing that.)