Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3590

Catalog transformation should pass through all non-measurement fields

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      3
    • Team:
      Data Release Production

      Description

      We need to include fields added by other tasks (e.g. the deblender) or the source minimal schema (e.g. parent) in transformed catalogs. I believe it should be safe to assume any such fields can simply be copied (i.e. they do not require any actual transformation - they're mostly flags), but we do need to make the list of fields to be copied by this mechanism dynamic.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment - - edited

            TransformTask already supports copying a configurable list of fields from the input to the output – see the copyFields configuration parameter. Does that do what you need, or am I misunderstanding?

            (I note that copyFields is documented in terms of having a doc parameter, but that doesn't seem to appear anywhere obvious on Doxygen. I should look at fixing that.)

            Show
            swinbank John Swinbank added a comment - - edited TransformTask already supports copying a configurable list of fields from the input to the output – see the copyFields configuration parameter. Does that do what you need, or am I misunderstanding? (I note that copyFields is documented in terms of having a doc parameter, but that doesn't seem to appear anywhere obvious on Doxygen. I should look at fixing that.)
            Hide
            jbosch Jim Bosch added a comment -

            copyFields can do what we need, but using it in this role leads to very fragile configuration, because we have to list a number of fields in config that we'd ideally just get from the original task configuration or schema somehow. I do think that makes this a lower priority, though, and I've adjusted it accordingly.

            Show
            jbosch Jim Bosch added a comment - copyFields can do what we need, but using it in this role leads to very fragile configuration, because we have to list a number of fields in config that we'd ideally just get from the original task configuration or schema somehow. I do think that makes this a lower priority, though, and I've adjusted it accordingly.
            Hide
            swinbank John Swinbank added a comment -

            I agree that this configuration is inelegant, but I'm not immediately sure how we could do better without significant work.

            I'm reluctant to add anything explicit to the configuration of the task that generated the catalog, since this isn't configuration for that task but for the transformation.

            Nicer would be to manipulate the schema. Ideally it would be possible to add some sort of indication to individual fields that they should be copied, but I don't think afw::table supports that sort of annotation (other than by overloading the doc strings). I may have missed something though, so I'll dig into that a bit further.

            A third option might be to add a static method to the measurement task itself which returns a list of fields to copy (in much the same way as we do for the measurement plugins at the moment). Again, this feels quite invasive on the source tasks, but maybe it's the best option.

            Suggestions welcome; I'll give this some more thought when I have time.

            Show
            swinbank John Swinbank added a comment - I agree that this configuration is inelegant, but I'm not immediately sure how we could do better without significant work. I'm reluctant to add anything explicit to the configuration of the task that generated the catalog, since this isn't configuration for that task but for the transformation. Nicer would be to manipulate the schema. Ideally it would be possible to add some sort of indication to individual fields that they should be copied, but I don't think afw::table supports that sort of annotation (other than by overloading the doc strings). I may have missed something though, so I'll dig into that a bit further. A third option might be to add a static method to the measurement task itself which returns a list of fields to copy (in much the same way as we do for the measurement plugins at the moment). Again, this feels quite invasive on the source tasks, but maybe it's the best option. Suggestions welcome; I'll give this some more thought when I have time.
            Hide
            jbosch Jim Bosch added a comment -

            I don't have any good ideas either. One half-formed thought is that we may want to approach this from the perspective of copying all fields that aren't explicitly handled by measurement transformers; that would "just" require adding a way for those transformers to claim the fields they're handling.

            Show
            jbosch Jim Bosch added a comment - I don't have any good ideas either. One half-formed thought is that we may want to approach this from the perspective of copying all fields that aren't explicitly handled by measurement transformers; that would "just" require adding a way for those transformers to claim the fields they're handling.
            Hide
            swinbank John Swinbank added a comment -

            Yes, sounds like that approach may well be the most feasible.

            Show
            swinbank John Swinbank added a comment - Yes, sounds like that approach may well be the most feasible.
            Hide
            swinbank John Swinbank added a comment -

            Catalog transformation is obsolete given ongoing work on the SDM standardization system (see also DM-3348).

            Show
            swinbank John Swinbank added a comment - Catalog transformation is obsolete given ongoing work on the SDM standardization system (see also DM-3348 ).

              People

              Assignee:
              swinbank John Swinbank
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.