Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35846

Investigate approaches for enforcing sdm_schema in the output of ap_association

    XMLWordPrintable

    Details

      Description

      My refactoring of DM-35331 broke ci_imsim because a field changed from int->float (pandas can do that on a whim). This reminded me that we need a way to validate the schema output by the ap_association (particularly the transform tasks). I don't know if there is an easy way to do that validation/enforcement outside of creating an APDB. At minimum, a test inside ap_association would be a start, but some more generic validation/enforcement would be better.

      Does DRP do schema validation on the output of their catalog transform tasks, and if so, how?

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35331 Add tests of sky source handling in transformDiaSourceCat

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            At minimum, a test inside ap_association would be a start, but some more generic validation/enforcement would be better.

            What does "more generic" mean? Are you proposing that the transform tasks check that their own output conforms to schema?

            Show
            krzys Krzysztof Findeisen added a comment - At minimum, a test inside ap_association would be a start, but some more generic validation/enforcement would be better. What does "more generic" mean? Are you proposing that the transform tasks check that their own output conforms to schema?
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Possibly that: I'm not sure what approaches are available.

            Show
            Parejkoj John Parejko added a comment - Possibly that: I'm not sure what approaches are available.
            Hide
            Permalink
            erykoff Eli Rykoff added a comment -

            Currently the only schema tests we have on the DRP side are similarly from ci_hsc and ci_imsim. I agree it would be nice to have something locally (in pipe_tasks for DRP).

            But this may be quite simple for this ticket. If ap_association depends on sdm_schemas, then to https://github.com/lsst/ap_association/blob/main/tests/test_transformDiaSourceCatalog.py e.g. you do this: https://github.com/lsst/ci_imsim/blob/main/tests/test_schema.py#L38-L40 and then something equivalent to https://github.com/lsst/ci_imsim/blob/main/tests/test_schema.py#L42 but using result.diaSourceTable directly.

            Show
            erykoff Eli Rykoff added a comment - Currently the only schema tests we have on the DRP side are similarly from ci_hsc and ci_imsim. I agree it would be nice to have something locally (in pipe_tasks for DRP). But this may be quite simple for this ticket. If ap_association depends on sdm_schemas , then to https://github.com/lsst/ap_association/blob/main/tests/test_transformDiaSourceCatalog.py e.g. you do this: https://github.com/lsst/ci_imsim/blob/main/tests/test_schema.py#L38-L40 and then something equivalent to https://github.com/lsst/ci_imsim/blob/main/tests/test_schema.py#L42 but using result.diaSourceTable directly.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Eli Rykoff, Eric Bellm, Ian Sullivan, John Parejko, Kenneth Herner, Krzysztof Findeisen, Meredith Rawls, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.