Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23103

Update functor unittests to no longer rely on test_multilevel_parq.csv.gz

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      test_functors.py in pipe_tasks relies on a dataset with fixed columns creating problems if new columns/values are needed for newly created functors. This ticket will modify the unittesting to be more similar (and reuse some of the code from) the test_parquet.py suite, creating the dataframes/parquet tables as needed within each test allowing control over a) the columns created b) the values input which in turn will allow for testing of a specific funtor's algorithm/calculation.

       

        Attachments

          Issue Links

            Activity

            Hide
            cmorrison Chris Morrison added a comment -

            Found bugs in ParquetTable that caused to certain methods to crash when the table was initialized with a DataFrame instead of a parquet file. These have been fixed by reording some methods and making use of some already written but not called "sanitation" methods.

            Show
            cmorrison Chris Morrison added a comment - Found bugs in ParquetTable that caused to certain methods to crash when the table was initialized with a DataFrame instead of a parquet file. These have been fixed by reording some methods and making use of some already written but not called "sanitation" methods.
            Show
            cmorrison Chris Morrison added a comment - - edited Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31158/pipeline/47/
            Hide
            tmorton Tim Morton added a comment -

            Comments on PR-- main concern is with how the fake "data" are being generated, and what that means to test for (in)equality in some of the tests.

            Show
            tmorton Tim Morton added a comment - Comments on PR-- main concern is with how the fake "data" are being generated, and what that means to test for (in)equality in some of the tests.
            Hide
            tmorton Tim Morton added a comment -

            Looks good! And I approve of future tickets to flesh out the content of these tests to make sure that functors compute the correct things.

            Show
            tmorton Tim Morton added a comment - Looks good! And I approve of future tickets to flesh out the content of these tests to make sure that functors compute the correct things.

              People

              • Assignee:
                cmorrison Chris Morrison
                Reporter:
                cmorrison Chris Morrison
                Reviewers:
                Tim Morton
                Watchers:
                Chris Morrison, Eric Bellm, John Swinbank, Tim Morton, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: