Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34703

Write (offline) regression test for quickFrameMeasurementTask

    XMLWordPrintable

Details

    • 5
    • Ops Pipelines 2023
    • Data Release Production
    • No

    Description

      QFM needs extending, and that will likely involve a significant refactor. The only way to test that it's still working robustly is to have a list of dataIds and human-confirmed results and run a cross check with the new code.

      This ticket is to make a list of dataIds and the current results, and confirm that the table of results are correct by eye, and write a [notebook? script? something...] that can be run manually, with the data file produced, that reruns all the dataIds and checks that the results haven't changed.

      Attachments

        Issue Links

          Activity

            csaunder Clare Saunders added a comment - - edited

            A few comments on the current status of the tool:

            • To rerun all 6105 samples on 15 cores took 20 minutes on the USDF login node for me. Because it is relatively quick, I decided not to bother with any interim saving. This can obviously be added if it starts to be an issue.
            • I made the "baseline" file in a notebook on the RSP, and the runtime seems to have been systematically faster there, so I think I'm going to rerun on the USDF login node and adjust the baseline runtimes.
            • I considered trying to capture the quickFrameMeasurement logs and include them in the comparison. Let me know if this would be helpful.
            • Lastly, the comparison report just prints to screen right now, and I didn't bother with any formal logging. Obviously this can be changed if needed, or I can make a more machine-readable output.
            csaunder Clare Saunders added a comment - - edited A few comments on the current status of the tool: To rerun all 6105 samples on 15 cores took 20 minutes on the USDF login node for me. Because it is relatively quick, I decided not to bother with any interim saving. This can obviously be added if it starts to be an issue. I made the "baseline" file in a notebook on the RSP, and the runtime seems to have been systematically faster there, so I think I'm going to rerun on the USDF login node and adjust the baseline runtimes. I considered trying to capture the quickFrameMeasurement logs and include them in the comparison. Let me know if this would be helpful. Lastly, the comparison report just prints to screen right now, and I didn't bother with any formal logging. Obviously this can be changed if needed, or I can make a more machine-readable output.

            People

              csaunder Clare Saunders
              mfisherlevine Merlin Fisher-Levine
              Joshua Meyers
              Clare Saunders, Joshua Meyers, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.