Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2739

integration test is too picky to be very useful

    Details

    • Type: Bug
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: lsst_dm_stack_demo
    • Labels:
      None
    • Team:
      SQuaRE

      Description

      The package lsst_dm_stack_demo is used by buildbot as an integration test by comparing an output file to an existing expected output data file. Unfortunately there are several problems with the test:

      • There is no statistical summary of the measured change, and in particular there is no report of the maximum error. When this test breaks we usually see tiny changes in many values. But if one or a few values have big changes (large enough to be really scary) we may not see this. In other words it is far too easy for minor innocuous changes to hide an crucial large change.
      • The data is written out using a number of digits that varies. I am not sure of all the causes of variability, but we have observed different files created on different systems. A recent commit (DM-2719) apparently caused the number of digits to change on buildbot, though I find it very hard to believe. The numbers match out to the number of digits written, but the number changed, resulting in a failed integration test.
      • Even if the data was written out to a consistent number of digits, the test should probably rely on fewer digits, as meaningless minor changes can cause failures.

      The result of is that we have many false negatives, which makes us take failures less seriously and wastes our time trying to figure out what is wrong when nothing is wrong.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            At risk of stating the obvious, we will soon be running other data sets, as well (which is partly useful because this particular demo is missing some key computations, such as ISR and astrometry). So whatever solution is used to make a more robust integration test here may well be wanted for those other data sets.

            Show
            rowen Russell Owen added a comment - At risk of stating the obvious, we will soon be running other data sets, as well (which is partly useful because this particular demo is missing some key computations, such as ISR and astrometry). So whatever solution is used to make a more robust integration test here may well be wanted for those other data sets.
            Hide
            frossie Frossie Economou added a comment -

            Absolutely. We are working to develop an integration harness that will run a variety of simulated and precursor data sets.

            Show
            frossie Frossie Economou added a comment - Absolutely. We are working to develop an integration harness that will run a variety of simulated and precursor data sets.
            Hide
            swinbank John Swinbank added a comment -

            It's been on my todo list now for some time to tweak the test script as part of RFC-72. I'm still hoping and intending to get to that shortly, although other urgent work keeps intruding. Would it be better to wait for whatever integration harness will be forthcoming?

            Show
            swinbank John Swinbank added a comment - It's been on my todo list now for some time to tweak the test script as part of RFC-72 . I'm still hoping and intending to get to that shortly, although other urgent work keeps intruding. Would it be better to wait for whatever integration harness will be forthcoming?
            Hide
            swinbank John Swinbank added a comment -

            I believe that the reworked approach to the lsst_dm_stack_demo in DM-3688 renders this ticket obsolete. I'll close this one when that's merged unless anybody objects.

            Show
            swinbank John Swinbank added a comment - I believe that the reworked approach to the lsst_dm_stack_demo in DM-3688 renders this ticket obsolete. I'll close this one when that's merged unless anybody objects.
            Hide
            tjenness Tim Jenness added a comment -

            DM-15044 definitely fixes the second problem described in this ticket. John Swinbank / Simon Krughoff should we follow through and close this ticket?

            Show
            tjenness Tim Jenness added a comment - DM-15044 definitely fixes the second problem described in this ticket. John Swinbank / Simon Krughoff should we follow through and close this ticket?
            Hide
            swinbank John Swinbank added a comment -

            Given that I said three years ago I'd close it when DM-3688 merged and nobody objected, I think we can close it for real now. Thanks for chasing, Tim Jenness.

            Show
            swinbank John Swinbank added a comment - Given that I said three years ago I'd close it when DM-3688 merged and nobody objected, I think we can close it for real now. Thanks for chasing, Tim Jenness .
            Hide
            swinbank John Swinbank added a comment -

            (Closing as Won't Fix since it's not a strict duplicate of any other ticket that I'm aware of.)

            Show
            swinbank John Swinbank added a comment - (Closing as Won't Fix since it's not a strict duplicate of any other ticket that I'm aware of.)

              People

              • Assignee:
                nidever David Nidever [X] (Inactive)
                Reporter:
                rowen Russell Owen
                Watchers:
                Dominique Boutigny, Frossie Economou, Jim Bosch, John Swinbank, Kian-Tat Lim, Russell Owen, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel