Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14552

centos6 4sigma outlier test numeric difference

    Details

    • Story Points:
      4
    • Sprint:
      AP S18-6
    • Team:
      Alert Production

      Description

      The added "4 sigma outlier rejection" test in jointcal produces different values on centos6, apparently failing to reject 2 outliers that are rejected on other architectures. This is causing jenkins failures in jointcal.

      I've installed a centos6 docker container and I'm going to try to reproduce the problem there and use it for debugging. Figuring out which outliers are not being rejected is the first step. Determining why is the next step.

      The "end of fit" numeric differences are:

      centos6:

      jointcal INFO: Fit completed with: chi2/ndof : 1509.83/3684=0.409833
      

      macOS/Ubuntu/centos7:

      jointcal INFO: Fit completed with: chi2/ndof : 1506.7/3682=0.409207
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            I just noticed that docker.io/lsstsqre/centos:6-stackbase-devtoolset-6 and docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 both still had the system gcc installed in addition to devtoolset-6. The system gcc was supposed to have gone away as part of the monthly rebuild of these images but it didn't happen because of a map key collision that was introduced during a refactor a month ago. This has been fixed and the images has been republished without the system gcc.

            (this should be a non-issue as long as the devtoolset-6 scl was being setup – which is the default for a login shell)

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited I just noticed that docker.io/lsstsqre/centos:6-stackbase-devtoolset-6 and docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 both still had the system gcc installed in addition to devtoolset-6 . The system gcc was supposed to have gone away as part of the monthly rebuild of these images but it didn't happen because of a map key collision that was introduced during a refactor a month ago. This has been fixed and the images has been republished without the system gcc. (this should be a non-issue as long as the devtoolset-6 scl was being setup – which is the default for a login shell)
            Show
            Parejkoj John Parejko added a comment - Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27990/pipeline
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I did test builds with 6-stackbase-devtoolset-7 and 7-stackbase-devtoolset-7 to see if the problem was specific to the centos 6 gcc 6 build. 7-dts-7 passed but 6-dts-7 failed with:

            AssertionError: True is not false : 1509.8253736858214 != 1506.7; diff=3.12537368582/1509.82537369=0.00207002328898 with rtol=1e-05, atol=2.220446049250313e-16 astrometry_final_chi2
            

            Show
            jhoblitt Joshua Hoblitt added a comment - I did test builds with 6-stackbase-devtoolset-7 and 7-stackbase-devtoolset-7 to see if the problem was specific to the centos 6 gcc 6 build. 7-dts-7 passed but 6-dts-7 failed with: AssertionError: True is not false : 1509.8253736858214 != 1506.7 ; diff= 3.12537368582 / 1509.82537369 = 0.00207002328898 with rtol=1e- 05 , atol= 2 .220446049250313e- 16 astrometry_final_chi2
            Hide
            Parejkoj John Parejko added a comment -

            Yes, that's the error. So it really does look like a problem specific to centos6.

            Show
            Parejkoj John Parejko added a comment - Yes, that's the error. So it really does look like a problem specific to centos6.
            Hide
            Parejkoj John Parejko added a comment -

            Fixed by disabling the decam test and adding cfht tests that accomplish the same thing. We can explore why decam is different on DM-14439.

            Simon Krughoff cleared the PR. Merged and done.

            Show
            Parejkoj John Parejko added a comment - Fixed by disabling the decam test and adding cfht tests that accomplish the same thing. We can explore why decam is different on DM-14439 . Simon Krughoff cleared the PR. Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Joshua Hoblitt, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel