Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14552

centos6 4sigma outlier test numeric difference

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Sprint:
      AP S18-6
    • Team:
      Alert Production

      Description

      The added "4 sigma outlier rejection" test in jointcal produces different values on centos6, apparently failing to reject 2 outliers that are rejected on other architectures. This is causing jenkins failures in jointcal.

      I've installed a centos6 docker container and I'm going to try to reproduce the problem there and use it for debugging. Figuring out which outliers are not being rejected is the first step. Determining why is the next step.

      The "end of fit" numeric differences are:

      centos6:

      jointcal INFO: Fit completed with: chi2/ndof : 1509.83/3684=0.409833

      macOS/Ubuntu/centos7:

      jointcal INFO: Fit completed with: chi2/ndof : 1506.7/3682=0.409207

        Attachments

          Issue Links

          is triggered by

          Improvement - An improvement or enhancement to an existing feature or task. DM-13671 Add option to skip cholmod.update step

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14549 Explore compiler optimization options

          • Won't Fix

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14439 Explore numeric difference in Mac/Linux ConstrainedPhotometry fitting

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            9 older comments
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment - - edited

            I just noticed that docker.io/lsstsqre/centos:6-stackbase-devtoolset-6 and docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 both still had the system gcc installed in addition to devtoolset-6. The system gcc was supposed to have gone away as part of the monthly rebuild of these images but it didn't happen because of a map key collision that was introduced during a refactor a month ago. This has been fixed and the images has been republished without the system gcc.

            (this should be a non-issue as long as the devtoolset-6 scl was being setup – which is the default for a login shell)

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited I just noticed that docker.io/lsstsqre/centos:6-stackbase-devtoolset-6 and docker.io/lsstsqre/centos:7-stackbase-devtoolset-6 both still had the system gcc installed in addition to devtoolset-6 . The system gcc was supposed to have gone away as part of the monthly rebuild of these images but it didn't happen because of a map key collision that was introduced during a refactor a month ago. This has been fixed and the images has been republished without the system gcc. (this should be a non-issue as long as the devtoolset-6 scl was being setup – which is the default for a login shell)
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27990/pipeline

            Show
            Parejkoj John Parejko added a comment - Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/27990/pipeline
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            I did test builds with 6-stackbase-devtoolset-7 and 7-stackbase-devtoolset-7 to see if the problem was specific to the centos 6 gcc 6 build. 7-dts-7 passed but 6-dts-7 failed with:

            AssertionError: True is not false : 1509.8253736858214 != 1506.7; diff=3.12537368582/1509.82537369=0.00207002328898 with rtol=1e-05, atol=2.220446049250313e-16 astrometry_final_chi2

            Show
            jhoblitt Joshua Hoblitt added a comment - I did test builds with 6-stackbase-devtoolset-7 and 7-stackbase-devtoolset-7 to see if the problem was specific to the centos 6 gcc 6 build. 7-dts-7 passed but 6-dts-7 failed with: AssertionError: True is not false : 1509.8253736858214 != 1506.7 ; diff= 3.12537368582 / 1509.82537369 = 0.00207002328898 with rtol=1e- 05 , atol= 2 .220446049250313e- 16 astrometry_final_chi2
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Yes, that's the error. So it really does look like a problem specific to centos6.

            Show
            Parejkoj John Parejko added a comment - Yes, that's the error. So it really does look like a problem specific to centos6.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Fixed by disabling the decam test and adding cfht tests that accomplish the same thing. We can explore why decam is different on DM-14439.

            Simon Krughoff cleared the PR. Merged and done.

            Show
            Parejkoj John Parejko added a comment - Fixed by disabling the decam test and adding cfht tests that accomplish the same thing. We can explore why decam is different on DM-14439 . Simon Krughoff cleared the PR. Merged and done.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Jim Bosch, John Parejko, John Swinbank, Joshua Hoblitt, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.