Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5005

Please trim config overrides in validate_drp

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:
      None
    • Story Points:
      0.5
    • Team:
      Alert Production

      Description

      validate_drp will test more of our code if it uses default config parameters wherever possible. To that effect I would like to ask you to eliminate all config overrides that are not essential and document the reasons for the remaining overrides.

      For DECam there are no overrides that are different than the defaults, so the file can simply be emptied (for now).

      For CFHT there are many overrides that are different, and an important question is whether the overrides in this package are better for CFHT data than the overrides in obs_cfht; if so, please move them to obs_cfht.

      As a heads up: the default star selector is changing from "secondMoment" to "objectSize" in DM-4692 and I hope to allow that in validate_drp, since it works better and is better supported.

      Sorry for the incorrect component, but validate_drp is not yet a supported component in JIRA (see DM-5004)

        Attachments

          Activity

          Hide
          rowen Russell Owen added a comment -

          I just pushed a change to obs_cfht master that tweaks a few params for improved performance. I carefully checked the output of validate_drp before doing this and it looks fine (though I encourage you to verify this). So at this point CFHT needs no config overrides on master.

          I think the DECam overrides were already unnecessary, but again, I encourage you to verify that.

          I suggest leaving a default override file around if you plan to keep offering the examples/runX scripts, to make it easier for users to try out the effects of temporary config overrides.

          In my opinion this is a fairly important change, because it makes the results easier to understand.

          Show
          rowen Russell Owen added a comment - I just pushed a change to obs_cfht master that tweaks a few params for improved performance. I carefully checked the output of validate_drp before doing this and it looks fine (though I encourage you to verify this). So at this point CFHT needs no config overrides on master. I think the DECam overrides were already unnecessary, but again, I encourage you to verify that. I suggest leaving a default override file around if you plan to keep offering the examples/runX scripts, to make it easier for users to try out the effects of temporary config overrides. In my opinion this is a fairly important change, because it makes the results easier to understand.
          Hide
          rowen Russell Owen added a comment - - edited

          Here it the output for examples/runCfhtTest.sh with tickets/DM-5005:

          1295 sources in ccd:  12
          1301 sources in ccd:  13
          1279 sources in ccd:  14
          1309 sources in ccd:  21
          1389 sources in ccd:  22
          1438 sources in ccd:  23
          2062 sources in ccd:  12
          2164 sources in ccd:  13
          2441 sources in ccd:  14
          2179 sources in ccd:  21
          2342 sources in ccd:  22
          2440 sources in ccd:  23
          Median value of the astrometric scatter - all magnitudes: 59.659 mas
          Astrometric scatter (median) - mag < 21.0 : 9.8 mas
          Median value of the photometric scatter - all magnitudes: 116.444 mmag
          Photometric scatter (median) - mag < 21.0 : 13.2 mmag
          No stars found that are 199.0--201.0 arcmin apart.
          PA1(RMS) = 18.09+-0.00 mmag
          PA1(IQR) = 18.65+-0.57 mmag
          minimum: PF1=20% of diffs more than PA2 = 22.73 mmag (target is < 15 mmag)
          design : PF1=10% of diffs more than PA2 = 29.00 mmag (target is < 15 mmag)
          stretch: PF1= 5% of diffs more than PA2 = 35.32 mmag (target is < 10 mmag)
          Median of distribution of RMS of distance of stellar pairs.
          DESIGN goals
          For stars from 17.00 < mag < 21.50
          from D = [4.00, 6.00] arcmin, AM1=7.77 mas (target is < 10 mas).
            6.32% of sample deviates by >30 mas (target is < 10%)
          Median of distribution of RMS of distance of stellar pairs.
          DESIGN goals
          For stars from 17.00 < mag < 21.50
          from D = [19.00, 21.00] arcmin, AM2=7.15 mas (target is < 10 mas).
            5.46% of sample deviates by >30 mas (target is < 10%)
          

          which gives better astrometric results and essentially identical photometric scatter to master (the astrometric improvement is expected due to tickets/DM-5005 using the newer, default, astrometry task):

          1295 sources in ccd:  12
          1301 sources in ccd:  13
          1279 sources in ccd:  14
          1309 sources in ccd:  21
          1389 sources in ccd:  22
          1438 sources in ccd:  23
          2062 sources in ccd:  12
          2164 sources in ccd:  13
          2441 sources in ccd:  14
          2179 sources in ccd:  21
          2342 sources in ccd:  22
          2440 sources in ccd:  23
          Median value of the astrometric scatter - all magnitudes: 60.449 mas
          Astrometric scatter (median) - mag < 21.0 : 12.0 mas
          Median value of the photometric scatter - all magnitudes: 112.331 mmag
          Photometric scatter (median) - mag < 21.0 : 13.1 mmag
          No stars found that are 199.0--201.0 arcmin apart.
          PA1(RMS) = 20.34+-0.00 mmag
          PA1(IQR) = 18.16+-0.58 mmag
          minimum: PF1=20% of diffs more than PA2 = 24.96 mmag (target is < 15 mmag)
          design : PF1=10% of diffs more than PA2 = 35.15 mmag (target is < 15 mmag)
          stretch: PF1= 5% of diffs more than PA2 = 38.15 mmag (target is < 10 mmag)
          Median of distribution of RMS of distance of stellar pairs.
          DESIGN goals
          For stars from 17.00 < mag < 21.50
          from D = [4.00, 6.00] arcmin, AM1=9.65 mas (target is < 10 mas).
            5.25% of sample deviates by >30 mas (target is < 10%)
          Median of distribution of RMS of distance of stellar pairs.
          DESIGN goals
          For stars from 17.00 < mag < 21.50
          from D = [19.00, 21.00] arcmin, AM2=8.72 mas (target is < 10 mas).
            3.79% of sample deviates by >30 mas (target is < 10%)
          

          DECam results are unaffected by the change with one exception whose change is well within sigma:

          PA1(IQR) = 29.50+-1.04 mmag # DM-4962
          PA1(IQR) = 29.32+-0.95 mmag # master
          

          Show
          rowen Russell Owen added a comment - - edited Here it the output for examples/runCfhtTest.sh with tickets/ DM-5005 : 1295 sources in ccd: 12 1301 sources in ccd: 13 1279 sources in ccd: 14 1309 sources in ccd: 21 1389 sources in ccd: 22 1438 sources in ccd: 23 2062 sources in ccd: 12 2164 sources in ccd: 13 2441 sources in ccd: 14 2179 sources in ccd: 21 2342 sources in ccd: 22 2440 sources in ccd: 23 Median value of the astrometric scatter - all magnitudes: 59.659 mas Astrometric scatter (median) - mag < 21.0 : 9.8 mas Median value of the photometric scatter - all magnitudes: 116.444 mmag Photometric scatter (median) - mag < 21.0 : 13.2 mmag No stars found that are 199.0--201.0 arcmin apart. PA1(RMS) = 18.09+-0.00 mmag PA1(IQR) = 18.65+-0.57 mmag minimum: PF1=20% of diffs more than PA2 = 22.73 mmag (target is < 15 mmag) design : PF1=10% of diffs more than PA2 = 29.00 mmag (target is < 15 mmag) stretch: PF1= 5% of diffs more than PA2 = 35.32 mmag (target is < 10 mmag) Median of distribution of RMS of distance of stellar pairs. DESIGN goals For stars from 17.00 < mag < 21.50 from D = [4.00, 6.00] arcmin, AM1=7.77 mas (target is < 10 mas). 6.32% of sample deviates by >30 mas (target is < 10%) Median of distribution of RMS of distance of stellar pairs. DESIGN goals For stars from 17.00 < mag < 21.50 from D = [19.00, 21.00] arcmin, AM2=7.15 mas (target is < 10 mas). 5.46% of sample deviates by >30 mas (target is < 10%) which gives better astrometric results and essentially identical photometric scatter to master (the astrometric improvement is expected due to tickets/ DM-5005 using the newer, default, astrometry task): 1295 sources in ccd: 12 1301 sources in ccd: 13 1279 sources in ccd: 14 1309 sources in ccd: 21 1389 sources in ccd: 22 1438 sources in ccd: 23 2062 sources in ccd: 12 2164 sources in ccd: 13 2441 sources in ccd: 14 2179 sources in ccd: 21 2342 sources in ccd: 22 2440 sources in ccd: 23 Median value of the astrometric scatter - all magnitudes: 60.449 mas Astrometric scatter (median) - mag < 21.0 : 12.0 mas Median value of the photometric scatter - all magnitudes: 112.331 mmag Photometric scatter (median) - mag < 21.0 : 13.1 mmag No stars found that are 199.0--201.0 arcmin apart. PA1(RMS) = 20.34+-0.00 mmag PA1(IQR) = 18.16+-0.58 mmag minimum: PF1=20% of diffs more than PA2 = 24.96 mmag (target is < 15 mmag) design : PF1=10% of diffs more than PA2 = 35.15 mmag (target is < 15 mmag) stretch: PF1= 5% of diffs more than PA2 = 38.15 mmag (target is < 10 mmag) Median of distribution of RMS of distance of stellar pairs. DESIGN goals For stars from 17.00 < mag < 21.50 from D = [4.00, 6.00] arcmin, AM1=9.65 mas (target is < 10 mas). 5.25% of sample deviates by >30 mas (target is < 10%) Median of distribution of RMS of distance of stellar pairs. DESIGN goals For stars from 17.00 < mag < 21.50 from D = [19.00, 21.00] arcmin, AM2=8.72 mas (target is < 10 mas). 3.79% of sample deviates by >30 mas (target is < 10%) DECam results are unaffected by the change with one exception whose change is well within sigma: PA1(IQR) = 29.50+-1.04 mmag # DM-4962 PA1(IQR) = 29.32+-0.95 mmag # master
          Hide
          rowen Russell Owen added a comment -

          All changes on in validate_drp tickets/DM-5005

          Show
          rowen Russell Owen added a comment - All changes on in validate_drp tickets/ DM-5005
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          This looks good.

          There's one really minor documentation clarification suggestion on the PR.

          Merge when ready.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - This looks good. There's one really minor documentation clarification suggestion on the PR. Merge when ready.
          Hide
          rowen Russell Owen added a comment -

          Thanks for catching the documentation issue. Fixed, rebased and merged to master.

          Show
          rowen Russell Owen added a comment - Thanks for catching the documentation issue. Fixed, rebased and merged to master.

            People

            • Assignee:
              rowen Russell Owen
              Reporter:
              rowen Russell Owen
              Reviewers:
              Michael Wood-Vasey
              Watchers:
              Michael Wood-Vasey, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: