Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23045

Validate linearity correction

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe, obs_base
    • Labels:
      None

      Description

      Demonstrate that the linearity model persisted as per DM-23044 (and generated as per DM-21221) are scientifically adequate. File tickets to address glaring algorithmic flaws.

        Attachments

        1. auxtel-NL-fractional-residuals.png
          auxtel-NL-fractional-residuals.png
          702 kB
        2. BOT_2020APR11.png
          BOT_2020APR11.png
          920 kB
        3. convertFitsToYaml.py
          0.4 kB
        4. lin-res-rms-and-lut-from-poly4-added.png
          lin-res-rms-and-lut-from-poly4-added.png
          631 kB
        5. no-linearity-correction.png
          no-linearity-correction.png
          700 kB
        6. non-linearity-auxtel-polynomial-2020APR06.png
          non-linearity-auxtel-polynomial-2020APR06.png
          459 kB
        7. PTC_det0-no-lin.pdf
          78 kB
        8. PTC_det0-yes-lin.pdf
          78 kB
        9. PTC_det95_one_filter_lin_false.pdf
          228 kB
        10. PTC_det95_one_filter_lin_true.pdf
          229 kB
        11. yes-linearity-correction.png
          yes-linearity-correction.png
          707 kB

          Issue Links

            Activity

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins tests: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31541/pipeline
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            OK, I've done the code part of the review, just need to look over the science part now, but I think that if there are problems that we said they'd be addressed elsewhere, so given that this isn't changing what's be calculated, only what's plotted, there's no reason not to merge this and file other tickets once the review comments are dealt with.

            More on the science part soon...

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - OK, I've done the code part of the review, just need to look over the science part now, but I think that if there are problems that we said they'd be addressed elsewhere, so given that this isn't changing what's be calculated, only what's plotted, there's no reason not to merge this and file other tickets once the review comments are dealt with. More on the science part soon...
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            OK, I think that's looking pretty good from a fairly quick look!

            The initial residuals don't look all the well described by a low order poly, so I'd never expect it to be amazing, so I think that's a pretty good start.

            It would be fun (in another ticket) to build a LUT from the data, and then check that, when running on the same data, the residuals are basically zero, but I'm certainly pretty convinced by that last plot that the 3rd and 4th order polys are improving things by quite a bit.

            Again, in another ticket, perhaps plot a) the RMS residual as a function of increasing poly order (including the pre-corrected value too), and b) the quality of fit as that increases, as it will start to be weird and unpredictable at high orders, I think.

            Probably worth Robert Lupton casting his eye over this too, but I think this is a good start.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - OK, I think that's looking pretty good from a fairly quick look! The initial residuals don't look all the well described by a low order poly, so I'd never expect it to be amazing, so I think that's a pretty good start. It would be fun (in another ticket) to build a LUT from the data, and then check that, when running on the same data, the residuals are basically zero, but I'm certainly pretty convinced by that last plot that the 3rd and 4th order polys are improving things by quite a bit. Again, in another ticket, perhaps plot a) the RMS residual as a function of increasing poly order (including the pre-corrected value too), and b) the quality of fit as that increases, as it will start to be weird and unpredictable at high orders, I think. Probably worth Robert Lupton casting his eye over this too, but I think this is a good start.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Sorry, forgot to mark that as review complete before, but it is.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Sorry, forgot to mark that as review complete before, but it is.
            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Latest Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31593/pipeline

              People

              • Assignee:
                plazas Andrés Alejandro Plazas Malagón
                Reporter:
                swinbank John Swinbank
                Reviewers:
                Merlin Fisher-Levine
                Watchers:
                Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine, Robert Lupton, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel