Added two linearizers to ip_isr: one for a squared model (obs_subaru) and one for lookup tables (probably DECam and LSST). Linearizers are treated as calibration data products, but they are functors. The squared linearizer gets its coefficient from the amp info, so there is only one and the Subaru camera mapper provides it (the same one regardless of data ID). The table-based linearizer stores the table and a few other coefficients, so users will persist (pickle) one per CCD in the data repo, and this will require an update to the camera mapper paf file.

I updated the subaru-specific ISR task and mapper to use the squared linearizer. It would be useful to run ci_hsc on master and this branch to confirm that nothing changes. Unfortunately I can't get it to run on El Capitan. Of course the unit tests pass (and a Jenkins run passed).

I added support for not applying linearity correction to a whole detector using linearity type = lsst.afw.cameraGeom.NullLinearityType. I then set the linearity type to that value in all amp info catalogs in obs_cfht, obs_decam, obs_lsstSim and obs_test, and set the linearity type LinearitySquared.LinearityType or NullLinearityType, as appropriate (the latter only if all squared coefficients are 0) for obs_subaru.

The reason NullLinearityType is in afw is to avoid circular imports for obs_test.

An overview of the system is available here: https://community.lsst.org/t/correcting-non-linearity/816

I have checked the code as follows:

• Jenkins CI
• validate_drp running the quick tests on DECam and CFHT
• ci_hsc for Subaru

Again, only obs_subaru presently uses non-linearity correction and the new code is designed to provide corrections that is identical to the old code.

I've added review comments on the github PRs. I have a few major concerns:

• The testing compares the output of the implementations to "reference implementations", but these "reference implementations" have no pedigree so we don't have any confidence that they are correct. More complete tests should be added.
• The functors that apply the linearity correction are iterating over amplifiers. I think the implementation would be cleaner if the functor was responsible for a single amplifier, and it is the caller's responsibility to iterate over amplifiers. That's how overscan subtraction works.
• There is a lot of code duplication in the tests in ip_isr that should be removed by refactoring.
• Pixels that are out of range of the non-linearity correction aren't masked. How do we know which pixels aren't to be trusted?

In obs_subaru (reported here because it's difficult to review on github due to the large number of changes to binary files):

• Is it possible to separate the commit in obs_subaru into smaller parts?
• self.log.log(self.log.INFO, --> self.log.info(

Your points about the unit tests are well taken. I think I have fully addressed them as follows:

• I added tests on small images with known expected results.
• I moved BoxGrid (renamed from BoxArr to afw.geom.testUtils and makeRampImage to afw.image.testUtils and added unit tests for them. I also enhanced BoxGrid to work with floating point boxes (Box2D.

Regarding the linearization functors: I plan to keep the current API for the following reasons:

• I think it more natural to persist and unpersist one detector's worth of linearization information at a time, since that matches the granularity of the ISR code. One indication of this is that the argument list is simpler with my API: linearize(fullImage, detector) instead of linearize(ampImage, detector, ampIndex). Admittedly the detector is is passed instead of the amp info record for sanity checking and not due to any fundamental need, but I think the sanity checking is useful.
• By the time we get to linearization we have already assembled the image and all operations are being performed on the entire detector. Thus the calls around there all look the same (pass in the full image).
• It keeps the ISR code simpler. We tend to need to reimplement ISR separately for each camera, so I think this results in a net simplification of code (or will when we have more cameras that implement linearization).
• Note that the existing Subaru code linearized the CCD image as a unit, so this is not a change for HSC and SuprimeCam.

Regarding out of range pixels: you raise a good point. However, pixels that are too large for the table will already be flagged as SAT or SUSPECT unless the table has a range that that is crazily too small. Pixels that are too small can safely be left alone, and we have no reasonable flag for them. It is conceivable that too-large pixels should trigger an exception (and that would be a trivial change); I think that is not needed for too-small pixels.

Regarding obs_subaru: I split the commit into two pieces: first standardize suspect masking, then linearize using the new code. I think it is a small a set of changes as makes sense. I also modified the log calls as you requested and fixed some pyflakes warnings and errors (in a few additional commits) while I was at it.

Please have another look.

I realized I had forgotten to update the camera generating code for those obs packages that use such. Fixed. I cleaned up some code in obs_decam at the same time (as a separate commit).

I disagree about the API, but I don't think it's so important. Thank you for thinking about it and considering my ideas.

I added some comments to the afw github PR about the test functions; nothing major.

Thank you for all the clean ups!

Thank you for the helpful cleanup suggestions.

Russell Owen do you want to add entries for the two new test routines into the developer guide now that we have entries for those types of routines in that document.