As we start supporting non-linearity correction for more cameras I would like to propose some rules, to keep things simple and allow a default version of non-linaerity correction in IsrTask.
We need to support correction based on lookup tables and have existing code that uses polynomials. We may want to support variations of these or other techniques. Thus I propose to make non-linearity correction a subtask of IsrTask that can be retargeted. I further propose that default task support polynomial correction, because it is simple and all the necessary data exists in the AmpInfo table.
For the polynomial-based correction there are a few issues:
- We have four coefficients available. This can easily be increased if needed, but there is no sign of that, rather the opposite...
- Unused coefficients are often set to nan (e.g. in obs_subaru and obs_test)
- The settings for various cameras have some quirks:
- obs_subaru, which has working non-linearity correction code, sets linearityType= "PROPORTIONAL" (or "NONE" if no correction is wanted) in the AmpInfo table, and the associated linearityCoeffs are as follows. This seems a very unnatural order, and I dislike forcing a value to 0 and raising an exception if it is not:
- coeff sqareCoeff: corrIm = uncorrIm + sqareCoeff*uncorrIm^2
- coeff threshold: this must be 0 else an exception is raised
- coeff maxUncorr: if maxUncorr > 0 then flag pixels as SUSPECT where uncorrIm > maxUncorr
- lsstSim sets linearityType="polynomial" and linearityCoeffs=(0, 1, 0, 0), presumably the first four polynomial coefficients. This data is ignored.
- The concept of maxUncorr from obs_subaru seems useful, but it is not a natural fit to a lookup table.
I propose the following:
- Add field suspectLevel to the AmpInfo table and set the SUSPECT flag using specific code for it, rather than relying on non-linearity correction code to do it. This makes the field more obvious than tucking it away as an element of linearityCoeff and also simplifies writing variant tasks to correct non-linearity, since they can focus on doing one thing well.
- Offer constants for linearityType names and use those in code. This catches typos earlier. Suggested names include NONE, POLYNOMIAL and LOOKUP_TABLE. These can be enums in python 3.
- The standard non-linearity correction tasks must accept NONE and do nothing for such detectors. Mosaic cameras may have detectors for which we don't want this correction.
- The standard non-linearity correction tasks will only support one linearityType or NONE. More heterogeneous cameras will require specialized tasks.
- Encode the number of coefficients by adding an extra field, e.g. numLinearityCoeffs. The specified number of fields are required to be finite. Yes we could (and perhaps should) also force the camera to make all unused fields 0, but this is an extra step and is clearly easy to forget. In any case, it is useful information; for instance it can save time to know the order of a polynomial in advance.
- For the standard polynomial non-linear correction task: the coefficient array starts with the x^2 coefficient and goes up from there. Constant and scale correction are explicitly omitted because they are not useful.