Having spent some time with Nocedal & Wright's "Numerical Optimization 2nd ed.", I'd like to rework the text in section 4.3 of the jointcal paper to better reflect how a least-squares style optimization problem is described in that book. This may or may not result in a change of notation, but it should allow simplification and clarification of the descriptions of how the Jacobian and Hessian matrices are derived, plus a better description of why an Cholesky LDL factorization is a good choice. I think that rewriting this text would be a good start on rethinking how we go about outlier removal and what our available choices are for optimization algorithms.
Chapter 10 and some of chapter 3 are particularly useful in thinking about the classes algorithm that we can apply.