Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30254

Fix jointcal crash when doing outlier rejection on only the model

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jointcal
    • Labels:
    • Story Points:
      2
    • Sprint:
      AP S21-6 (May), AP F21-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      While investigating DM-30252, Clare Saunders and I re-discovered a C++ crash in jointcal-that she had noticed during previous testing-when fitting only the model with outlier rejection turned on. FitterBase::findOutliers cannot mark parameters as changed if those parameters are outside the portion of the matrix being fit (i.e. if _nParTot does not include _nParPositions). The simplest fix for this is to not reject outliers that are not being fit; they are not directly contributing to the fit chisq anyway!

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            On this ticket I added Associations.cleanFittedStars(), which removes FittedStars that have no measuredStars, but I don't actually use it anywhere here. A use case would be to change the existing _iterate_fit() call in _fit_astrometry to only fit "Distortions" and add the below immediately after. That may or may not actually help with runtime, but might "finalize" the positions after we get much closer by only fitting the model.

                    associations.cleanFittedStars()
                    chi2 = self._iterate_fit(associations,
                                             fit,
                                             1,  # only do one "outer" fit loop to finalize positions
                                             "astrometry",
                                             "DistortionsPositions",
                                             doRankUpdate=self.config.astrometryDoRankUpdate,
                                             dataName=dataName)
            

            Alternately, you could do this at the same place, which wouldn't remove FittedStar/RefStar outliers, but would also "finalize" the positions:

                    associations.cleanFittedStars()
                    fit.minimize("DistortionsPositions")
                    self._logChi2AndValidate(associations, fit, model, "Finalize DistortionsPositions",
                                             writeChi2Name=getChi2Name("FinalDistortionsPositions"))
            

            Show
            Parejkoj John Parejko added a comment - On this ticket I added Associations.cleanFittedStars() , which removes FittedStars that have no measuredStars, but I don't actually use it anywhere here. A use case would be to change the existing _iterate_fit() call in _fit_astrometry to only fit "Distortions" and add the below immediately after. That may or may not actually help with runtime, but might "finalize" the positions after we get much closer by only fitting the model. associations.cleanFittedStars() chi2 = self._iterate_fit(associations, fit, 1, # only do one "outer" fit loop to finalize positions "astrometry", "DistortionsPositions", doRankUpdate=self.config.astrometryDoRankUpdate, dataName=dataName) Alternately, you could do this at the same place, which wouldn't remove FittedStar/RefStar outliers, but would also "finalize" the positions: associations.cleanFittedStars() fit.minimize("DistortionsPositions") self._logChi2AndValidate(associations, fit, model, "Finalize DistortionsPositions", writeChi2Name=getChi2Name("FinalDistortionsPositions"))
            Hide
            Parejkoj John Parejko added a comment - - edited

            Clare Saunders: see the note above about how to use this while exploring DM-30252. It's probably best to review each commit separately: they are all atomic.

            Also, I'm definitely interested alternatives for the "Values" name added here: that was what I suggested in DM-8046 (to replace "Positions" and "Fluxes"), but it's not ideal.

            Show
            Parejkoj John Parejko added a comment - - edited Clare Saunders : see the note above about how to use this while exploring DM-30252 . It's probably best to review each commit separately: they are all atomic. Also, I'm definitely interested alternatives for the "Values" name added here: that was what I suggested in DM-8046 (to replace "Positions" and "Fluxes"), but it's not ideal.
            Show
            Parejkoj John Parejko added a comment - - edited Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34362/pipeline
            Hide
            csaunder Clare Saunders added a comment -

            Thanks for the changes! Looks good.

            Show
            csaunder Clare Saunders added a comment - Thanks for the changes! Looks good.

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Clare Saunders
              Watchers:
              Clare Saunders, Ian Sullivan, Jim Bosch, John Parejko
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.