Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14172

Optimize cModel config parameters

    XMLWordPrintable

    Details

    • Story Points:
      7
    • Epic Link:
    • Sprint:
      DRP S18-5, DRP S18-6, DRP F18-1
    • Team:
      Data Release Production

      Description

      I ran multiBandDriver.py on HSC-R 9813 3,4 to determine if any cModel configuration parameters can/should be changed to improve the running time and/or best-fit model parameters. My conclusions for the tested parameters are:

          initial.nComponents (default 3): For 2, initial fits run in 0.67x the time, but the exp/dev fits run more slowly, so the total fit time is 0.94x. Initial fits are slightly worse on average (with large scatter) and a sizeable fraction of the dev/exp fits are significantly worse (deltalnP < -10).
              For nComponents=6, the initial fitting time is more than twice as long, with modest increases in the exp/dev lnP (<1 for most galaxies, with scatter).
              Recommendation: leave it at 3 or lower to 2.
          initial.optimizer.gradientThreshold (default 1e-2): 1e-3 runs very slightly faster than default and very slightly improves fits. 1e-1 is slightly slower and also results in slightly worse initial/final fits.
              Recommendation: lower to 1e-3.
          initial.optimizer.maxInnerIterations (default 20): Zero impact on fits, running time for 8 is virtually identical; 50 is 1% slower.
              Recommendation: leave at 20.
          initial.optimizer.noSR1Term (default False): True radically changes fitting times by factors of (0.33, 4.37, 4.98)x for (initial, exp, dev) i.e. the initial fit is faster but the subsequent exp/dev are much slower.
              Confusingly, the initial fits are better on average, whereas the final dev/exp are worse - this may warrant follow-up. Many of the very slow fits have faint PSF mags and may be pathological cases.
              Recommendation: leave as False.
          region.nFitRadiiMax (default 3): Surprisingly, nfitrad=2 initial fits are 0.5% slower than the default, whereas nfitrad=5 is 2% faster. However, the total fitting time scales by a factor of (0.86, 1.38) for the smaller/larger size.
              Smaller fitting regions naturally have larger likelihoods, but I can't make a meaningful comparison between likelihoods - presumably the smaller region fits better in the common are and worse outside.

              Recommendation: leave as is pending further investigation.

      One common feature is that there is a non-negligible fraction of objects (usually with faint mag_psf) with very small differences in initial fit likelihood but large ones in the dev/exp fit, which should not occur frequently.

      The quoted figures can be generated by the notebook created in DM-14118 (lsst-dev:/home/dtaranu/src/mine/taranu_lsst/cModelConfigs.ipynb).

        Attachments

          Issue Links

            Activity

            Hide
            dtaranu Dan Taranu added a comment -

            The notebook I based this on:

            https://github.com/lsst-dm/modelling_research/blob/master/jupyternotebooks/lsst_cmodel_configs.ipynb

            Jim Bosch, I could make more readable summary tables/plots and test on more tracts, bands and/or wide data as well if you think it's worthwhile, but none of the results on this patch seemed very encouraging to me.

            Show
            dtaranu Dan Taranu added a comment - The notebook I based this on: https://github.com/lsst-dm/modelling_research/blob/master/jupyternotebooks/lsst_cmodel_configs.ipynb Jim Bosch , I could make more readable summary tables/plots and test on more tracts, bands and/or wide data as well if you think it's worthwhile, but none of the results on this patch seemed very encouraging to me.
            Hide
            jbosch Jim Bosch added a comment -

            Sounds like we should just go ahead and make the one change you recommend here (initial.optimizer.gradientThreshold = 1E-3.  Feel free to do that on this ticket or a new one; either is fine with me.

            The noSR1Term behavior certainly is confusing.  If we were interested in putting more effort into CModel, it would definitely be worth a closer look (but we aren't, so I think we won't, at least not now).  I still think this optimizer - with the SR1 term enabled (so noSR1Term=False) - is at least theoretically the right one for the kind of problem we're trying to solve (nonlinear least squares with a prior), though I'm less confident in my implementation of it.  In any case, I'd be interested in trying it again once we have the "what model should we fit" problem better under control.  When we get to that stage, we'll have to pay close attention to what including the SR1 term actually does in practice.

            By the way, if you're interested in why I like this optimizer (theoretically), I'm happy to discuss that.  There's also a good book on optimizers in my office that you're welcome to borrow.  It doesn't describe the exact one I implemented, but it explains all the components that went into it.

            Show
            jbosch Jim Bosch added a comment - Sounds like we should just go ahead and make the one change you recommend here ( initial.optimizer.gradientThreshold = 1E-3 .  Feel free to do that on this ticket or a new one; either is fine with me. The noSR1Term behavior certainly is confusing.  If we were interested in putting more effort into CModel, it would definitely be worth a closer look (but we aren't, so I think we won't, at least not now).  I still think this optimizer - with the SR1 term enabled (so noSR1Term=False) - is at least theoretically the right one for the kind of problem we're trying to solve (nonlinear least squares with a prior), though I'm less confident in my implementation of it.  In any case, I'd be interested in trying it again once we have the "what model should we fit" problem better under control.  When we get to that stage, we'll have to pay close attention to what including the SR1 term actually does in practice. By the way, if you're interested in why I like this optimizer (theoretically), I'm happy to discuss that.  There's also a good book on optimizers in my office that you're welcome to borrow.  It doesn't describe the exact one I implemented, but it explains all the components that went into it.

              People

              Assignee:
              dtaranu Dan Taranu
              Reporter:
              dtaranu Dan Taranu
              Reviewers:
              Jim Bosch
              Watchers:
              Dan Taranu, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Start date:
                End date:

                  Jenkins

                  No builds found.