Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8465

Wrap meas_modelfit with pybind11

    Details

    • Story Points:
      5
    • Sprint:
      DRP S17-3
    • Team:
      Data Release Production

      Attachments

        Issue Links

          Activity

          Hide
          jbosch Jim Bosch added a comment - - edited

          This is a big review, and I'd like to split it up across files to reduce the burden on any one reviewer.

          For Pim Schellart [X]:

          • Changes in afw, meas_base, and shapelet.
          • model, unitSystem, mixture, priors, likelihood, unitTransformedLikelihood modules
          • testMixture and testProjectedLikelihood

          For Russell Owen:

          • sampler, adaptiveImportanceSampler, integrals, truncatedGaussian, and optimizer modules
          • testIntegrals changes
          • Optimizer header and source

          For Krzysztof Findeisen:

          • psf, pixelFitRegion, and cmodel modules
          • CModel headers and source
          • testCModel*, testGeneralPsf*, testDoubleShapeletPsfApprox

          Some notes:

          • A lot of the code here is not tested, and some of it is a mess. This is a long-standing issue that has been pretty well ticketed (just not completed).
          • Many of these files involve wrappers for multiple classes, and I've taken slightly different approaches to organizing them in different cases - mostly different choices about when to use typedefs and declare functions. I think all of those approaches are consistent with the guidelines we've agreed on, and I think they all make sense in isolation. I don't think we need to standardize further to make them more similar, but I think this is a good test case for feedback on that.
          • I have made some small changes to the main C++ code that were inspired by finding and wanting to remove SWIG-related #ifdefs I found (the two smaller commits). They didn't turn out to be as closely linked to pybind11 vs. Swig as I thought when I started them, but I think they're improvements and I don't think there's any harm including them in this ticket.
          • I plan to soon make a trivial tickets/DM-8465 branch for obs_subaru to re-enable meas_modelfit usage in ci_hsc there, so don't be surprised if you see a new branch for this ticket pop up shortly. It should not have any effect on what needs to be reviewed here.
          Show
          jbosch Jim Bosch added a comment - - edited This is a big review, and I'd like to split it up across files to reduce the burden on any one reviewer. For Pim Schellart [X] : Changes in afw, meas_base, and shapelet. model , unitSystem , mixture , priors , likelihood , unitTransformedLikelihood modules testMixture and testProjectedLikelihood For Russell Owen : sampler , adaptiveImportanceSampler , integrals , truncatedGaussian , and optimizer modules testIntegrals changes Optimizer header and source For Krzysztof Findeisen : psf , pixelFitRegion , and cmodel modules CModel headers and source testCModel* , testGeneralPsf* , testDoubleShapeletPsfApprox Some notes: A lot of the code here is not tested, and some of it is a mess. This is a long-standing issue that has been pretty well ticketed (just not completed). Many of these files involve wrappers for multiple classes, and I've taken slightly different approaches to organizing them in different cases - mostly different choices about when to use typedefs and declare functions. I think all of those approaches are consistent with the guidelines we've agreed on, and I think they all make sense in isolation. I don't think we need to standardize further to make them more similar, but I think this is a good test case for feedback on that. I have made some small changes to the main C++ code that were inspired by finding and wanting to remove SWIG-related #ifdefs I found (the two smaller commits). They didn't turn out to be as closely linked to pybind11 vs. Swig as I thought when I started them, but I think they're improvements and I don't think there's any harm including them in this ticket. I plan to soon make a trivial tickets/ DM-8465 branch for obs_subaru to re-enable meas_modelfit usage in ci_hsc there, so don't be surprised if you see a new branch for this ticket pop up shortly. It should not have any effect on what needs to be reviewed here.
          Hide
          krzys Krzysztof Findeisen added a comment -

          Who should review priors?

          Show
          krzys Krzysztof Findeisen added a comment - Who should review priors ?
          Hide
          jbosch Jim Bosch added a comment -

          Good catch! I've added it to Pim Schellart [X]'s list (he has the other classes those are most closely related to).

          Show
          jbosch Jim Bosch added a comment - Good catch! I've added it to Pim Schellart [X] 's list (he has the other classes those are most closely related to).
          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Review done for my part, see comments on PR.
          I like the overall approach with type aliases used consistently.
          I also think combining wrappers for multiple types into one module as is frequently done here (even when not strictly necessary in places) doesn't seem to bad.
          It has a price of course, making wrappers harder to find, and making the build a little bit more interdependent.
          But when done in relative moderation, as is done here, I think that is ok.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Review done for my part, see comments on PR. I like the overall approach with type aliases used consistently. I also think combining wrappers for multiple types into one module as is frequently done here (even when not strictly necessary in places) doesn't seem to bad. It has a price of course, making wrappers harder to find, and making the build a little bit more interdependent. But when done in relative moderation, as is done here, I think that is ok.
          Hide
          rowen Russell Owen added a comment -

          Review done for my part. In addition to my assigned bits I looked at common.py and multiModel.cc and the build files. The code is very nice. I had a few minor requests (most for code comments in a few places) but didn't catch anything big.

          My only concern of any magnitude is that you often rely on virtual methods being run by the derived class without explicitly wrapping these methods. We have found cases where this does not work (and I confess I don't understand why it ever ought to work), so I hope all such usage is thoroughly tested.

          Show
          rowen Russell Owen added a comment - Review done for my part. In addition to my assigned bits I looked at common.py and multiModel.cc and the build files. The code is very nice. I had a few minor requests (most for code comments in a few places) but didn't catch anything big. My only concern of any magnitude is that you often rely on virtual methods being run by the derived class without explicitly wrapping these methods. We have found cases where this does not work (and I confess I don't understand why it ever ought to work), so I hope all such usage is thoroughly tested.
          Hide
          krzys Krzysztof Findeisen added a comment -

          Lots of minor style/documentation comments, but also one potential showstopper: a number of the python files (e.g., cmodelContinued.py) declare types that are neither used elsewhere in that file nor exported as part of __all__. I would think this could cause problems downstream, but I wanted to confirm that there isn't something more subtle going on.

          Show
          krzys Krzysztof Findeisen added a comment - Lots of minor style/documentation comments, but also one potential showstopper: a number of the python files (e.g., cmodelContinued.py ) declare types that are neither used elsewhere in that file nor exported as part of __all__ . I would think this could cause problems downstream, but I wanted to confirm that there isn't something more subtle going on.
          Hide
          jbosch Jim Bosch added a comment -

          Thanks all for the quick and substantive reviews. I believe I've addressed all comments, but testing with ci_hsc revealed a problem with butler persistence of some plugin config objects that I'll need to fix before merging.

          Show
          jbosch Jim Bosch added a comment - Thanks all for the quick and substantive reviews. I believe I've addressed all comments, but testing with ci_hsc revealed a problem with butler persistence of some plugin config objects that I'll need to fix before merging.
          Hide
          jbosch Jim Bosch added a comment -

          I've fixed the problem discovered in ci_hsc (and added a tiny test for it in meas_modelfit). Merged to tickets/DM-8467!

          Show
          jbosch Jim Bosch added a comment - I've fixed the problem discovered in ci_hsc (and added a tiny test for it in meas_modelfit). Merged to tickets/ DM-8467 !

            People

            • Assignee:
              jbosch Jim Bosch
              Reporter:
              pschella Pim Schellart [X] (Inactive)
              Reviewers:
              Krzysztof Findeisen, Pim Schellart [X] (Inactive), Russell Owen
              Watchers:
              Jim Bosch, Krzysztof Findeisen, Pim Schellart [X] (Inactive), Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: