Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14980

Reinstate the ability of a Detector to find the Camera in which it lives

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • afw
    • 6
    • BG3_F18_08, BG3_F18_09, BG3_F18_10
    • Data Release Production

    Description

      The Detector objects are unable to find the Camera wherein they dwell, meaning that e.g. ccd.transform(pos, camGeom.PIXELS, camGeom.FIELD_ANGLE) is impossible (you need a camera), and the functionality requested in DM-14932 cannot be implemented given only a Detector.

      Please add e.g. detector.getCamera() to the Detector API.  This may require moving Camera back into C++.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            Calling this gen3-middleware because we're going to want this Camera refactor to happen before (at least some of) overhauling its persistence.

            jbosch Jim Bosch added a comment - Calling this gen3-middleware because we're going to want this Camera refactor to happen before (at least some of) overhauling its persistence.
            jbosch Jim Bosch added a comment -

            Starting this as pair-programming by czw and jbosch.

            jbosch Jim Bosch added a comment - Starting this as pair-programming by czw and jbosch .
            jbosch Jim Bosch added a comment -

            The crash we saw last week was indeed fixed by DM-15478, which turned it into an exception and revealed the minor changes necessary to get the pybind11 module working.  I've pushed a commit with those changes.  Ideally we can return to this next pair-programming session, but it's moderately high-priority for Gen3, so I may steal it back and do the rest myself if I get all of the higher-priority tickets done.

            jbosch Jim Bosch added a comment - The crash we saw last week was indeed fixed by DM-15478 , which turned it into an exception and revealed the minor changes necessary to get the pybind11 module working.  I've pushed a commit with those changes.  Ideally we can return to this next pair-programming session, but it's moderately high-priority for Gen3, so I may steal it back and do the rest myself if I get all of the higher-priority tickets done.
            jbosch Jim Bosch added a comment -

            I'm afraid it's time for me to steal this back and finish it off.  Hope it was a good learning experience.

            jbosch Jim Bosch added a comment - I'm afraid it's time for me to steal this back and finish it off.  Hope it was a good learning experience.
            jbosch Jim Bosch added a comment -

            rhl, I think I've accomplished the spirit of this issue, but not the letter: a Camera and all of its Detectors now share the same TransformMap, making it possible to do any coordinate transform with just a Detector object.  Detector does not have a getCamera, however - my original plan to do that fizzled when I realized that it's actually quite hard to do without cycles:

            • if Detector only has a weak_ptr to its Camera, there's nothing to keep the Camera from being destroyed in the common case where you have an Exposure with just a Detector attached;
            • if Camera only has weak_ptrs to its Detectors, then there's nothing to keep the Detectors alive in the also-common case where you're getting a Camera from a Butler;
            • any scheme involving attaching Cameras separately to Exposures runs into messy questions of how to guarantee that that Camera instance is the same as the instance the Exposure Detector has a pointer to;
            • I'd really rather not try to do custom allocation and garbage collection here.

            Those problems are neatly solved by having Camera and Detector each have a shared_ptr to a TransformMap that does not have any shared_ptrs back to the Camera or Detector.  In fact, this solution doesn't even require moving Camera back to C++ (though I think we'll still need that for DM-15918).

            I should also add that this new functionality is only enabled if the Camera is built via the makeCameraFromCatalogs or makeCameraFromPath functions in cameraFactory.py, and I'm not sure if that's the case for the YAML cameras.  It is isn't, the changes I've made to makeCameraFromCatalogs should demonstrate how to fix it.  In particular, you can no longer call makeDetector directly, because we need to aggregate the transforms from all of them before actually constructing any of them.  I've kept that API around for backwards compatibility (I really don't have time to deal with random obs_ package breakage right now), but I hope we'll be able to remove it in the future.  It'd actually be ideal to ultimately replace all of cameraFactory.py and cameraConfig.py with native support for building cameras from YAML in afw, but that's definitely a problem for the future.

            Anyhow, if all of that sounds reasonable to you, I also wanted to see if you might have time to review this ticket.  If not, please let me know and I'll find someone else.

            afw PR (the only one) is here: https://github.com/lsst/afw/pull/399

            jbosch Jim Bosch added a comment - rhl , I think I've accomplished the spirit of this issue, but not the letter: a Camera and all of its Detectors now share the same TransformMap, making it possible to do any coordinate transform with just a Detector object.  Detector does not have a getCamera , however - my original plan to do that fizzled when I realized that it's actually quite hard to do without cycles: if Detector only has a weak_ptr to its Camera, there's nothing to keep the Camera from being destroyed in the common case where you have an Exposure with just a Detector attached; if Camera only has weak_ptrs to its Detectors, then there's nothing to keep the Detectors alive in the also-common case where you're getting a Camera from a Butler; any scheme involving attaching Cameras separately to Exposures runs into messy questions of how to guarantee that that Camera instance is the same as the instance the Exposure Detector has a pointer to; I'd really rather not try to do custom allocation and garbage collection here. Those problems are neatly solved by having Camera and Detector each have a shared_ptr to a TransformMap that does not have any shared_ptrs back to the Camera or Detector.  In fact, this solution doesn't even require moving Camera back to C++ (though I think we'll still need that for DM-15918 ). I should also add that this new functionality is only enabled if the Camera is built via the makeCameraFromCatalogs or makeCameraFromPath functions in cameraFactory.py, and I'm not sure if that's the case for the YAML cameras.  It is isn't, the changes I've made to makeCameraFromCatalogs should demonstrate how to fix it.  In particular, you can no longer call makeDetector directly, because we need to aggregate the transforms from all of them before actually constructing any of them.  I've kept that API around for backwards compatibility (I really don't have time to deal with random obs_ package breakage right now), but I hope we'll be able to remove it in the future.  It'd actually be ideal to ultimately replace all of cameraFactory.py and cameraConfig.py with native support for building cameras from YAML in afw, but that's definitely a problem for the future. Anyhow, if all of that sounds reasonable to you, I also wanted to see if you might have time to review this ticket.  If not, please let me know and I'll find someone else. afw PR (the only one) is here:  https://github.com/lsst/afw/pull/399
            jbosch Jim Bosch added a comment -

            salnikov, mind taking over this review?  I realize this is a corner of the codebase you're not super familiar with, but we're running low on Science Pipelines C++ reviewers and I've hit them all pretty hard right recently.  I think this shouldn't require too much of that context to review, especially if you start by looking at the previous comment here on Jira, and then review commit by commit in GitHub.  And of course I'm happy to answer any background questions you might have.

            jbosch Jim Bosch added a comment - salnikov , mind taking over this review?  I realize this is a corner of the codebase you're not super familiar with, but we're running low on Science Pipelines C++ reviewers and I've hit them all pretty hard right recently.  I think this shouldn't require too much of that context to review, especially if you start by looking at the previous comment here on Jira, and then review commit by commit in GitHub.  And of course I'm happy to answer any background questions you might have.

            No problem, will look at it later today.

            salnikov Andy Salnikov added a comment - No problem, will look at it later today.

            I have read all Jira comments and looked at the code, changes look reasonable, with few minor comments.

            salnikov Andy Salnikov added a comment - I have read all Jira comments and looked at the code, changes look reasonable, with few minor comments.

            People

              jbosch Jim Bosch
              rhl Robert Lupton
              Andy Salnikov
              Andy Salnikov, Jim Bosch, John Swinbank, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.