Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7199

afwTable's .getX()/.getY() do not appear in dir()

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Sprint:
      Alert Production F16 - 9, Alert Production F16 - 10, Alert Production S17 - 5, Alert Production S17 - 6
    • Team:
      Alert Production

      Description

      On a SourceCatalog, I can call cat.getX() or cat.getY() to return an array of pixel coordinates, but neither of these functions appear in dir(cat). This breaks introspection and makes it difficult to know what other such magic functions exist (e.g., I can't tell if there is an equivalent for ra/dec).

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -
            Show
            price Paul Price added a comment - I believe this is desired behaviour: https://github.com/lsst/afw/blob/master/python/lsst/afw/table/Catalog.i#L248-L250
            Hide
            ctslater Colin Slater added a comment -

            Agreed that the functions themselves are desired behavior. I should have been clearer that I am suggesting some extra/different magic so that they are also introspectable.

            Show
            ctslater Colin Slater added a comment - Agreed that the functions themselves are desired behavior. I should have been clearer that I am suggesting some extra/different magic so that they are also introspectable.
            Hide
            rowen Russell Owen added a comment -

            See also DM-5855 which appears to be another undesired side effect of the code in question (kudos to John Parejko for pointing that out).

            Show
            rowen Russell Owen added a comment - See also DM-5855 which appears to be another undesired side effect of the code in question (kudos to John Parejko for pointing that out).
            Hide
            Parejkoj John Parejko added a comment - - edited

            Meredith Rawls and I confirmed this ticket is still a problem post-pybind11. In addition to getX and getY not appearing in dir(), we also found that calling getX on a SourceCatalog which has no centroid results in

            schema = lsst.afw.table.SourceTable.makeMinimalSchema()
            catalog = lsst.afw.table.SourceCatalog(schema)
            record = catalog.addNew()
            record['coord_dec'] = lsst.afw.geom.degrees*(-5.0)
            record['coord_ra'] = lsst.afw.geom.degrees*(22)
            record['id'] = 8
            record['parent'] = 3
            catalog.getX()
            [  1.01184644e-320]
            catalog.getY()
            [  1.01184644e-320]
            catalog.getCentroidKey().getX()
            Key<D>(offset=-1, nElements=1)
            

            This suggests that getCentroidKey is accessing uninitialized memory. Instead, it should probably raise an exception.

            Note that if you make a SimpleCatalog instead of a SourceCatalog, an exception is raised as expected.

            Show
            Parejkoj John Parejko added a comment - - edited Meredith Rawls and I confirmed this ticket is still a problem post-pybind11. In addition to getX and getY not appearing in dir() , we also found that calling getX on a SourceCatalog which has no centroid results in schema = lsst.afw.table.SourceTable.makeMinimalSchema() catalog = lsst.afw.table.SourceCatalog(schema) record = catalog.addNew() record['coord_dec'] = lsst.afw.geom.degrees*(-5.0) record['coord_ra'] = lsst.afw.geom.degrees*(22) record['id'] = 8 record['parent'] = 3 catalog.getX() [ 1.01184644e-320] catalog.getY() [ 1.01184644e-320] catalog.getCentroidKey().getX() Key<D>(offset=-1, nElements=1) This suggests that getCentroidKey is accessing uninitialized memory. Instead, it should probably raise an exception. Note that if you make a SimpleCatalog instead of a SourceCatalog , an exception is raised as expected.
            Hide
            rowen Russell Owen added a comment -

            I suspect the problem of methods not appearing in dir listings is due to catalog delegating unrecognized accessors to records. If so, we could perhaps replace that bit of magic in some way. That code is somewhat complicated (so as to avoid infinite loops on lookup) and it would be great if we could find a better solution. Some possible ideas:

            • Instead of on-the-fly lookup of attributes, have code that copies methods from record to catalog. The trick there is figuring out when to run it, and making sure it works reliably even when we add fields (manually or using a schema mapper). (We could have a cache that is added to as methods are looked for and not found, but that would not help this ticket, so I hope we can do better).
            • Manually duplicate all methods. That seems error-prone (it would be easy to forget to add a method to one class or the other).
            Show
            rowen Russell Owen added a comment - I suspect the problem of methods not appearing in dir listings is due to catalog delegating unrecognized accessors to records. If so, we could perhaps replace that bit of magic in some way. That code is somewhat complicated (so as to avoid infinite loops on lookup) and it would be great if we could find a better solution. Some possible ideas: Instead of on-the-fly lookup of attributes, have code that copies methods from record to catalog. The trick there is figuring out when to run it, and making sure it works reliably even when we add fields (manually or using a schema mapper). (We could have a cache that is added to as methods are looked for and not found, but that would not help this ticket, so I hope we can do better). Manually duplicate all methods. That seems error-prone (it would be easy to forget to add a method to one class or the other).
            Hide
            mrawls Meredith Rawls added a comment -

            Russell Owen and I created DM-10626 to address the issue raised by John Parejko and me above. This bug is not limited to just getX and getY, but happens for any numerical field that should be gett-able from a catalog via a record but has not been given a value.

            Show
            mrawls Meredith Rawls added a comment - Russell Owen and I created DM-10626 to address the issue raised by John Parejko and me above. This bug is not limited to just getX and getY, but happens for any numerical field that should be gett-able from a catalog via a record but has not been given a value.
            Hide
            mrawls Meredith Rawls added a comment -

            Since you helped me verify this problem still existed a few weeks ago, John Parejko, would you be willing to review the fix I wrote with help from Russell Owen during pair coding today? It should be relatively quick. Thanks!

            Show
            mrawls Meredith Rawls added a comment - Since you helped me verify this problem still existed a few weeks ago, John Parejko , would you be willing to review the fix I wrote with help from Russell Owen during pair coding today? It should be relatively quick. Thanks!
            Hide
            Parejkoj John Parejko added a comment -

            See further comments on PR. Expand the test set, add a MemoryTestCase, and try it in a jupyter notebook, and I think you're good.

            Show
            Parejkoj John Parejko added a comment - See further comments on PR. Expand the test set, add a MemoryTestCase, and try it in a jupyter notebook, and I think you're good.

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              ctslater Colin Slater
              Reviewers:
              John Parejko
              Watchers:
              Colin Slater, John Parejko, Meredith Rawls, Paul Price, Robert Lupton, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.