Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1576

Sanitize AstrometryTask interface

    XMLWordPrintable

    Details

    • Story Points:
      10
    • Sprint:
      Science Pipelines DM-W15-3, Science Pipelines DM-W15-4, Science Pipelines DM-S15-1
    • Team:
      Alert Production

      Description

      Currently the AstrometryTask and Astrometry class share work. E.g. distortion is done in AstrometryTask but matching is done in Astrometry. AstrometryTask also makes assumptions about what fields are available in the solver config.

      The AstrometryTask interface should be sanitized so that it can be used as a thin wrapper for calling any astrometry solver. Top level config params should go in the AstrometryTaskConfig and solver level work should be done in the solver class and configured at the solver level.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Review complete. I didn't see any major issues, but I left a lot of comments on the PR (as you've already seen). I very much appreciate the new unit tests, and it looks great overall.

          I didn't look too strictly at the new matcher implementation code, as I know we're trying to strike a balance between cleaning up what was in hscAstrom and not doing anything that could change its behavior.

          I also didn't worry too much about the overall structure and where the astrometry.net code was still being used, as I know there's more restructuring yet to be done.

          Show
          jbosch Jim Bosch added a comment - Review complete. I didn't see any major issues, but I left a lot of comments on the PR (as you've already seen). I very much appreciate the new unit tests, and it looks great overall. I didn't look too strictly at the new matcher implementation code, as I know we're trying to strike a balance between cleaning up what was in hscAstrom and not doing anything that could change its behavior. I also didn't worry too much about the overall structure and where the astrometry.net code was still being used, as I know there's more restructuring yet to be done.
          Hide
          rowen Russell Owen added a comment -

          I implemented the simpler changes in a squashed push. That leaves changing the reference loader to return magnitudes instead of fluxes (which will be fairly disruptive, but is well worth doing), revisiting how dense regions (e.g. M31) are avoided in the matcher, and adding magnitude limits to the loader (which may be postponed for another ticket). These changes will be separate commits.

          Show
          rowen Russell Owen added a comment - I implemented the simpler changes in a squashed push. That leaves changing the reference loader to return magnitudes instead of fluxes (which will be fairly disruptive, but is well worth doing), revisiting how dense regions (e.g. M31) are avoided in the matcher, and adding magnitude limits to the loader (which may be postponed for another ticket). These changes will be separate commits.
          Hide
          rowen Russell Owen added a comment - - edited

          I have pushed more changes to afw, ip_diffIm, meas_algorithms, meas_astrom and obs_sdss, all on tickets/DM-1576, including:

          • afw's Config.h has convenience functions that compute flux in Jy to and from AB magnitudes, and using those cleaned up a lot of code
          • I moved the abstract base class LoadReferenceObjectsTask to meas_astrom, which is where it belongs (for now) and so that other code could take advantage of its tools for generating reference catalog schema and retrieving flux fields.
          • ip_diffIm needed a few fixes due to the updated reference catalog schema.
          • obs_sdss calibrate task and config overrides needed a few tweaks.
          • The reference loader returns flux in Jy
          • The matcher handles version 0 and version 1 source schemas. This feature will not be needed in a few weeks, when catalogs will be converted as they are read in, but I propose leaving it in for now because some tests and examples have catalogs using the old schema. It will be fairly easy to take out later. Still, if you have a better idea, please let me know. I really don't want to have to wait to merge the code until the catalog unpersistence handles the conversion automatically, but I am a bit nervous about the matcher's "sourceFluxType" config field, since more traditionally we ask the user to provide the full field name. Still...we can switch that field without much trouble, as it is not used extensively.
          • obs_sdss's photometry task now works and buildbot runs to completion, though some values have changed slightly, making the final test fail
          • the PhotoCal task was updated to handle reference flux in Jy

          Known issues:

          • The PhotoCal task does not handle flux errors fully correctly; changing reference flux to Jy highlighted the problem (the scale of the reference flux and source flux were so different that the reference flux errors were being ignored). I implemented a workaround that I would like checked. It would be even better to handle the errors fully correctly, but I don't know how to do that.
          • the astrometry task's handling of schema is nonstandard in that the _init_ does not take a reference schema and expand it. There are several reasons for this: C++ code computes the schema from scratch and some of the needed information is not available at init time.
          Show
          rowen Russell Owen added a comment - - edited I have pushed more changes to afw, ip_diffIm, meas_algorithms, meas_astrom and obs_sdss, all on tickets/ DM-1576 , including: afw's Config.h has convenience functions that compute flux in Jy to and from AB magnitudes, and using those cleaned up a lot of code I moved the abstract base class LoadReferenceObjectsTask to meas_astrom, which is where it belongs (for now) and so that other code could take advantage of its tools for generating reference catalog schema and retrieving flux fields. ip_diffIm needed a few fixes due to the updated reference catalog schema. obs_sdss calibrate task and config overrides needed a few tweaks. The reference loader returns flux in Jy The matcher handles version 0 and version 1 source schemas. This feature will not be needed in a few weeks, when catalogs will be converted as they are read in, but I propose leaving it in for now because some tests and examples have catalogs using the old schema. It will be fairly easy to take out later. Still, if you have a better idea, please let me know. I really don't want to have to wait to merge the code until the catalog unpersistence handles the conversion automatically, but I am a bit nervous about the matcher's "sourceFluxType" config field, since more traditionally we ask the user to provide the full field name. Still...we can switch that field without much trouble, as it is not used extensively. obs_sdss's photometry task now works and buildbot runs to completion, though some values have changed slightly, making the final test fail the PhotoCal task was updated to handle reference flux in Jy Known issues: The PhotoCal task does not handle flux errors fully correctly; changing reference flux to Jy highlighted the problem (the scale of the reference flux and source flux were so different that the reference flux errors were being ignored). I implemented a workaround that I would like checked. It would be even better to handle the errors fully correctly, but I don't know how to do that. the astrometry task's handling of schema is nonstandard in that the _ init _ does not take a reference schema and expand it. There are several reasons for this: C++ code computes the schema from scratch and some of the needed information is not available at init time.
          Hide
          jbosch Jim Bosch added a comment -

          I believe I've covered everything that's changed since my last pass, with more comments appearing on the various PRs. Let me know if you think there's something I missed; I didn't look at all of meas_astrom again (instead I just tried to look at the bits of code you'd pointed me at here).

          Show
          jbosch Jim Bosch added a comment - I believe I've covered everything that's changed since my last pass, with more comments appearing on the various PRs. Let me know if you think there's something I missed; I didn't look at all of meas_astrom again (instead I just tried to look at the bits of code you'd pointed me at here).
          Hide
          rowen Russell Owen added a comment -

          Thank you for your second review. I have merged all the code changes to master.

          I still have to update the expected results files in the lsst demo, but don't yet know how to push changes to that. It will probably have to be done in two steps, since only one of the two results files is generated if the first one fails to match.

          Show
          rowen Russell Owen added a comment - Thank you for your second review. I have merged all the code changes to master. I still have to update the expected results files in the lsst demo, but don't yet know how to push changes to that. It will probably have to be done in two steps, since only one of the two results files is generated if the first one fails to match.

            People

            Assignee:
            rowen Russell Owen
            Reporter:
            krughoff Simon Krughoff
            Reviewers:
            Jim Bosch
            Watchers:
            Dominique Boutigny, Jim Bosch, Paul Price, Russell Owen, Simon Krughoff
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.