Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4157

Provide a recommendation for how to manage Wcs in LSST

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Story Points:
      4
    • Sprint:
      Alert Production X16 - 03, Alert Production X16 - 5
    • Team:
      Alert Production

      Description

      The technical report from DM-4155 and DM-4156 should have as its conclusion a recommendation for what Wcs system the LSST stack should adopt (either our own implementation, a third-party library, or some combination thereof). Beyond the conclusion section of that report, this will be provided as an RFC that includes:

      • executive summary
      • mock API
      • links to the relevant documentation (beyond just the above technical report)
      • a bullet list of the rationale for this decision

      The conclusion of this RFC represents the end of this Epic.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            As things stand I think there are two papers cited:

            @ARTICLE{2016A&C....15...33B,
               author = {{Berry}, D.~S. and {Warren-Smith}, R.~F. and {Jenness}, T.},
                title = "{AST: A library for modelling and manipulating coordinate systems}",
              journal = {Astronomy and Computing},
            archivePrefix = "arXiv",
               eprint = {1602.06681},
             primaryClass = "astro-ph.IM",
             keywords = {World Coordinate Systems, Data models, Starlink},
                 year = 2016,
                month = apr,
               volume = 15,
                pages = {33-49},
                  doi = {10.1016/j.ascom.2016.02.003},
               adsurl = {http://adsabs.harvard.edu/abs/2016A%26C....15...33B},
              adsnote = {Provided by the SAO/NASA Astrophysics Data System}
            }
             
            @ARTICLE{2002A&A...395.1061G,
               author = {{Greisen}, E.~W. and {Calabretta}, M.~R.},
                title = "{Representations of world coordinates in FITS}",
              journal = {\aap},
               eprint = {astro-ph/0207407},
             keywords = {methods: data analysis, techniques: image processing, astronomical data bases: miscellaneous},
                 year = 2002,
                month = dec,
               volume = 395,
                pages = {1061-1075},
                  doi = {10.1051/0004-6361:20021326},
               adsurl = {http://adsabs.harvard.edu/abs/2002A%26A...395.1061G},
              adsnote = {Provided by the SAO/NASA Astrophysics Data System}
            }
            

            Show
            tjenness Tim Jenness added a comment - As things stand I think there are two papers cited: @ARTICLE{2016A&C....15...33B, author = {{Berry}, D.~S. and {Warren-Smith}, R.~F. and {Jenness}, T.}, title = "{AST: A library for modelling and manipulating coordinate systems}", journal = {Astronomy and Computing}, archivePrefix = "arXiv", eprint = {1602.06681}, primaryClass = "astro-ph.IM", keywords = {World Coordinate Systems, Data models, Starlink}, year = 2016, month = apr, volume = 15, pages = {33-49}, doi = {10.1016/j.ascom.2016.02.003}, adsurl = {http://adsabs.harvard.edu/abs/2016A%26C....15...33B}, adsnote = {Provided by the SAO/NASA Astrophysics Data System} }   @ARTICLE{2002A&A...395.1061G, author = {{Greisen}, E.~W. and {Calabretta}, M.~R.}, title = "{Representations of world coordinates in FITS}", journal = {\aap}, eprint = {astro-ph/0207407}, keywords = {methods: data analysis, techniques: image processing, astronomical data bases: miscellaneous}, year = 2002, month = dec, volume = 395, pages = {1061-1075}, doi = {10.1051/0004-6361:20021326}, adsurl = {http://adsabs.harvard.edu/abs/2002A%26A...395.1061G}, adsnote = {Provided by the SAO/NASA Astrophysics Data System} }
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the comments. I'll incorporate them.

            There are at least 4 papers cited (3 FITS WCS, plus AST). I linked them all to the ADS records.

            Show
            Parejkoj John Parejko added a comment - Thanks for the comments. I'll incorporate them. There are at least 4 papers cited (3 FITS WCS, plus AST). I linked them all to the ADS records.
            Hide
            ctslater Colin Slater added a comment - - edited

            I was expecting another section between the list of options and the recommendations that would detail some sort of decision making process between the two or three most viable options. You have a lot of pros and cons, how do you weigh them against each other? The recommendation is an AST rewrite-which is a significant undertaking-and the justification is basically that GWCS is slow. If one had to justify this decision outside the project I don't think this would be sufficient to make the case.

            Open questions that I have after reading:

            • My understanding is that there are probably two milestones here, one is a short-term effort to get a working implementation of composable transforms, probably sacrificing code quality or speed. There's a second milestone of having an implementation that we are happy with top to bottom. I'd like to know how much effort is required to reach those two states via either of the routes suggested. I think you have some implicit ideas about this (obviously you don't think AST++ will take 10 years), and it would help to try to be concrete about this. I assume that in the AST route, #1 is met by just putting a minimal wrapper on AST or using it directly, and it would significantly help the case for the AST route if you assured the reader that it was viable even without the rewrite (and if it could be done incrementally).
            • In general I think the conclusions come off as comparing GWCS as a pure drop-in replacement of WCSLIB against AST+significant rewrite effort. It would be much clearer if these were compared on something like a constant-effort basis. E.g., if we spent AST-rewrite level of effort towards implementing GWCS, what would that result look like? (e.g. Cython?)
            • The treatment of GWCS doesn't really ask the question of how we would make the best use of it. What would our warping algorithms have to look like? The document seems to write this off as too much effort, but it's not clear how that level of effort compares with the AST++ effort.
            • GWCS is slower at coordinate conversion, but how much slower does that actually make warping? What fraction of the current warping time is spent in coordinate conversion. Are there possible hybrid schemes? Can we specify stackable transforms in GWCS, but then on a per-ccd or per-footprint level extract a polynomial (or such) that we can call from C++ ? I discussed this with Simon a bit and he mentioned that sharp features might be hard to extract, so I'm not really advocating for this solution specifically, but more generally that there might be options beyond our current brute-force C++ loops.
            • As a small note, I'm much more interested in having some results presented from the speed test code than having the full source code in the document. Where is the result of your work with Russell on implementing AST models?

            I understand that this might be asking for more than the scope of the document you were envisioning. I think you'll end up getting some of these questions when you RFC it though.

            Show
            ctslater Colin Slater added a comment - - edited I was expecting another section between the list of options and the recommendations that would detail some sort of decision making process between the two or three most viable options. You have a lot of pros and cons, how do you weigh them against each other? The recommendation is an AST rewrite- which is a significant undertaking -and the justification is basically that GWCS is slow. If one had to justify this decision outside the project I don't think this would be sufficient to make the case. Open questions that I have after reading: My understanding is that there are probably two milestones here, one is a short-term effort to get a working implementation of composable transforms, probably sacrificing code quality or speed. There's a second milestone of having an implementation that we are happy with top to bottom. I'd like to know how much effort is required to reach those two states via either of the routes suggested. I think you have some implicit ideas about this (obviously you don't think AST++ will take 10 years), and it would help to try to be concrete about this. I assume that in the AST route, #1 is met by just putting a minimal wrapper on AST or using it directly, and it would significantly help the case for the AST route if you assured the reader that it was viable even without the rewrite (and if it could be done incrementally). In general I think the conclusions come off as comparing GWCS as a pure drop-in replacement of WCSLIB against AST+significant rewrite effort. It would be much clearer if these were compared on something like a constant-effort basis. E.g., if we spent AST-rewrite level of effort towards implementing GWCS, what would that result look like? (e.g. Cython?) The treatment of GWCS doesn't really ask the question of how we would make the best use of it. What would our warping algorithms have to look like? The document seems to write this off as too much effort, but it's not clear how that level of effort compares with the AST++ effort. GWCS is slower at coordinate conversion, but how much slower does that actually make warping? What fraction of the current warping time is spent in coordinate conversion. Are there possible hybrid schemes? Can we specify stackable transforms in GWCS, but then on a per-ccd or per-footprint level extract a polynomial (or such) that we can call from C++ ? I discussed this with Simon a bit and he mentioned that sharp features might be hard to extract, so I'm not really advocating for this solution specifically, but more generally that there might be options beyond our current brute-force C++ loops. As a small note, I'm much more interested in having some results presented from the speed test code than having the full source code in the document. Where is the result of your work with Russell on implementing AST models? I understand that this might be asking for more than the scope of the document you were envisioning. I think you'll end up getting some of these questions when you RFC it though.
            Hide
            Parejkoj John Parejko added a comment -

            Thanks Tim Jenness for your comments. I believe I've incorporated all of them into the latest version, including references (with a cite for the AST paper).

            Colin Slater Thank you for the overview notes. That's just what I was looking for. I will try to fold in your thoughts here, and post when I have an updated version. Most of your questions are definitely in scope, thought some might just warrant a brief mention. I'll see what I can do.

            Show
            Parejkoj John Parejko added a comment - Thanks Tim Jenness for your comments. I believe I've incorporated all of them into the latest version, including references (with a cite for the AST paper). Colin Slater Thank you for the overview notes. That's just what I was looking for. I will try to fold in your thoughts here, and post when I have an updated version. Most of your questions are definitely in scope, thought some might just warrant a brief mention. I'll see what I can do.
            Hide
            Parejkoj John Parejko added a comment -

            Big thanks to Russell Owen and Colin Slater for detailed comments in the last few days. I have now merged to master, and will tag and post the RFC.

            Show
            Parejkoj John Parejko added a comment - Big thanks to Russell Owen and Colin Slater for detailed comments in the last few days. I have now merged to master, and will tag and post the RFC.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Colin Slater, Tim Jenness
                Watchers:
                Colin Slater, John Parejko, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel