Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-562

Drop support for astrometry.net reference catalogs with Gen3 middleware migration

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:
      None

      Description

      The Gen3 Butler moves spatial lookups for reference catalogs (and essentially all other managed data products) into its internal Registry database.* That works fine for the HTM-sharded reference catalogs we've been using essentially exclusively since we added support for them - it removes the need for much of the logic in LoadIndexedReferenceObjectsTask, but the reference catalog datasets themselves are usable as-is.  It will also automatically add support for any sky pixelization supported by sphgeom.

      The astrometry.net ReferenceObjectLoaderTask relies on its own internal sharding system, however, and that makes its interaction with the Gen3 Butler problematic. We have three options:

      • We can continue to treat the sharding as opaque and internal to astrometry.net. This will make it impossible for the Butler to identify which astrometry.net-sharded files need to be transferred to any particular shared-nothing worker node, essentially requiring that they be available via a shared filesystem. The same problem will affect users that want to select a small but internally-complete subset of a repository to transfer to e.g. their laptops.  It will also make a missing reference catalog a runtime failure, rather than a prior-to-pipeline-launch failure.
      • We can try to add support for astrometry.net's sharding system to the Gen3 Butler (probably by adding it to sphgeom). I'm not sure how much it deviates (if at all) from vanilla HEALPIx; if it does, this could involve a lot of work either reimplementing astrometry.net functionality ourselves or invoking APIs it considers private. But even if it doesn't deviate from vanilla HEALPix, it's at least currently blocked by the lack of HEALPix support in sphgeom (something we ultimately need to address but have not yet scheduled).
      • We can drop support for astrometry.net entirely when we drop support for the Gen2 middleware.

      As the summary of this RFC suggests, I'd like to start by proposing that last option, as it's by far the least amount of work, it rids us of a third-party dependency that has been problematic at times (in terms of cross-platform build/install), and I don't think anyone is using the astrometry.net code now anyway.

      *More precisely, it provides possibly-coarse first-pass spatial lookups and relationships that downstream code should in general refine, but that's beside the point for this RFC.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Wholeheartedly support. In fact, I'd like to do this earlier, if possible: the APIs for the A.net and Indexed loaders is subtly different, in was that aren't really fixable. It is also a much older and more fragile codebase.

            We will need to finish DM-11395 (which is now possible, thanks to DM-14868 being finished), and file some tickets to replace the catalogs in the other testdata repositories (had been DM-13376, but that was dropped).

            Show
            Parejkoj John Parejko added a comment - Wholeheartedly support. In fact, I'd like to do this earlier, if possible: the APIs for the A.net and Indexed loaders is subtly different, in was that aren't really fixable. It is also a much older and more fragile codebase. We will need to finish DM-11395 (which is now possible, thanks to DM-14868 being finished), and file some tickets to replace the catalogs in the other testdata repositories (had been DM-13376 , but that was dropped).
            Hide
            rowen Russell Owen added a comment -

            +1 for any option that involves dropping support for astrometry.net catalogs. It has been a significant effort to maintain compatibility with astrometry.net as our astrometry code evolves, an effort with little visible benefit to LSST.

            Show
            rowen Russell Owen added a comment - +1 for any option that involves dropping support for astrometry.net catalogs. It has been a significant effort to maintain compatibility with astrometry.net as our astrometry code evolves, an effort with little visible benefit to LSST.
            Hide
            ctslater Colin Slater added a comment -

            Seconding John's comment: we should drop support, but also identify what work needs to be done to update any lurking a.net catalogs.

            Show
            ctslater Colin Slater added a comment - Seconding John's comment: we should drop support, but also identify what work needs to be done to update any lurking a.net catalogs.
            Hide
            erykoff Eli Rykoff added a comment -

            I agree with John as well. This is a good idea independent of the obvious advantages for the Gen3 migration.

            Show
            erykoff Eli Rykoff added a comment - I agree with John as well. This is a good idea independent of the obvious advantages for the Gen3 migration.
            Hide
            jbosch Jim Bosch added a comment -

            Adopting this as proposed.  Implementation ticket to follow.

            Show
            jbosch Jim Bosch added a comment - Adopting this as proposed.  Implementation ticket to follow.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Colin Slater, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Russell Owen
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Planned End: