Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-474

Document how to create an astrometry_net_data repository

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_lsstSim
    • Labels:
    • Story Points:
      6
    • Sprint:
      DevOps Sprint 1, DevOps Sprint 2
    • Team:
      SQuaRE

      Description

      Document how to create astrometry_net_data index files for an arbitrary astronomical dataset, as well as how to create a repository of such data to support processing with the LSST Stack. It must be possible to limit the area of sky coverage in the index files to that appropriate for the images that the user wishes to process.

        Attachments

          Activity

          No builds found.
          shaw Richard Shaw [X] (Inactive) created issue -
          shaw Richard Shaw [X] (Inactive) made changes -
          Field Original Value New Value
          Component/s obs_lsstSim [ 10764 ]
          shaw Richard Shaw [X] (Inactive) made changes -
          Remaining Estimate 2 minutes [ 120 ]
          Original Estimate 2 minutes [ 120 ]
          shaw Richard Shaw [X] (Inactive) made changes -
          Remaining Estimate 2 minutes [ 120 ]
          Original Estimate 2 minutes [ 120 ]
          Hide
          price Paul Price added a comment -

          Here's how I create a_n_d index files from a PS1 database query, from https://github.com/HyperSuprime-Cam/hscMisc/blob/master/ps1dbToAnd.py. I suspect it wouldn't be hard to modify this to pull down SDSS or UCAC or similiar, or to have it operate on our own measurements so as to bootstrap a calibration.

          Show
          price Paul Price added a comment - Here's how I create a_n_d index files from a PS1 database query, from https://github.com/HyperSuprime-Cam/hscMisc/blob/master/ps1dbToAnd.py . I suspect it wouldn't be hard to modify this to pull down SDSS or UCAC or similiar, or to have it operate on our own measurements so as to bootstrap a calibration.
          price Paul Price made changes -
          Attachment ps1dbToAnd.py [ 11013 ]
          shaw Richard Shaw [X] (Inactive) made changes -
          Parent DM-471 [ 12051 ]
          Story Points 6
          Issue Type Technical task [ 10002 ] Story [ 10001 ]
          shaw Richard Shaw [X] (Inactive) made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Show
          shaw Richard Shaw [X] (Inactive) added a comment - Page in progress may be viewed at: https://confluence.lsstcorp.org/display/LSWUG/Astrometry.net+Index+Files
          robyn Robyn Allsman [X] (Inactive) made changes -
          Sprint DevOps Sprint 1 [ 42 ] DevOps Sprint 1, DevOps Sprint 2 [ 42, 52 ]
          robyn Robyn Allsman [X] (Inactive) made changes -
          Rank Ranked higher
          Hide
          shaw Richard Shaw [X] (Inactive) added a comment -

          This tutorial on Confluence is ready for review (see https://confluence.lsstcorp.org/display/LSWUG/Building+Astrometry.net+Index+Files). The one caveat is that general users will only be able to build the index files when the astrometry_net package in the Stack is upgraded to v0.46. Still I mention a clue for how to do this on lsst-dev (by pointing to Simon's private copy of v0.38).

          Show
          shaw Richard Shaw [X] (Inactive) added a comment - This tutorial on Confluence is ready for review (see https://confluence.lsstcorp.org/display/LSWUG/Building+Astrometry.net+Index+Files ). The one caveat is that general users will only be able to build the index files when the astrometry_net package in the Stack is upgraded to v0.46. Still I mention a clue for how to do this on lsst-dev (by pointing to Simon's private copy of v0.38).
          shaw Richard Shaw [X] (Inactive) made changes -
          Reviewers Simon Krughoff [ krughoff ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          krughoff Simon Krughoff added a comment -

          Review comments, these are pretty minor:

          1. It would be good to point to the original documentation for reference (http://astrometry.net/doc/build-index.html)
          2. wget is not universal. curl is more common, so I'd suggest changing to: curl -O http://www.astro.washington.edu/users/ivezic/sdss/catalogs/stripe82calibStars_v2.6.dat.gz
          3. Provide downloadable links for the files you embed. It's a little annoying to have to cut and paste to follow along (if that's what you intend).
          4. Why is there no 'hide source' button
          5. The script you wrote could pretty easily include the capability to detect whether the file is gzipped an unzip it for the user.
          6. You suggest that one can create the andConfig.py by hand, so you should probably point to an example and/or a description of the contents.

          Show
          krughoff Simon Krughoff added a comment - Review comments, these are pretty minor: 1. It would be good to point to the original documentation for reference ( http://astrometry.net/doc/build-index.html ) 2. wget is not universal. curl is more common, so I'd suggest changing to: curl -O http://www.astro.washington.edu/users/ivezic/sdss/catalogs/stripe82calibStars_v2.6.dat.gz 3. Provide downloadable links for the files you embed. It's a little annoying to have to cut and paste to follow along (if that's what you intend). 4. Why is there no 'hide source' button 5. The script you wrote could pretty easily include the capability to detect whether the file is gzipped an unzip it for the user. 6. You suggest that one can create the andConfig.py by hand, so you should probably point to an example and/or a description of the contents.
          krughoff Simon Krughoff made changes -
          Status In Review [ 10004 ] Review Complete [ 10101 ]
          Hide
          price Paul Price added a comment -

          Is it worth mentioning hpsplit? Its use seems important for large catalogues.

          Show
          price Paul Price added a comment - Is it worth mentioning hpsplit? Its use seems important for large catalogues.
          price Paul Price made changes -
          Watchers Mario Juric, Paul Price, Richard Shaw, Simon Krughoff [ Mario Juric, Paul Price, Richard Shaw, Simon Krughoff ] Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Simon Krughoff [ Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Simon Krughoff ]
          Hide
          krughoff Simon Krughoff added a comment -

          Good point about hpsplit. It's mentioned in the astrometry.net docs, but not called out specifically in the LSST page.

          Show
          krughoff Simon Krughoff added a comment - Good point about hpsplit. It's mentioned in the astrometry.net docs, but not called out specifically in the LSST page.
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          There is some generic documentation on index building; no need to repeat
          that part. Really the only LSST-specific part is re-using the same star
          trees (via "build-index -1").

          http://astrometry.net/doc/build-index.html

          cheers,
          --dstn

          Simon Krughoff<https://jira.lsstcorp.org/secure/ViewProfile.jspa?name=krughoff>

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - There is some generic documentation on index building; no need to repeat that part. Really the only LSST-specific part is re-using the same star trees (via "build-index -1"). http://astrometry.net/doc/build-index.html cheers, --dstn Simon Krughoff< https://jira.lsstcorp.org/secure/ViewProfile.jspa?name=krughoff >
          Hide
          shaw Richard Shaw [X] (Inactive) added a comment -

          Simon, thanks for the review. Responses below:

          1. Links to the astrometry.net website already existed under the subheadings:

          • Build a FITS Binary Table (one link)
          • Build Indexes (3 links)

          Another was added to the new info block (Split the Sky into Pieces) in:

          • Build Indexes

          Where else would you recommend links?

          2. Updated download command to curl.

          3. The scripts are now also attached to the page and links to download the files are embedded in the text.

          4. Regrettably, a "hide source" feature is not offered in Confluence.

          5. Files could be compressed in a variety of formats, and the supporting libraries for the various MIME types do not appear to be uniformly rich. The suggested capability may be added in a future rev of this page.

          6. Example of andConfig.py added.

          7. (from Paul) hpsplit is now mentioned in an Info block. It is not necessary for the catalog used in this tutorial, and would clutter the narrative to explain it fully. So I've mostly deferred to the Astrometry.net page. I can always embellish this tutorial later if users stumble on this point.

          Show
          shaw Richard Shaw [X] (Inactive) added a comment - Simon, thanks for the review. Responses below: 1. Links to the astrometry.net website already existed under the subheadings: Build a FITS Binary Table (one link) Build Indexes (3 links) Another was added to the new info block (Split the Sky into Pieces) in: Build Indexes Where else would you recommend links? 2. Updated download command to curl. 3. The scripts are now also attached to the page and links to download the files are embedded in the text. 4. Regrettably, a "hide source" feature is not offered in Confluence. 5. Files could be compressed in a variety of formats, and the supporting libraries for the various MIME types do not appear to be uniformly rich. The suggested capability may be added in a future rev of this page. 6. Example of andConfig.py added. 7. (from Paul) hpsplit is now mentioned in an Info block. It is not necessary for the catalog used in this tutorial, and would clutter the narrative to explain it fully. So I've mostly deferred to the Astrometry.net page. I can always embellish this tutorial later if users stumble on this point.
          shaw Richard Shaw [X] (Inactive) made changes -
          Resolution Done [ 10000 ]
          Status Review Complete [ 10101 ] Done [ 10002 ]
          Hide
          krughoff Simon Krughoff added a comment -

          Re: bullet #1

          I don't know how I missed those. Thanks for pointing out their existence. I don't think anything others are necessary.

          Show
          krughoff Simon Krughoff added a comment - Re: bullet #1 I don't know how I missed those. Thanks for pointing out their existence. I don't think anything others are necessary.
          tjenness Tim Jenness made changes -
          Team SQuaRE [ 10302 ]
          Watchers Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Robyn Allsman, Simon Krughoff [ Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Robyn Allsman, Simon Krughoff ] Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Simon Krughoff [ Dustin Lang, Mario Juric, Paul Price, Richard Shaw, Simon Krughoff ]

            People

            Assignee:
            shaw Richard Shaw [X] (Inactive)
            Reporter:
            shaw Richard Shaw [X] (Inactive)
            Reviewers:
            Simon Krughoff
            Watchers:
            Dustin Lang [X] (Inactive), Mario Juric, Paul Price, Richard Shaw [X] (Inactive), Simon Krughoff
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.