Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5989

Generate PS1 3pi PV3 reference catalogs

    Details

    • Story Points:
      10
    • Sprint:
      DRP S17-1, DRP S17-2
    • Team:
      Data Release Production

      Description

      The PS1 3pi PV3 catalog has been released to the PS1 Science Consortium, and will soon be released generally. Convert the PS1 DVO catalog into the new LSST reference catalog format, and provide a script that can be used to convert this format into astrometry.net format.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            The database at STScI was opened on 2016-12-09, but it's not yet ready for bulk download. I'm downloading the DVO data from UH. Gene assures me that it contains the most recently-updated files, and no additional calculations need to be made.

            Show
            price Paul Price added a comment - The database at STScI was opened on 2016-12-09, but it's not yet ready for bulk download. I'm downloading the DVO data from UH. Gene assures me that it contains the most recently-updated files, and no additional calculations need to be made.
            Hide
            price Paul Price added a comment -

            ingester INFO: Extracted a total of 2990470528 sources
            

            Seems to be working! More tests tomorrow.

            Show
            price Paul Price added a comment - ingester INFO: Extracted a total of 2990470528 sources Seems to be working! More tests tomorrow.
            Hide
            price Paul Price added a comment -

            Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572

            There's code to be reviewed as well; I hope to submit it on Monday.

            Show
            price Paul Price added a comment - Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572 There's code to be reviewed as well; I hope to submit it on Monday.
            Hide
            price Paul Price added a comment -

            Yusra AlSayyad, would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map, so I recast what I had for map into reduce.

            price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $ git sub
            commit d4c28174a6f155897ed36afc7e847a43dd559c6a
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Jan 4 14:25:31 2017 -0500
             
                parallel: disable implicit threading
                
                As when doing multiprocessing with pipe_base, we don't want the
                user to have trouble with math libraries using as many threads as
                cores because we're controlling the parallelism through the pool.
             
             python/lsst/ctrl/pool/parallel.py | 1 +
             1 file changed, 1 insertion(+)
             
            commit 597ab040f0e29153739302ea827c32773e69b835
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Jan 4 14:28:12 2017 -0500
             
                pool: add reduce functions
                
                This allows the master node to do the reduce operation while
                waiting for slaves to complete the work, instead of operating
                on the list after the slaves are done.
                
                Recast the 'map' operations as 'reduce' operations since they're
                so similar, but preserved the old API.
             
             python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++-------
             1 file changed, 168 insertions(+), 30 deletions(-)
             
            commit 618f8801dae71cd5347f41cd207ae83275b27bb3
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Jan 4 14:30:37 2017 -0500
             
                demo: use Pool.reduce instead of Pool.map
                
                as it's now more fundamental.
             
             python/lsst/ctrl/pool/test/demoTask.py | 4 ++--
             1 file changed, 2 insertions(+), 2 deletions(-)
             
            commit 62f1124a25f618406b526f2b829b68dbf954d49d
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Fri Jan 6 15:17:35 2017 -0500
             
                Pool: do reduction on slaves or in a thread
                
                This is more efficient (especially if the reduction operation
                is complicated), since it saves the master from having to do
                lots of the same operation, when it can be done by many workers.
             
             python/lsst/ctrl/pool/pool.py          | 110 ++++++++++++++++++++++++++++-----
             python/lsst/ctrl/pool/test/demoTask.py |   3 +-
             2 files changed, 97 insertions(+), 16 deletions(-)
            

            Show
            price Paul Price added a comment - Yusra AlSayyad , would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map , so I recast what I had for map into reduce . price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $ git sub commit d4c28174a6f155897ed36afc7e847a43dd559c6a Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:25:31 2017 -0500   parallel: disable implicit threading As when doing multiprocessing with pipe_base, we don't want the user to have trouble with math libraries using as many threads as cores because we're controlling the parallelism through the pool.   python/lsst/ctrl/pool/parallel.py | 1 + 1 file changed, 1 insertion(+)   commit 597ab040f0e29153739302ea827c32773e69b835 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:28:12 2017 -0500   pool: add reduce functions This allows the master node to do the reduce operation while waiting for slaves to complete the work, instead of operating on the list after the slaves are done. Recast the 'map' operations as 'reduce' operations since they're so similar, but preserved the old API.   python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++------- 1 file changed, 168 insertions(+), 30 deletions(-)   commit 618f8801dae71cd5347f41cd207ae83275b27bb3 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:30:37 2017 -0500   demo: use Pool.reduce instead of Pool.map as it's now more fundamental.   python/lsst/ctrl/pool/test/demoTask.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 62f1124a25f618406b526f2b829b68dbf954d49d Author: Paul Price <price@astro.princeton.edu> Date: Fri Jan 6 15:17:35 2017 -0500   Pool: do reduction on slaves or in a thread This is more efficient (especially if the reduction operation is complicated), since it saves the master from having to do lots of the same operation, when it can be done by many workers.   python/lsst/ctrl/pool/pool.py | 110 ++++++++++++++++++++++++++++----- python/lsst/ctrl/pool/test/demoTask.py | 3 +- 2 files changed, 97 insertions(+), 16 deletions(-)
            Hide
            yusra Yusra AlSayyad added a comment -

            Sure

            Show
            yusra Yusra AlSayyad added a comment - Sure
            Hide
            yusra Yusra AlSayyad added a comment -

            Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?

            Show
            yusra Yusra AlSayyad added a comment - Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?
            Hide
            price Paul Price added a comment -

            I've updated the docs in response to your PR comment.

            PoolMaster.reduce provides the implementation for both reduce and map. The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map, you can use reduce with reducer=None, or you can just call the map method, which does that for you.

            Show
            price Paul Price added a comment - I've updated the docs in response to your PR comment. PoolMaster.reduce provides the implementation for both reduce and map . The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map , you can use reduce with reducer=None , or you can just call the map method, which does that for you.
            Hide
            price Paul Price added a comment -

            Merged to master.

            Thanks for the review, Yusra!

            Show
            price Paul Price added a comment - Merged to master. Thanks for the review, Yusra!

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Yusra AlSayyad
                Watchers:
                John Swinbank, Paul Price, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel