# Generate PS1 3pi PV3 reference catalogs

## Details

• Type: Story
• Status: Done
• Priority: Major
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Templates:
• Story Points:
10
• Sprint:
DRP S17-1, DRP S17-2
• Team:
Data Release Production

## Description

The PS1 3pi PV3 catalog has been released to the PS1 Science Consortium, and will soon be released generally. Convert the PS1 DVO catalog into the new LSST reference catalog format, and provide a script that can be used to convert this format into astrometry.net format.

## Activity

Hide
Paul Price added a comment -

Show
Hide
Paul Price added a comment -

 ingester INFO: Extracted a total of 2990470528 sources 

Seems to be working! More tests tomorrow.

Show
Paul Price added a comment - ingester INFO: Extracted a total of 2990470528 sources Seems to be working! More tests tomorrow.
Hide
Paul Price added a comment -

There's code to be reviewed as well; I hope to submit it on Monday.

Show
Paul Price added a comment - Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572 There's code to be reviewed as well; I hope to submit it on Monday.
Hide
Paul Price added a comment -

Yusra AlSayyad, would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map, so I recast what I had for map into reduce.

 price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $git sub commit d4c28174a6f155897ed36afc7e847a43dd559c6a Author: Paul Price  Date: Wed Jan 4 14:25:31 2017 -0500    parallel: disable implicit threading    As when doing multiprocessing with pipe_base, we don't want the  user to have trouble with math libraries using as many threads as  cores because we're controlling the parallelism through the pool.    python/lsst/ctrl/pool/parallel.py | 1 +  1 file changed, 1 insertion(+)   commit 597ab040f0e29153739302ea827c32773e69b835 Author: Paul Price  Date: Wed Jan 4 14:28:12 2017 -0500    pool: add reduce functions    This allows the master node to do the reduce operation while  waiting for slaves to complete the work, instead of operating  on the list after the slaves are done.    Recast the 'map' operations as 'reduce' operations since they're  so similar, but preserved the old API.    python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++-------  1 file changed, 168 insertions(+), 30 deletions(-)   commit 618f8801dae71cd5347f41cd207ae83275b27bb3 Author: Paul Price  Date: Wed Jan 4 14:30:37 2017 -0500    demo: use Pool.reduce instead of Pool.map    as it's now more fundamental.    python/lsst/ctrl/pool/test/demoTask.py | 4 ++--  1 file changed, 2 insertions(+), 2 deletions(-)   commit 62f1124a25f618406b526f2b829b68dbf954d49d Author: Paul Price  Date: Fri Jan 6 15:17:35 2017 -0500    Pool: do reduction on slaves or in a thread    This is more efficient (especially if the reduction operation  is complicated), since it saves the master from having to do  lots of the same operation, when it can be done by many workers.    python/lsst/ctrl/pool/pool.py | 110 ++++++++++++++++++++++++++++-----  python/lsst/ctrl/pool/test/demoTask.py | 3 +-  2 files changed, 97 insertions(+), 16 deletions(-)  Show Paul Price added a comment - Yusra AlSayyad , would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map , so I recast what I had for map into reduce . price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=)$ git sub commit d4c28174a6f155897ed36afc7e847a43dd559c6a Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:25:31 2017 -0500   parallel: disable implicit threading As when doing multiprocessing with pipe_base, we don't want the user to have trouble with math libraries using as many threads as cores because we're controlling the parallelism through the pool.   python/lsst/ctrl/pool/parallel.py | 1 + 1 file changed, 1 insertion(+)   commit 597ab040f0e29153739302ea827c32773e69b835 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:28:12 2017 -0500   pool: add reduce functions This allows the master node to do the reduce operation while waiting for slaves to complete the work, instead of operating on the list after the slaves are done. Recast the 'map' operations as 'reduce' operations since they're so similar, but preserved the old API.   python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++------- 1 file changed, 168 insertions(+), 30 deletions(-)   commit 618f8801dae71cd5347f41cd207ae83275b27bb3 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:30:37 2017 -0500   demo: use Pool.reduce instead of Pool.map as it's now more fundamental.   python/lsst/ctrl/pool/test/demoTask.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 62f1124a25f618406b526f2b829b68dbf954d49d Author: Paul Price <price@astro.princeton.edu> Date: Fri Jan 6 15:17:35 2017 -0500   Pool: do reduction on slaves or in a thread This is more efficient (especially if the reduction operation is complicated), since it saves the master from having to do lots of the same operation, when it can be done by many workers.   python/lsst/ctrl/pool/pool.py | 110 ++++++++++++++++++++++++++++----- python/lsst/ctrl/pool/test/demoTask.py | 3 +- 2 files changed, 97 insertions(+), 16 deletions(-)
Hide

Sure

Show
Hide

Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?

Show
Yusra AlSayyad added a comment - Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?
Hide
Paul Price added a comment -

I've updated the docs in response to your PR comment.

PoolMaster.reduce provides the implementation for both reduce and map. The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map, you can use reduce with reducer=None, or you can just call the map method, which does that for you.

Show
Paul Price added a comment - I've updated the docs in response to your PR comment. PoolMaster.reduce provides the implementation for both reduce and map . The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map , you can use reduce with reducer=None , or you can just call the map method, which does that for you.
Hide
Paul Price added a comment -

Merged to master.

Thanks for the review, Yusra!

Show
Paul Price added a comment - Merged to master. Thanks for the review, Yusra!

## People

• Assignee:
Paul Price
Reporter:
Paul Price
Reviewers:
Watchers:
John Swinbank, Paul Price, Yusra AlSayyad