Paul Price added a comment - 21/Dec/16 2:59 PM The database at STScI was opened on 2016-12-09, but it's not yet ready for bulk download. I'm downloading the DVO data from UH. Gene assures me that it contains the most recently-updated files, and no additional calculations need to be made.

Paul Price added a comment - 10/Jan/17 9:12 PM ingester INFO: Extracted a total of 2990470528 sources Seems to be working! More tests tomorrow.

Paul Price added a comment - 20/Jan/17 6:57 PM Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572 There's code to be reviewed as well; I hope to submit it on Monday.

Paul Price added a comment - 27/Jan/17 2:18 PM Yusra AlSayyad , would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map , so I recast what I had for map into reduce . price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $ git sub commit d4c28174a6f155897ed36afc7e847a43dd559c6a Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:25:31 2017 -0500   parallel: disable implicit threading As when doing multiprocessing with pipe_base, we don't want the user to have trouble with math libraries using as many threads as cores because we're controlling the parallelism through the pool.   python/lsst/ctrl/pool/parallel.py | 1 + 1 file changed, 1 insertion(+)   commit 597ab040f0e29153739302ea827c32773e69b835 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:28:12 2017 -0500   pool: add reduce functions This allows the master node to do the reduce operation while waiting for slaves to complete the work, instead of operating on the list after the slaves are done. Recast the 'map' operations as 'reduce' operations since they're so similar, but preserved the old API.   python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++------- 1 file changed, 168 insertions(+), 30 deletions(-)   commit 618f8801dae71cd5347f41cd207ae83275b27bb3 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:30:37 2017 -0500   demo: use Pool.reduce instead of Pool.map as it's now more fundamental.   python/lsst/ctrl/pool/test/demoTask.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 62f1124a25f618406b526f2b829b68dbf954d49d Author: Paul Price <price@astro.princeton.edu> Date: Fri Jan 6 15:17:35 2017 -0500   Pool: do reduction on slaves or in a thread This is more efficient (especially if the reduction operation is complicated), since it saves the master from having to do lots of the same operation, when it can be done by many workers.   python/lsst/ctrl/pool/pool.py | 110 ++++++++++++++++++++++++++++----- python/lsst/ctrl/pool/test/demoTask.py | 3 +- 2 files changed, 97 insertions(+), 16 deletions(-)

Yusra AlSayyad added a comment - 27/Jan/17 2:22 PM Sure

Yusra AlSayyad added a comment - 31/Jan/17 12:16 AM Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?

Paul Price added a comment - 31/Jan/17 9:37 AM I've updated the docs in response to your PR comment. PoolMaster.reduce provides the implementation for both reduce and map . The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map , you can use reduce with reducer=None , or you can just call the map method, which does that for you.

Paul Price added a comment - 31/Jan/17 9:38 AM Merged to master. Thanks for the review, Yusra!