Details

    • Type: Story
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ctrl_pool, meas_algorithms
    • Labels:
      None
    • Templates:
    • Story Points:
      10
    • Sprint:
      DRP S17-1, DRP S17-2
    • Team:
      Data Release Production

      Description

      The PS1 3pi PV3 catalog has been released to the PS1 Science Consortium, and will soon be released generally. Convert the PS1 DVO catalog into the new LSST reference catalog format, and provide a script that can be used to convert this format into astrometry.net format.

        Issue Links

          Activity

          Hide
          price Paul Price added a comment -

          The database at STScI was opened on 2016-12-09, but it's not yet ready for bulk download. I'm downloading the DVO data from UH. Gene assures me that it contains the most recently-updated files, and no additional calculations need to be made.

          Show
          price Paul Price added a comment - The database at STScI was opened on 2016-12-09, but it's not yet ready for bulk download. I'm downloading the DVO data from UH. Gene assures me that it contains the most recently-updated files, and no additional calculations need to be made.
          Hide
          price Paul Price added a comment -

          ingester INFO: Extracted a total of 2990470528 sources
          

          Seems to be working! More tests tomorrow.

          Show
          price Paul Price added a comment - ingester INFO: Extracted a total of 2990470528 sources Seems to be working! More tests tomorrow.
          Hide
          price Paul Price added a comment -

          Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572

          There's code to be reviewed as well; I hope to submit it on Monday.

          Show
          price Paul Price added a comment - Catalogs released: https://community.lsst.org/t/pan-starrs-reference-catalog-in-lsst-format/1572 There's code to be reviewed as well; I hope to submit it on Monday.
          Hide
          price Paul Price added a comment -

          Yusra AlSayyad, would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map, so I recast what I had for map into reduce.

          price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $ git sub
          commit d4c28174a6f155897ed36afc7e847a43dd559c6a
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Jan 4 14:25:31 2017 -0500
           
              parallel: disable implicit threading
              
              As when doing multiprocessing with pipe_base, we don't want the
              user to have trouble with math libraries using as many threads as
              cores because we're controlling the parallelism through the pool.
           
           python/lsst/ctrl/pool/parallel.py | 1 +
           1 file changed, 1 insertion(+)
           
          commit 597ab040f0e29153739302ea827c32773e69b835
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Jan 4 14:28:12 2017 -0500
           
              pool: add reduce functions
              
              This allows the master node to do the reduce operation while
              waiting for slaves to complete the work, instead of operating
              on the list after the slaves are done.
              
              Recast the 'map' operations as 'reduce' operations since they're
              so similar, but preserved the old API.
           
           python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++-------
           1 file changed, 168 insertions(+), 30 deletions(-)
           
          commit 618f8801dae71cd5347f41cd207ae83275b27bb3
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Wed Jan 4 14:30:37 2017 -0500
           
              demo: use Pool.reduce instead of Pool.map
              
              as it's now more fundamental.
           
           python/lsst/ctrl/pool/test/demoTask.py | 4 ++--
           1 file changed, 2 insertions(+), 2 deletions(-)
           
          commit 62f1124a25f618406b526f2b829b68dbf954d49d
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Fri Jan 6 15:17:35 2017 -0500
           
              Pool: do reduction on slaves or in a thread
              
              This is more efficient (especially if the reduction operation
              is complicated), since it saves the master from having to do
              lots of the same operation, when it can be done by many workers.
           
           python/lsst/ctrl/pool/pool.py          | 110 ++++++++++++++++++++++++++++-----
           python/lsst/ctrl/pool/test/demoTask.py |   3 +-
           2 files changed, 97 insertions(+), 16 deletions(-)
          

          Show
          price Paul Price added a comment - Yusra AlSayyad , would you mind reviewing this work, please? These are changes to ctrl_pool that I made to support the creation of the Pan-STARRS catalog: I needed a reduce operation instead of just map , so I recast what I had for map into reduce . price@price-laptop:~/LSST/ctrl/pool (tickets/DM-5989=) $ git sub commit d4c28174a6f155897ed36afc7e847a43dd559c6a Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:25:31 2017 -0500   parallel: disable implicit threading As when doing multiprocessing with pipe_base, we don't want the user to have trouble with math libraries using as many threads as cores because we're controlling the parallelism through the pool.   python/lsst/ctrl/pool/parallel.py | 1 + 1 file changed, 1 insertion(+)   commit 597ab040f0e29153739302ea827c32773e69b835 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:28:12 2017 -0500   pool: add reduce functions This allows the master node to do the reduce operation while waiting for slaves to complete the work, instead of operating on the list after the slaves are done. Recast the 'map' operations as 'reduce' operations since they're so similar, but preserved the old API.   python/lsst/ctrl/pool/pool.py | 198 +++++++++++++++++++++++++++++++++++------- 1 file changed, 168 insertions(+), 30 deletions(-)   commit 618f8801dae71cd5347f41cd207ae83275b27bb3 Author: Paul Price <price@astro.princeton.edu> Date: Wed Jan 4 14:30:37 2017 -0500   demo: use Pool.reduce instead of Pool.map as it's now more fundamental.   python/lsst/ctrl/pool/test/demoTask.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 62f1124a25f618406b526f2b829b68dbf954d49d Author: Paul Price <price@astro.princeton.edu> Date: Fri Jan 6 15:17:35 2017 -0500   Pool: do reduction on slaves or in a thread This is more efficient (especially if the reduction operation is complicated), since it saves the master from having to do lots of the same operation, when it can be done by many workers.   python/lsst/ctrl/pool/pool.py | 110 ++++++++++++++++++++++++++++----- python/lsst/ctrl/pool/test/demoTask.py | 3 +- 2 files changed, 97 insertions(+), 16 deletions(-)
          Hide
          yusra Yusra AlSayyad added a comment -

          Sure

          Show
          yusra Yusra AlSayyad added a comment - Sure
          Hide
          yusra Yusra AlSayyad added a comment -

          Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?

          Show
          yusra Yusra AlSayyad added a comment - Code looks fine (I made one comment about docstring in the PR). Being the first time reading ctrl_pool, I was a little confused on way reduce now wraps map. Naively, when I see "map," I expect to see an operation run on each chunk of data, and when I see "reduce" I expect to see an operation run on those results. Do I understand right that PoolMaster.reduce does both? It maps "func" and reduces with "reducer"? And if you want to just map you pass None as the reducer?
          Hide
          price Paul Price added a comment -

          I've updated the docs in response to your PR comment.

          PoolMaster.reduce provides the implementation for both reduce and map. The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map, you can use reduce with reducer=None, or you can just call the map method, which does that for you.

          Show
          price Paul Price added a comment - I've updated the docs in response to your PR comment. PoolMaster.reduce provides the implementation for both reduce and map . The reason for this is that the code is so similar: the only difference is that reduce does a reduction, while map stores the results. If you want to map , you can use reduce with reducer=None , or you can just call the map method, which does that for you.
          Hide
          price Paul Price added a comment -

          Merged to master.

          Thanks for the review, Yusra!

          Show
          price Paul Price added a comment - Merged to master. Thanks for the review, Yusra!

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Yusra AlSayyad
              Watchers:
              John Swinbank, Paul Price, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile