Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13244

Port parallel ingest from HSC

    Details

    • Team:
      External

      Description

      HSC used to have a script hscIngestImagesParallel.py which ran ingest in parallel (using what is now ctrl_pool), making ingest of many images go much faster. Port this functionality to LSST.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          Hsin-Fang Chiang, would you mind reviewing this, please?

          price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13244=) $ git sub
          commit edc3a91d2eb39309ac3f0d09239e8029a9601abd
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Jan 16 15:14:48 2018 -0500
           
              ingest: pull functional code from run into method
              
              This allows subclass-and-override to be done without copying code.
           
           python/lsst/pipe/tasks/ingest.py | 59 ++++++++++++++++++++++++----------------
           1 file changed, 35 insertions(+), 24 deletions(-)
           
          commit 798fd9adb1731af73dd80f999ddef5fac39746d6 (HEAD -> tickets/DM-13244, origin/tickets/DM-13244)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Jan 16 19:54:19 2018 -0500
           
              ingest: remove dict from default args
              
              Using an empty dict as default args is always a bad idea, and it was
              producing bugs in the parallel version in pipe_drivers.
           
           python/lsst/pipe/tasks/ingest.py | 4 +++-
           1 file changed, 3 insertions(+), 1 deletion(-)
           
           
          price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13244=) $ git sub
          commit dfc5fa3b9798df41e1d9f1decec933b4212a43b7 (HEAD -> tickets/DM-13244, origin/tickets/DM-13244)
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Jan 16 15:27:56 2018 -0500
           
              add ingestDriver.py
              
              A parallel version of IngestTask, for ingesting lots of images quickly.
           
           bin.src/ingestDriver.py                  |  3 ++
           python/lsst/pipe/drivers/ingestDriver.py | 60 ++++++++++++++++++++++++++++++++
           2 files changed, 63 insertions(+)
          

          Show
          price Paul Price added a comment - Hsin-Fang Chiang , would you mind reviewing this, please? price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13244=) $ git sub commit edc3a91d2eb39309ac3f0d09239e8029a9601abd Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 16 15:14:48 2018 -0500   ingest: pull functional code from run into method This allows subclass-and-override to be done without copying code.   python/lsst/pipe/tasks/ingest.py | 59 ++++++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 24 deletions(-)   commit 798fd9adb1731af73dd80f999ddef5fac39746d6 (HEAD -> tickets/DM-13244, origin/tickets/DM-13244) Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 16 19:54:19 2018 -0500   ingest: remove dict from default args Using an empty dict as default args is always a bad idea, and it was producing bugs in the parallel version in pipe_drivers.   python/lsst/pipe/tasks/ingest.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)     price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13244=) $ git sub commit dfc5fa3b9798df41e1d9f1decec933b4212a43b7 (HEAD -> tickets/DM-13244, origin/tickets/DM-13244) Author: Paul Price <price@astro.princeton.edu> Date: Tue Jan 16 15:27:56 2018 -0500   add ingestDriver.py A parallel version of IngestTask, for ingesting lots of images quickly.   bin.src/ingestDriver.py | 3 ++ python/lsst/pipe/drivers/ingestDriver.py | 60 ++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+)
          Hide
          price Paul Price added a comment -

          Ingesting HSC-SSP data from 2017:

          pprice@tigressdata:/tigress/HSC/HSC-SSP $ ingestDriver.py /tigress/HSC/HSC '2017-*/*.fits' --cores 20 @$HOME/LSST/obs_subaru/hscIngestImages.badargs --mode=link -c clobber=True allowError=True register.ignore=True
          

          Show
          price Paul Price added a comment - Ingesting HSC-SSP data from 2017: pprice@tigressdata:/tigress/HSC/HSC-SSP $ ingestDriver.py /tigress/HSC/HSC '2017-*/*.fits' --cores 20 @$HOME/LSST/obs_subaru/hscIngestImages.badargs --mode=link -c clobber=True allowError=True register.ignore=True
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Looks good to me

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Looks good to me
          Hide
          price Paul Price added a comment -

          Thanks Hsin-Fang Chiang!

          Merged to master.

          Show
          price Paul Price added a comment - Thanks Hsin-Fang Chiang ! Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel