Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3373

Port HSC code for generation of calibration products

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      Science Pipelines DM-W16-6, DRP W16-7
    • Team:
      Data Release Production

      Description

      Port HSC code for building calibration products (flats, bias frames, etc.).

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            I can now construct and ingest calibration products with HSC. I have not yet tried to construct calibs with other cameras, but the port has been done.

            This is a fairly large submission (1000 SLOC added in pipe_drivers), so I'd like to divide it into two logical parts:

            • Nate Lust, would you please look at pipe_drivers and the small changes in ctrl_pool and obs_subaru? I hope you will be familiar with this kind of code from your recent work on the coadd driver. In pipe_drivers, I have branched from your work in DM-4855 so that I'm not going to get massive conflicts when merging to master. The commits in obs_subaru contain one commit that I will remove once Lauren MacArthur's work on rotations for calibs and detrends are merged, so no need to review that one; and the rest in obs_subaru are mostly renames so it looks larger than it really is.
            • Hsin-Fang Chiang, would you please look at the changes to ingestCalib in pipe_tasks and obs_decam? This was originally your code, so would you please check it's true to your intention and that I haven't broken it for your use case with DECam? (I don't have any DECam calibs, or I would check myself.)

            price@price-laptop:~/LSST/pipe/drivers (tickets/DM-3373=) $ git sub tickets/DM-3373 origin/tickets/DM-4855
            commit 06ec186bc0a2a0fb93285f784ab5d49d0d7b81c6
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Feb 9 21:39:47 2016 -0500
             
                Add calib construction code from hscPipe
             
             bin.src/constructBias.py                    |   3 +
             bin.src/constructDark.py                    |   3 +
             bin.src/constructFlat.py                    |   3 +
             bin.src/constructFringe.py                  |   3 +
             bin.src/constructMask.py                    |   3 +
             python/lsst/pipe/drivers/checksum.py        |  59 ++
             python/lsst/pipe/drivers/constructCalibs.py | 978 ++++++++++++++++++++++++++++
             7 files changed, 1052 insertions(+)
             
            commit b7f2955de1152fde4ec335425b05490cad084ba8
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 16:51:34 2016 -0500
             
                git: ignore generated files
             
             .gitignore | 4 +++-
             1 file changed, 3 insertions(+), 1 deletion(-)
             
             
            commit 5792d30413fbf76bcf7a180a1924f987635b51e1
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 12:22:47 2016 -0500
             
                SmpBatch: don't use mpiexec for a single core
                
                This allows us to remove the influence of mpiexec for more
                simple debugging.
             
             python/lsst/ctrl/pool/parallel.py | 5 ++++-
             1 file changed, 4 insertions(+), 1 deletion(-)
             
             
            price@price-laptop:~/LSST/obs/subaru (tickets/DM-3373=) $ git sub
            commit 895cf80bd07430d2c7bfe2c1b1966679d9271d2a
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:50:13 2016 -0500
             
                Adjust to new calib construction code ported from HSC
                
                We have ported the calib construction code (detrends.py from hscPipe) into
                LSST, which involved some renames (detrend --> calib; hsc.pipe.tasks -->
                lsst.pipe.drivers).
             
             config/hsc/flat.py              |   2 +-
             policy/HscMapper.paf            |   8 +--
             python/lsst/obs/hsc/calibs.py   | 112 ++++++++++++++++++++++++++++++++++++++++
             python/lsst/obs/hsc/detrends.py | 112 ----------------------------------------
             4 files changed, 117 insertions(+), 117 deletions(-)
             
            commit f8c99cdd17840729485028f0efcc8fe22cee1ba4
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:51:36 2016 -0500
             
                Support calib registry creation with ingestCalibs
                
                ingestCalibs.py is the camera-agnostic way of building a calib registry,
                replacing genCalibRegistry.py (as ingest.py replaced genInputRegistry.py).
                It requires a little customisation for HSC.
             
             config/ingestCalibs.py           | 20 ++++++++++++++++++++
             python/lsst/obs/subaru/ingest.py | 19 +++++++++++++++++++
             2 files changed, 39 insertions(+)
             
            commit 74be2ce3e57ebf010789562fd2e54b0ebe6937d2
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:52:57 2016 -0500
             
                isr: don't attempt linearity correction on focus CCDs
                
                We don't have linearity coefficients for the focus CCDs, so can't do a
                linearity correction: just warn the user.
             
             python/lsst/obs/subaru/isr.py | 5 +++++
             1 file changed, 5 insertions(+)
             
            commit 5865221760d12a66d6a2fb308231954cee0fd27f
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:58:34 2016 -0500
             
                XXX Grab-bag of fixes from Lauren to get calib construction working before she merges her work.
                
                Conflicts:
                	python/lsst/obs/subaru/isr.py
             
             python/lsst/obs/subaru/crosstalk.py |  8 ++--
             python/lsst/obs/subaru/isr.py       | 78 ++++++++++++++++++++++++++++---------
             2 files changed, 64 insertions(+), 22 deletions(-)
             
             
            price@price-laptop:~/LSST/pipe/tasks (tickets/DM-3373=) $ git sub
            commit d55d48c0b22b208c85f3d5e4ba30490d50585414
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 16:59:29 2016 -0500
             
                ingestCalibs: fix output directory for registry
                
                We aim to have the calib registry end up in the calib root directory. To
                this end, we modify RegisterTask.openRegistry to take a directory instead
                of the butler (the butler was only being used to determine the directory).
                
                Our only access to the calib root directory is through the "--calib"
                command-line argument; if that is not provided, write to the current
                working directory for the user to move into the correct place.
             
             python/lsst/pipe/tasks/ingest.py       | 8 ++++----
             python/lsst/pipe/tasks/ingestCalibs.py | 7 ++++---
             2 files changed, 8 insertions(+), 7 deletions(-)
             
            commit 380e99f6bd759382478447f276bd211cd1c52867
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:16:05 2016 -0500
             
                ingestCalibs: make camera-agnostic
                
                Several assumptions within ingestCalibs were specific to obs_decam (esp.
                the use of the hard-coded "ccdnum" data identifier keyword). Made these
                assumptions configurable.
                
                This also involved updating the validity range calculations, which have
                been moved to a common place (fixSubsetValidity) instead of being in two
                places (IngestCalibsTask.run and fixSubsetValidity) and requiring users
                to set validStart and validEnd to dummy values.
                
                In the process, updated SQL to use placeholders instead of stringified
                values to proactively prevent escaping problems (similar to SQL injection).
             
             python/lsst/pipe/tasks/ingestCalibs.py | 104 ++++++++++++++++++---------------
             1 file changed, 56 insertions(+), 48 deletions(-)
             
             
            price@price-laptop:~/LSST/obs/decam (tickets/DM-3373=) $ git sub
            commit 5c3fcb5653c6574cd3a892ac7144e1035aca6596
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:20:11 2016 -0500
             
                config: update ingestCalibs following changes in pipe_tasks
                
                ingestCalibs was updated to make it camera-agnostic, which
                requires coordinated changes here. We want to use 'ccdnum' instead
                of 'ccd' in the registry, and need to include the path.
             
             config/ingestCalibs.py                | 1 +
             python/lsst/obs/decam/ingestCalibs.py | 1 +
             2 files changed, 2 insertions(+)
            

            Show
            price Paul Price added a comment - I can now construct and ingest calibration products with HSC. I have not yet tried to construct calibs with other cameras, but the port has been done. This is a fairly large submission (1000 SLOC added in pipe_drivers), so I'd like to divide it into two logical parts: Nate Lust , would you please look at pipe_drivers and the small changes in ctrl_pool and obs_subaru? I hope you will be familiar with this kind of code from your recent work on the coadd driver. In pipe_drivers, I have branched from your work in DM-4855 so that I'm not going to get massive conflicts when merging to master. The commits in obs_subaru contain one commit that I will remove once Lauren MacArthur 's work on rotations for calibs and detrends are merged, so no need to review that one; and the rest in obs_subaru are mostly renames so it looks larger than it really is. Hsin-Fang Chiang , would you please look at the changes to ingestCalib in pipe_tasks and obs_decam? This was originally your code, so would you please check it's true to your intention and that I haven't broken it for your use case with DECam? (I don't have any DECam calibs, or I would check myself.) price@price-laptop:~/LSST/pipe/drivers (tickets/DM-3373=) $ git sub tickets/DM-3373 origin/tickets/DM-4855 commit 06ec186bc0a2a0fb93285f784ab5d49d0d7b81c6 Author: Paul Price <price@astro.princeton.edu> Date: Tue Feb 9 21:39:47 2016 -0500   Add calib construction code from hscPipe   bin.src/constructBias.py | 3 + bin.src/constructDark.py | 3 + bin.src/constructFlat.py | 3 + bin.src/constructFringe.py | 3 + bin.src/constructMask.py | 3 + python/lsst/pipe/drivers/checksum.py | 59 ++ python/lsst/pipe/drivers/constructCalibs.py | 978 ++++++++++++++++++++++++++++ 7 files changed, 1052 insertions(+)   commit b7f2955de1152fde4ec335425b05490cad084ba8 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 16:51:34 2016 -0500   git: ignore generated files   .gitignore | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)     commit 5792d30413fbf76bcf7a180a1924f987635b51e1 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 12:22:47 2016 -0500   SmpBatch: don't use mpiexec for a single core This allows us to remove the influence of mpiexec for more simple debugging.   python/lsst/ctrl/pool/parallel.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)     price@price-laptop:~/LSST/obs/subaru (tickets/DM-3373=) $ git sub commit 895cf80bd07430d2c7bfe2c1b1966679d9271d2a Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:50:13 2016 -0500   Adjust to new calib construction code ported from HSC We have ported the calib construction code (detrends.py from hscPipe) into LSST, which involved some renames (detrend --> calib; hsc.pipe.tasks --> lsst.pipe.drivers).   config/hsc/flat.py | 2 +- policy/HscMapper.paf | 8 +-- python/lsst/obs/hsc/calibs.py | 112 ++++++++++++++++++++++++++++++++++++++++ python/lsst/obs/hsc/detrends.py | 112 ---------------------------------------- 4 files changed, 117 insertions(+), 117 deletions(-)   commit f8c99cdd17840729485028f0efcc8fe22cee1ba4 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:51:36 2016 -0500   Support calib registry creation with ingestCalibs ingestCalibs.py is the camera-agnostic way of building a calib registry, replacing genCalibRegistry.py (as ingest.py replaced genInputRegistry.py). It requires a little customisation for HSC.   config/ingestCalibs.py | 20 ++++++++++++++++++++ python/lsst/obs/subaru/ingest.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)   commit 74be2ce3e57ebf010789562fd2e54b0ebe6937d2 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:52:57 2016 -0500   isr: don't attempt linearity correction on focus CCDs We don't have linearity coefficients for the focus CCDs, so can't do a linearity correction: just warn the user.   python/lsst/obs/subaru/isr.py | 5 +++++ 1 file changed, 5 insertions(+)   commit 5865221760d12a66d6a2fb308231954cee0fd27f Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:58:34 2016 -0500   XXX Grab-bag of fixes from Lauren to get calib construction working before she merges her work. Conflicts: python/lsst/obs/subaru/isr.py   python/lsst/obs/subaru/crosstalk.py | 8 ++-- python/lsst/obs/subaru/isr.py | 78 ++++++++++++++++++++++++++++--------- 2 files changed, 64 insertions(+), 22 deletions(-)     price@price-laptop:~/LSST/pipe/tasks (tickets/DM-3373=) $ git sub commit d55d48c0b22b208c85f3d5e4ba30490d50585414 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 16:59:29 2016 -0500   ingestCalibs: fix output directory for registry We aim to have the calib registry end up in the calib root directory. To this end, we modify RegisterTask.openRegistry to take a directory instead of the butler (the butler was only being used to determine the directory). Our only access to the calib root directory is through the "--calib" command-line argument; if that is not provided, write to the current working directory for the user to move into the correct place.   python/lsst/pipe/tasks/ingest.py | 8 ++++---- python/lsst/pipe/tasks/ingestCalibs.py | 7 ++++--- 2 files changed, 8 insertions(+), 7 deletions(-)   commit 380e99f6bd759382478447f276bd211cd1c52867 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:16:05 2016 -0500   ingestCalibs: make camera-agnostic Several assumptions within ingestCalibs were specific to obs_decam (esp. the use of the hard-coded "ccdnum" data identifier keyword). Made these assumptions configurable. This also involved updating the validity range calculations, which have been moved to a common place (fixSubsetValidity) instead of being in two places (IngestCalibsTask.run and fixSubsetValidity) and requiring users to set validStart and validEnd to dummy values. In the process, updated SQL to use placeholders instead of stringified values to proactively prevent escaping problems (similar to SQL injection).   python/lsst/pipe/tasks/ingestCalibs.py | 104 ++++++++++++++++++--------------- 1 file changed, 56 insertions(+), 48 deletions(-)     price@price-laptop:~/LSST/obs/decam (tickets/DM-3373=) $ git sub commit 5c3fcb5653c6574cd3a892ac7144e1035aca6596 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:20:11 2016 -0500   config: update ingestCalibs following changes in pipe_tasks ingestCalibs was updated to make it camera-agnostic, which requires coordinated changes here. We want to use 'ccdnum' instead of 'ccd' in the registry, and need to include the path.   config/ingestCalibs.py | 1 + python/lsst/obs/decam/ingestCalibs.py | 1 + 2 files changed, 2 insertions(+)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for fixing ingestCalibs and making it camera-agnostic! With some minor edits it works with DECam data. Most is about the changes of handling validStart/validEnd. Just to make them clear, I added them as extra commits in branch u/hfc/DM-3733 but please take as you see fit and squash them.

            I used to translate 'validStart' and 'validEnd', and default them to the same value as 'calibDate' (https://github.com/lsst/obs_decam/blob/master/config/ingestCalibs.py#L13-L15 ). Now they no longer need to be included in config.parse.translators in obs_decam. But an extra line is needed in pipe_tasks (see the commit in u/hfc/DM-3733).

            Also a correction about setting 'path' in obs_decam. I probably should comment those code better. They are to handle the Community Pipeline calibration products which come in multiple different forms...

            Finally, if the command-line argument "--validity" is not given, it's possible to get errors like this:

             File "/Users/hchiang2/lsst_packages/pipe_tasks/python/lsst/pipe/tasks/ingestCalibs.py", line 129, in fixSubsetValidity
                valids[dd] = [dd - datetime.timedelta(validity), dd + datetime.timedelta(validity)]
            TypeError: unsupported type for timedelta days component: NoneType
            

            This error message may look a bit obscure to some users. Maybe give a warning, or require it, or add a default? A default of 0 would be equivalent to the old behavior in obs_decam, that is, the data are only valid during the same observing night. But we don't have to keep that old behavior.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for fixing ingestCalibs and making it camera-agnostic! With some minor edits it works with DECam data. Most is about the changes of handling validStart/validEnd. Just to make them clear, I added them as extra commits in branch u/hfc/ DM-3733 but please take as you see fit and squash them. I used to translate 'validStart' and 'validEnd', and default them to the same value as 'calibDate' ( https://github.com/lsst/obs_decam/blob/master/config/ingestCalibs.py#L13-L15 ). Now they no longer need to be included in config.parse.translators in obs_decam. But an extra line is needed in pipe_tasks (see the commit in u/hfc/ DM-3733 ). Also a correction about setting 'path' in obs_decam. I probably should comment those code better. They are to handle the Community Pipeline calibration products which come in multiple different forms... Finally, if the command-line argument "--validity" is not given, it's possible to get errors like this: File "/Users/hchiang2/lsst_packages/pipe_tasks/python/lsst/pipe/tasks/ingestCalibs.py" , line 129 , in fixSubsetValidity valids[dd] = [dd - datetime.timedelta(validity), dd + datetime.timedelta(validity)] TypeError: unsupported type for timedelta days component: NoneType This error message may look a bit obscure to some users. Maybe give a warning, or require it, or add a default? A default of 0 would be equivalent to the old behavior in obs_decam, that is, the data are only valid during the same observing night. But we don't have to keep that old behavior.
            Hide
            nlust Nate Lust added a comment -

            A few minor comments in places. Feel free to fix them up, or justify what is already there and merge.

            Show
            nlust Nate Lust added a comment - A few minor comments in places. Feel free to fix them up, or justify what is already there and merge.
            Hide
            price Paul Price added a comment -

            Thank you both for your reviews. I didn't see anything major, so cleaned a few things up, removed the unused code, and am about to merge now that DM-4855 has merged.

            Show
            price Paul Price added a comment - Thank you both for your reviews. I didn't see anything major, so cleaned a few things up, removed the unused code, and am about to merge now that DM-4855 has merged.
            Hide
            price Paul Price added a comment -

            Going to test this on some HSC data, try a Jenkins run, then merge.

            price@price-laptop:~/LSST/pipe/drivers (tickets/DM-3373=) $ git sub origin/tickets/DM-3373 origin/tickets/DM-4855
            commit 35330de73c263c3105ac5c5039291a404842f4b8
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Feb 9 21:39:47 2016 -0500
             
                Add calib construction code from hscPipe
             
             bin.src/constructBias.py                    |   3 +
             bin.src/constructDark.py                    |   3 +
             bin.src/constructFlat.py                    |   3 +
             bin.src/constructFringe.py                  |   3 +
             python/lsst/pipe/drivers/checksum.py        |  58 ++++
             python/lsst/pipe/drivers/constructCalibs.py | 864 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
             6 files changed, 934 insertions(+)
             
            commit c23f91d18fbc01773962ec0b8a48ba726be95ed4
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 16:51:34 2016 -0500
             
                git: ignore generated files
             
             .gitignore | 4 +++-
             1 file changed, 3 insertions(+), 1 deletion(-)
             
             
            price@price-laptop:~/LSST/ctrl/pool (tickets/DM-3373=) $ git sub
            commit 5792d30413fbf76bcf7a180a1924f987635b51e1
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 12:22:47 2016 -0500
             
                SmpBatch: don't use mpiexec for a single core
                
                This allows us to remove the influence of mpiexec for more
                simple debugging.
             
             python/lsst/ctrl/pool/parallel.py | 5 ++++-
             1 file changed, 4 insertions(+), 1 deletion(-)
             
             
            price@price-laptop:~/LSST/obs/subaru (tickets/DM-3373=) $ git sub
            commit f546dcf3be33fd335ea2ff83904cfb40c272c9a2
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:50:13 2016 -0500
             
                Adjust to new calib construction code ported from HSC
                
                We have ported the calib construction code (detrends.py from hscPipe) into
                LSST, which involved some renames (detrend --> calib; hsc.pipe.tasks -->
                lsst.pipe.drivers).
             
             config/hsc/flat.py              |   2 +-
             policy/HscMapper.paf            |   8 +++---
             python/lsst/obs/hsc/calibs.py   | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
             python/lsst/obs/hsc/detrends.py | 112 -----------------------------------------------------------------------
             4 files changed, 117 insertions(+), 117 deletions(-)
             
            commit a79c5a7c0dde4fd55c1ce714b67a276ac7b0aab3
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:51:36 2016 -0500
             
                Support calib registry creation with ingestCalibs
                
                ingestCalibs.py is the camera-agnostic way of building a calib registry,
                replacing genCalibRegistry.py (as ingest.py replaced genInputRegistry.py).
                It requires a little customisation for HSC.
             
             config/ingestCalibs.py           | 20 ++++++++++++++++++++
             python/lsst/obs/subaru/ingest.py | 19 +++++++++++++++++++
             2 files changed, 39 insertions(+)
             
            commit bfe262ad10591f3edbe9482c0120307f790e370b
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:52:57 2016 -0500
             
                isr: don't attempt linearity correction on focus CCDs
                
                We don't have linearity coefficients for the focus CCDs, so can't do a
                linearity correction: just warn the user.
             
             python/lsst/obs/subaru/isr.py | 5 +++++
             1 file changed, 5 insertions(+)
             
             
            price@price-laptop:~/LSST/pipe/tasks (tickets/DM-3373=) $ git sub
            commit d55d48c0b22b208c85f3d5e4ba30490d50585414
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 16:59:29 2016 -0500
             
                ingestCalibs: fix output directory for registry
                
                We aim to have the calib registry end up in the calib root directory. To
                this end, we modify RegisterTask.openRegistry to take a directory instead
                of the butler (the butler was only being used to determine the directory).
                
                Our only access to the calib root directory is through the "--calib"
                command-line argument; if that is not provided, write to the current
                working directory for the user to move into the correct place.
             
             python/lsst/pipe/tasks/ingest.py       | 8 ++++----
             python/lsst/pipe/tasks/ingestCalibs.py | 7 ++++---
             2 files changed, 8 insertions(+), 7 deletions(-)
             
            commit 380e99f6bd759382478447f276bd211cd1c52867
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:16:05 2016 -0500
             
                ingestCalibs: make camera-agnostic
                
                Several assumptions within ingestCalibs were specific to obs_decam (esp.
                the use of the hard-coded "ccdnum" data identifier keyword). Made these
                assumptions configurable.
                
                This also involved updating the validity range calculations, which have
                been moved to a common place (fixSubsetValidity) instead of being in two
                places (IngestCalibsTask.run and fixSubsetValidity) and requiring users
                to set validStart and validEnd to dummy values.
                
                In the process, updated SQL to use placeholders instead of stringified
                values to proactively prevent escaping problems (similar to SQL injection).
             
             python/lsst/pipe/tasks/ingestCalibs.py | 104 ++++++++++++++++++++++++++++++++++------------------------------
             1 file changed, 56 insertions(+), 48 deletions(-)
             
             
            git subprice@price-laptop:~/LSST/obs/decam (tickets/DM-3373=) $ git sub
            commit f03acd9222daf5c413f51cd8df897a145ac2e199
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 11 17:20:11 2016 -0500
             
                config: update ingestCalibs following changes in pipe_tasks
                
                ingestCalibs was updated to make it camera-agnostic, which
                requires coordinated changes here. We want to use 'ccdnum' instead
                of 'ccd' in the registry, and need to include the path.
                
                Thanks for Hsin-Fang Chiang for validating this on DECam data
                and providing some additional fixes.
             
             config/ingestCalibs.py                | 3 +--
             python/lsst/obs/decam/ingestCalibs.py | 7 ++++---
             2 files changed, 5 insertions(+), 5 deletions(-)
            

            Show
            price Paul Price added a comment - Going to test this on some HSC data, try a Jenkins run, then merge. price@price-laptop:~/LSST/pipe/drivers (tickets/DM-3373=) $ git sub origin/tickets/DM-3373 origin/tickets/DM-4855 commit 35330de73c263c3105ac5c5039291a404842f4b8 Author: Paul Price <price@astro.princeton.edu> Date: Tue Feb 9 21:39:47 2016 -0500   Add calib construction code from hscPipe   bin.src/constructBias.py | 3 + bin.src/constructDark.py | 3 + bin.src/constructFlat.py | 3 + bin.src/constructFringe.py | 3 + python/lsst/pipe/drivers/checksum.py | 58 ++++ python/lsst/pipe/drivers/constructCalibs.py | 864 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 934 insertions(+)   commit c23f91d18fbc01773962ec0b8a48ba726be95ed4 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 16:51:34 2016 -0500   git: ignore generated files   .gitignore | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)     price@price-laptop:~/LSST/ctrl/pool (tickets/DM-3373=) $ git sub commit 5792d30413fbf76bcf7a180a1924f987635b51e1 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 12:22:47 2016 -0500   SmpBatch: don't use mpiexec for a single core This allows us to remove the influence of mpiexec for more simple debugging.   python/lsst/ctrl/pool/parallel.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)     price@price-laptop:~/LSST/obs/subaru (tickets/DM-3373=) $ git sub commit f546dcf3be33fd335ea2ff83904cfb40c272c9a2 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:50:13 2016 -0500   Adjust to new calib construction code ported from HSC We have ported the calib construction code (detrends.py from hscPipe) into LSST, which involved some renames (detrend --> calib; hsc.pipe.tasks --> lsst.pipe.drivers).   config/hsc/flat.py | 2 +- policy/HscMapper.paf | 8 +++--- python/lsst/obs/hsc/calibs.py | 112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ python/lsst/obs/hsc/detrends.py | 112 ----------------------------------------------------------------------- 4 files changed, 117 insertions(+), 117 deletions(-)   commit a79c5a7c0dde4fd55c1ce714b67a276ac7b0aab3 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:51:36 2016 -0500   Support calib registry creation with ingestCalibs ingestCalibs.py is the camera-agnostic way of building a calib registry, replacing genCalibRegistry.py (as ingest.py replaced genInputRegistry.py). It requires a little customisation for HSC.   config/ingestCalibs.py | 20 ++++++++++++++++++++ python/lsst/obs/subaru/ingest.py | 19 +++++++++++++++++++ 2 files changed, 39 insertions(+)   commit bfe262ad10591f3edbe9482c0120307f790e370b Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:52:57 2016 -0500   isr: don't attempt linearity correction on focus CCDs We don't have linearity coefficients for the focus CCDs, so can't do a linearity correction: just warn the user.   python/lsst/obs/subaru/isr.py | 5 +++++ 1 file changed, 5 insertions(+)     price@price-laptop:~/LSST/pipe/tasks (tickets/DM-3373=) $ git sub commit d55d48c0b22b208c85f3d5e4ba30490d50585414 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 16:59:29 2016 -0500   ingestCalibs: fix output directory for registry We aim to have the calib registry end up in the calib root directory. To this end, we modify RegisterTask.openRegistry to take a directory instead of the butler (the butler was only being used to determine the directory). Our only access to the calib root directory is through the "--calib" command-line argument; if that is not provided, write to the current working directory for the user to move into the correct place.   python/lsst/pipe/tasks/ingest.py | 8 ++++---- python/lsst/pipe/tasks/ingestCalibs.py | 7 ++++--- 2 files changed, 8 insertions(+), 7 deletions(-)   commit 380e99f6bd759382478447f276bd211cd1c52867 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:16:05 2016 -0500   ingestCalibs: make camera-agnostic Several assumptions within ingestCalibs were specific to obs_decam (esp. the use of the hard-coded "ccdnum" data identifier keyword). Made these assumptions configurable. This also involved updating the validity range calculations, which have been moved to a common place (fixSubsetValidity) instead of being in two places (IngestCalibsTask.run and fixSubsetValidity) and requiring users to set validStart and validEnd to dummy values. In the process, updated SQL to use placeholders instead of stringified values to proactively prevent escaping problems (similar to SQL injection).   python/lsst/pipe/tasks/ingestCalibs.py | 104 ++++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 56 insertions(+), 48 deletions(-)     git subprice@price-laptop:~/LSST/obs/decam (tickets/DM-3373=) $ git sub commit f03acd9222daf5c413f51cd8df897a145ac2e199 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 11 17:20:11 2016 -0500   config: update ingestCalibs following changes in pipe_tasks ingestCalibs was updated to make it camera-agnostic, which requires coordinated changes here. We want to use 'ccdnum' instead of 'ccd' in the registry, and need to include the path. Thanks for Hsin-Fang Chiang for validating this on DECam data and providing some additional fixes.   config/ingestCalibs.py | 3 +-- python/lsst/obs/decam/ingestCalibs.py | 7 ++++--- 2 files changed, 5 insertions(+), 5 deletions(-)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -
            Show
            hchiang2 Hsin-Fang Chiang added a comment - Paul Price This line seems to be left out I guess it's an oversight? https://github.com/lsst/pipe_tasks/commit/0b69c1a097967161784f95a253e569a1fed68fbd
            Hide
            price Paul Price added a comment -

            Thanks Hsin-Fang Chiang, got it!

            Show
            price Paul Price added a comment - Thanks Hsin-Fang Chiang , got it!
            Hide
            price Paul Price added a comment -

            Merged to master.

            Show
            price Paul Price added a comment - Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Hsin-Fang Chiang, Nate Lust
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, Nate Lust, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: