Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_pool, meas_astrom, pipe_tasks
-
Labels:
-
Epic Link:
-
Team:Data Release Production
Description
Port outstanding work and release.
Essential:
DM-9541: Bug related to MPI pickling when running coaddDriver
DM-9812: Clean up outputs from CharacterizeImageTask and CalibrateTaskDM-9438: Switch default reference catalog for HSC to PS1 in LSST formatDM-9855: Select images for coadd based on PSF quality
DM-9810: Make PSFEx oversampling configurableDM-9856: Add option to deblender to turn off smoothing.DM-9795: CModel priors are weighted incorrectly relative to likelihoodDM-9871: Move wcs and fcr datasets out of pointing directory in obs_subaruDM-9874: Increase matcher.numBrightStarsDM-9862: Update meas_mosaic's wcs/fcr output files to reflect LSST coordinate system
If possible (otherwise, will go into post-release update):
Attachments
Issue Links
- blocks
-
- Done
- is blocked by
-
DM-9541 Bug related to MPI pickling when running coaddDriver
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
Activity
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git rebase -p --onto master f76c79a lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/meas_extensions_psfex ((423c6dd...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
|
423c6dd Merge branch 'tickets/DM-9810'
|
9b2b692 Add config parameter for BASIS_TYPE
|
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
|
git branch <new-branch-name> 423c6dd
|
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git merge 423c6dd
|
Updating 1b4ab0c..423c6dd
|
Fast-forward
|
python/lsst/meas/extensions/psfex/psfexPsfDeterminer.py | 13 +++++++++++++
|
1 file changed, 13 insertions(+)
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git out
|
commit 423c6dd4409b74f58f625e99ccea30baf5033800
|
Merge: 1b4ab0c 9b2b692
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:38:24 2017 -0400
|
|
|
Merge branch 'tickets/DM-9810'
|
|
|
commit 9b2b69271208aaddd3a1aaeb77066c6fac4fecdc
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Fri Mar 17 13:05:18 2017 -0400
|
|
|
Add config parameter for BASIS_TYPE
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git push
|
Counting objects: 9, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (8/8), done.
|
Writing objects: 100% (9/9), 1.05 KiB | 0 bytes/s, done.
|
Total 9 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/meas_extensions_psfex.git
|
1b4ab0c..423c6dd master -> master
|
|
|
|
|
|
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 60c4f2d lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/obs_subaru ((3ec4d6f1...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
|
3ec4d6f1 Merge branch 'tickets/DM-9810'
|
ee7cef3f Change psfex oversampling options.
|
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
|
git branch <new-branch-name> 3ec4d6f1
|
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge 3ec4d6f1
|
Updating ddceaac6..3ec4d6f1
|
Fast-forward
|
config/processCcd.py | 3 +++
|
1 file changed, 3 insertions(+)
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
|
commit 3ec4d6f12002a8396fa636398d5920e0bf56511e
|
Merge: ddceaac6 ee7cef3f
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:42:08 2017 -0400
|
|
|
Merge branch 'tickets/DM-9810'
|
|
|
commit ee7cef3ff06583f094a897451d9e6f157f1bc164
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 12:42:32 2017 -0400
|
|
|
Change psfex oversampling options.
|
|
This commit forces psfex to use the native pixel basis
|
if the FWHM > 3 pixels. Otherwise, it will 2x oversample the
|
PSF model.
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git push
|
Counting objects: 5, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (5/5), done.
|
Writing objects: 100% (5/5), 661 bytes | 0 bytes/s, done.
|
Total 5 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/obs_subaru.git
|
ddceaac6..3ec4d6f1 master -> master
|
|
|
|
|
|
|
price@pap-laptop:~/hsc/meas_deblender (master=) $ git rebase -p --onto master e6413fc lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/meas_deblender ((dadfd13...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
|
dadfd13 Merge branch 'tickets/DM-9856'
|
042d331 Add medianSmoothTemplate as config parameter
|
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
|
git branch <new-branch-name> dadfd13
|
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/meas_deblender (master=) $ git merge dadfd13
|
Updating fd75620..dadfd13
|
Fast-forward
|
python/lsst/meas/deblender/deblend.py | 5 ++++-
|
1 file changed, 4 insertions(+), 1 deletion(-)
|
price@pap-laptop:~/hsc/meas_deblender (master>) $ git out
|
commit dadfd1301d4505e3d505e6cf0a5034402a9dfabb
|
Merge: fd75620 042d331
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:37:14 2017 -0400
|
|
|
Merge branch 'tickets/DM-9856'
|
|
|
commit 042d33132eaabeb9e597c994a4af21396c79983c
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Fri Mar 17 13:06:15 2017 -0400
|
|
|
Add medianSmoothTemplate as config parameter
|
price@pap-laptop:~/hsc/meas_deblender (master>) $ git push
|
Counting objects: 8, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (7/7), done.
|
Writing objects: 100% (8/8), 836 bytes | 0 bytes/s, done.
|
Total 8 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/meas_deblender.git
|
fd75620..dadfd13 master -> master
|
Paul Price
added a comment - DM-9810 and DM-9856 done:
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git rebase -p --onto master f76c79a lsst/master
Successfully rebased and updated detached HEAD.
price@pap-laptop:~/hsc/meas_extensions_psfex ((423c6dd...)) $ git co master
Warning: you are leaving 2 commits behind, not connected to
any of your branches:
423c6dd Merge branch 'tickets/DM-9810'
9b2b692 Add config parameter for BASIS_TYPE
If you want to keep them by creating a new branch, this may be a good time
to do so with:
git branch <new-branch-name> 423c6dd
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git merge 423c6dd
Updating 1b4ab0c..423c6dd
Fast-forward
python/lsst/meas/extensions/psfex/psfexPsfDeterminer.py | 13 +++++++++++++
1 file changed, 13 insertions(+)
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git out
commit 423c6dd4409b74f58f625e99ccea30baf5033800
Merge: 1b4ab0c 9b2b692
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Tue Mar 21 15:38:24 2017 -0400
Merge branch 'tickets/DM-9810'
commit 9b2b69271208aaddd3a1aaeb77066c6fac4fecdc
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Fri Mar 17 13:05:18 2017 -0400
Add config parameter for BASIS_TYPE
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git push
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (9/9), 1.05 KiB | 0 bytes/s, done.
Total 9 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
To github.com:HyperSuprime-Cam/meas_extensions_psfex.git
1b4ab0c..423c6dd master -> master
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 60c4f2d lsst/master
Successfully rebased and updated detached HEAD.
price@pap-laptop:~/hsc/obs_subaru ((3ec4d6f1...)) $ git co master
Warning: you are leaving 2 commits behind, not connected to
any of your branches:
3ec4d6f1 Merge branch 'tickets/DM-9810'
ee7cef3f Change psfex oversampling options.
If you want to keep them by creating a new branch, this may be a good time
to do so with:
git branch <new-branch-name> 3ec4d6f1
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge 3ec4d6f1
Updating ddceaac6..3ec4d6f1
Fast-forward
config/processCcd.py | 3 +++
1 file changed, 3 insertions(+)
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
commit 3ec4d6f12002a8396fa636398d5920e0bf56511e
Merge: ddceaac6 ee7cef3f
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Tue Mar 21 15:42:08 2017 -0400
Merge branch 'tickets/DM-9810'
commit ee7cef3ff06583f094a897451d9e6f157f1bc164
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Tue Mar 21 12:42:32 2017 -0400
Change psfex oversampling options.
This commit forces psfex to use the native pixel basis
if the FWHM > 3 pixels. Otherwise, it will 2x oversample the
PSF model.
price@pap-laptop:~/hsc/obs_subaru (master>) $ git push
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 661 bytes | 0 bytes/s, done.
Total 5 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
To github.com:HyperSuprime-Cam/obs_subaru.git
ddceaac6..3ec4d6f1 master -> master
price@pap-laptop:~/hsc/meas_deblender (master=) $ git rebase -p --onto master e6413fc lsst/master
Successfully rebased and updated detached HEAD.
price@pap-laptop:~/hsc/meas_deblender ((dadfd13...)) $ git co master
Warning: you are leaving 2 commits behind, not connected to
any of your branches:
dadfd13 Merge branch 'tickets/DM-9856'
042d331 Add medianSmoothTemplate as config parameter
If you want to keep them by creating a new branch, this may be a good time
to do so with:
git branch <new-branch-name> dadfd13
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
price@pap-laptop:~/hsc/meas_deblender (master=) $ git merge dadfd13
Updating fd75620..dadfd13
Fast-forward
python/lsst/meas/deblender/deblend.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
price@pap-laptop:~/hsc/meas_deblender (master>) $ git out
commit dadfd1301d4505e3d505e6cf0a5034402a9dfabb
Merge: fd75620 042d331
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Tue Mar 21 15:37:14 2017 -0400
Merge branch 'tickets/DM-9856'
commit 042d33132eaabeb9e597c994a4af21396c79983c
Author: Bob Armstrong <rearmstr@gmail.com>
Date: Fri Mar 17 13:06:15 2017 -0400
Add medianSmoothTemplate as config parameter
price@pap-laptop:~/hsc/meas_deblender (master>) $ git push
Counting objects: 8, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 836 bytes | 0 bytes/s, done.
Total 8 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
To github.com:HyperSuprime-Cam/meas_deblender.git
fd75620..dadfd13 master -> master
Planning to defer the following "essential" tickets until the next beta release. This will allow us to start verification testing.
Paul Price
added a comment - Planning to defer the following "essential" tickets until the next beta release. This will allow us to start verification testing.
DM-9855 : Select images for coadd based on PSF quality
DM-9795 : CModel priors are weighted incorrectly relative to likelihood
DM-9862 : Update meas_mosaic's wcs/fcr output files to reflect LSST coordinate system
DM-9812 done:
price@pap-laptop:~/hsc/obs_base (master=) $ git rebase -p --onto master 7f06ade lsst/master
|
|
|
price@pap-laptop:~/hsc/meas_astrom (master=) $ git rebase -p --onto master e3a4b3c lsst/master
|
==> Conflict in __init__.py easily resolved
|
==> Added commit to adapt use of SchemaMapper.join
|
|
|
price@pap-laptop:~/hsc/pipe_tasks (master=) $ git rebase -p --onto master 4569002 lsst/master
|
|
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 7d65835 lsst/master
|
Paul Price
added a comment - DM-9812 done:
price@pap-laptop:~/hsc/obs_base (master=) $ git rebase -p --onto master 7f06ade lsst/master
price@pap-laptop:~/hsc/meas_astrom (master=) $ git rebase -p --onto master e3a4b3c lsst/master
==> Conflict in __init__.py easily resolved
==> Added commit to adapt use of SchemaMapper.join
price@pap-laptop:~/hsc/pipe_tasks (master=) $ git rebase -p --onto master 4569002 lsst/master
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 7d65835 lsst/master
Released:
This new beta release incorporates some features and bugfixes we anticipate needing for the production run:
•
DM-9541: Bug related to MPI pickling when running coaddDriver
•DM-9812: Clean up outputs from CharacterizeImageTask and CalibrateTask
•DM-9438: Switch default reference catalog for HSC to PS1 in LSST format
•DM-9810: Make PSFEx oversampling configurable
•DM-9856: Add option to deblender to turn off smoothing.
•DM-9871: Move wcs and fcr datasets out of pointing directory in obs_subaru
•DM-9874: Increase matcher.numBrightStarsThis should allow some basic verification runs to start so we can iron out any obvious bugs.
Installation instructions are appended below.
Cheers,
Paul.
Installation notes:
1. Do not install the new hscPipe into the same eups stack as previous versions; prepare a new directory that will hold this and subsequent releases of hscPipe.
2. If you have installed the LSST pipeline, you can try installing into that eups stack (e.g., EUPS_PKGROOT=http://master.ipmu.jp/~price/packages-hscPipe5 eups distrib install hscPipe 5.0-beta1). I haven’t tested that mode yet, but I hope that it will work and not corrupt anything...
3. The software now uses the system version of various third-party packages (e.g., python, numpy, scipy) if they are recent enough. You will be given an option to install Miniconda Python (a miniature version of Anaconda Python) that contains the required packages. We recommend that you do choose to install Miniconda, even if you are confident that you have recent packages, because it will simplify the process in the event you have any problems or notice any bugs.
4. The HSC pipeline now has support for python 3, but I've not yet forked the miniconda3 package. You can try using your own python 3 packages if you want...
5. We now support building the HSC pipeline on Mac OSX with the System Integrity Protection (SIP) feature.
6. Make sure you have the necessary pre-requisites installed. In particular, you need a recent C++ compiler that can handle C++11 (e.g., gcc >= 4.8). See the LSST install page (https://pipelines.lsst.io/install/newinstall.html#prerequisites) for details and installation instructions.Installation instructions:
-1. Install the pre-requisites. See https://pipelines.lsst.io/install/newinstall.html#prerequisites .
0. Start off with a fresh shell.
1. Set up the directory that will host the installation.
mkdir /path/to/hscPipe5
cd /path/to/hscPipe5
2. Set the environment: drop any old stack environment, point to the new stack package source (which will likely change for future releases).
unset EUPS_DIR EUPS_SHELL EUPS_PATH EUPS_PKGROOT SETUP_EUPS
export EUPS_PKGROOT=http://master.ipmu.jp/~price/packages-hscPipe5/
3. Ensure you're using a suitable C++ compiler: need gcc >= 4.8 on Linux. If you don't have it in your environment, take the necessary steps to load it.
4. Download the install script:
wget $EUPS_PKGROOT/newinstall.sh
5. Run the install script; we recommend entering "yes" when asked if you want to install Miniconda.
bash newinstall.sh
6. Source the appropriate setup script for your shell. You'll want to do this each time you use the HSC stack; you may want to put it in your shell startup script.
source /path/to/hscPipe5/loadLSST.bash
7. Install the rest of the stack. For future releases, you'll be able to just run this. A rather long pause (5-10 minutes?) after installing the first package (apr) appears to be common — be patient.
eups distrib install hscPipe 5.0-beta1Support:
1. Look at the LSST installation pages: https://pipelines.lsst.io/install/newinstall.html
2. Search the LSST forum for reports of similar problems: https://community.lsst.org
3. Ask a question on the LSST forum
4. Ask a question on this mailing list
Paul Price
added a comment - Released:
This new beta release incorporates some features and bugfixes we anticipate needing for the production run:
• DM-9541 : Bug related to MPI pickling when running coaddDriver
• DM-9812 : Clean up outputs from CharacterizeImageTask and CalibrateTask
• DM-9438 : Switch default reference catalog for HSC to PS1 in LSST format
• DM-9810 : Make PSFEx oversampling configurable
• DM-9856 : Add option to deblender to turn off smoothing.
• DM-9871 : Move wcs and fcr datasets out of pointing directory in obs_subaru
• DM-9874 : Increase matcher.numBrightStars
This should allow some basic verification runs to start so we can iron out any obvious bugs.
Installation instructions are appended below.
Cheers,
Paul.
Installation notes:
1. Do not install the new hscPipe into the same eups stack as previous versions; prepare a new directory that will hold this and subsequent releases of hscPipe.
2. If you have installed the LSST pipeline, you can try installing into that eups stack (e.g., EUPS_PKGROOT= http://master.ipmu.jp/~price/packages-hscPipe5 eups distrib install hscPipe 5.0-beta1). I haven’t tested that mode yet, but I hope that it will work and not corrupt anything...
3. The software now uses the system version of various third-party packages (e.g., python, numpy, scipy) if they are recent enough. You will be given an option to install Miniconda Python (a miniature version of Anaconda Python) that contains the required packages. We recommend that you do choose to install Miniconda, even if you are confident that you have recent packages, because it will simplify the process in the event you have any problems or notice any bugs.
4. The HSC pipeline now has support for python 3, but I've not yet forked the miniconda3 package. You can try using your own python 3 packages if you want...
5. We now support building the HSC pipeline on Mac OSX with the System Integrity Protection (SIP) feature.
6. Make sure you have the necessary pre-requisites installed. In particular, you need a recent C++ compiler that can handle C++11 (e.g., gcc >= 4.8). See the LSST install page ( https://pipelines.lsst.io/install/newinstall.html#prerequisites ) for details and installation instructions.
Installation instructions:
-1. Install the pre-requisites. See https://pipelines.lsst.io/install/newinstall.html#prerequisites .
0. Start off with a fresh shell.
1. Set up the directory that will host the installation.
mkdir /path/to/hscPipe5
cd /path/to/hscPipe5
2. Set the environment: drop any old stack environment, point to the new stack package source (which will likely change for future releases).
unset EUPS_DIR EUPS_SHELL EUPS_PATH EUPS_PKGROOT SETUP_EUPS
export EUPS_PKGROOT= http://master.ipmu.jp/~price/packages-hscPipe5/
3. Ensure you're using a suitable C++ compiler: need gcc >= 4.8 on Linux. If you don't have it in your environment, take the necessary steps to load it.
4. Download the install script:
wget $EUPS_PKGROOT/newinstall.sh
5. Run the install script; we recommend entering "yes" when asked if you want to install Miniconda.
bash newinstall.sh
6. Source the appropriate setup script for your shell. You'll want to do this each time you use the HSC stack; you may want to put it in your shell startup script.
source /path/to/hscPipe5/loadLSST.bash
7. Install the rest of the stack. For future releases, you'll be able to just run this. A rather long pause (5-10 minutes?) after installing the first package (apr) appears to be common — be patient.
eups distrib install hscPipe 5.0-beta1
Support:
1. Look at the LSST installation pages: https://pipelines.lsst.io/install/newinstall.html
2. Search the LSST forum for reports of similar problems: https://community.lsst.org
3. Ask a question on the LSST forum
4. Ask a question on this mailing list 
DM-9438done:price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase --onto master e98f607 lsst/masterFirst, rewinding head to replay your work on top of it...Applying: config: use PS1 reference catalog in LSST formatprice@pap-laptop:~/hsc/obs_subaru ((ddceaac6...)) $ git co masterWarning: you are leaving 1 commit behind, not connected toany of your branches:ddceaac6 config: use PS1 reference catalog in LSST formatIf you want to keep it by creating a new branch, this may be a good timeto do so with:git branch <new-branch-name> ddceaac6Switched to branch 'master'Your branch is up-to-date with 'origin/master'.price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge ddceaac6Updating e09d034b..ddceaac6Fast-forwardconfig/Mosaic.py | 13 ++++---------config/measureCoaddSources.py | 4 +++-config/processCcd.py | 14 ++++----------3 files changed, 11 insertions(+), 20 deletions(-)price@pap-laptop:~/hsc/obs_subaru (master>) $ git outcommit ddceaac6341899e2a1ee6c30cc02ca031617e08dAuthor: Paul Price <price@astro.princeton.edu>Date: Mon Mar 20 13:55:30 2017 -0400config: use PS1 reference catalog in LSST formatThis frees us from dependency on astrometry_net, and removes our relianceon eups for setting the configuration.price@pap-laptop:~/hsc/obs_subaru (master>) $ git pushCounting objects: 6, done.Delta compression using up to 8 threads.Compressing objects: 100% (6/6), done.Writing objects: 100% (6/6), 961 bytes | 0 bytes/s, done.Total 6 (delta 5), reused 0 (delta 0)remote: Resolving deltas: 100% (5/5), completed with 4 local objects.To github.com:HyperSuprime-Cam/obs_subaru.gite09d034b..ddceaac6 master -> master