Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_pool, meas_astrom, pipe_tasks
-
Labels:
-
Epic Link:
-
Team:Data Release Production
Description
Port outstanding work and release.
Essential:
DM-9541: Bug related to MPI pickling when running coaddDriverDM-9812: Clean up outputs from CharacterizeImageTask and CalibrateTaskDM-9438: Switch default reference catalog for HSC to PS1 in LSST formatDM-9855: Select images for coadd based on PSF qualityDM-9810: Make PSFEx oversampling configurableDM-9856: Add option to deblender to turn off smoothing.DM-9795: CModel priors are weighted incorrectly relative to likelihoodDM-9871: Move wcs and fcr datasets out of pointing directory in obs_subaruDM-9874: Increase matcher.numBrightStarsDM-9862: Update meas_mosaic's wcs/fcr output files to reflect LSST coordinate system
If possible (otherwise, will go into post-release update):
Attachments
Issue Links
- blocks
-
DM-10129 Process HSC RC dataset using Stack version chosen for the full S17B reprocessing
- Done
- is blocked by
-
DM-9541 Bug related to MPI pickling when running coaddDriver
- Done
-
DM-9795 CModel priors are weighted incorrectly relative to likelihood
- Done
-
DM-9438 Switch default reference catalog for HSC to PS1 in LSST format
- Done
-
DM-9553 Investigate the best algorithm to compute derivatives for the Brighter-Fatter correction
- Done
-
DM-9862 Update meas_mosaic's wcs/fcr output files to reflect LSST coordinate system
- Done
-
DM-9810 Make PSFEx oversampling configurable
- Done
-
DM-9812 Clean up outputs from CharacterizeImageTask and CalibrateTask
- Done
-
DM-9855 Select images for coadd based on PSF quality.
- Done
-
DM-9856 Add option to deblender to turn off smoothing.
- Done
-
DM-9871 Move wcs and fcr datasets out of {pointing} directory in obs_subaru
- Done
-
DM-9874 Increase matcher.numBrightStars
- Done
Activity
That's correct.
DM-9871 done:
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 4173d19 lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/obs_subaru ((ef3f7c6b...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
ef3f7c6b Merge branch 'tickets/DM-9871'
|
16a93b58 mappers: change wcs,fcr file templates
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> ef3f7c6b
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge ef3f7c6b
|
Updating cd9e2ab7..ef3f7c6b
|
Fast-forward
|
policy/HscMapper.paf | 4 ++--
|
policy/SuprimecamMapper.paf | 4 ++--
|
2 files changed, 4 insertions(+), 4 deletions(-)
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
|
commit ef3f7c6bc5f9d4808105a0582bf276583e9f420a
|
Merge: cd9e2ab7 16a93b58
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Mon Mar 20 12:39:58 2017 -0400
|
|
Merge branch 'tickets/DM-9871'
|
|
commit 16a93b58b0ebc7feb504ea3f2534cc3bffd7aca7
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Mon Mar 20 11:23:32 2017 -0400
|
|
mappers: change wcs,fcr file templates
|
|
Writing in a directory outside of the 'pointing' directory
|
avoids problems in copying/updating directories at NAOJ during
|
incremental releases.
|
DM-9874 done:
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master efab3bc lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/obs_subaru ((e09d034b...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
e09d034b Merge branch 'tickets/DM-9874'
|
3f4a1808 config: set matcher.numBrightStars=150
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> e09d034b
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge e09d034b
|
Updating ef3f7c6b..e09d034b
|
Fast-forward
|
config/processCcd.py | 4 ++++
|
1 file changed, 4 insertions(+)
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
|
commit e09d034bb4f3febd61997accae39b01cfb5c757b
|
Merge: ef3f7c6b 3f4a1808
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Mon Mar 20 14:00:52 2017 -0400
|
|
Merge branch 'tickets/DM-9874'
|
|
commit 3f4a1808e53f137e639dd1baa3d7499e088d76cd
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Mon Mar 20 13:00:34 2017 -0400
|
|
config: set matcher.numBrightStars=150
|
|
This greatly increases the stability of the matcher, at least for
|
MADCASH dwarf galaxy fields. There's also evidence from UW astrometry
|
experiments that it helps in general.
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git push
|
Counting objects: 5, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (5/5), done.
|
Writing objects: 100% (5/5), 684 bytes | 0 bytes/s, done.
|
Total 5 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/obs_subaru.git
|
ef3f7c6b..e09d034b master -> master
|
DM-9438 done:
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase --onto master e98f607 lsst/master
|
First, rewinding head to replay your work on top of it...
|
Applying: config: use PS1 reference catalog in LSST format
|
price@pap-laptop:~/hsc/obs_subaru ((ddceaac6...)) $ git co master
|
Warning: you are leaving 1 commit behind, not connected to
|
any of your branches:
|
|
ddceaac6 config: use PS1 reference catalog in LSST format
|
|
If you want to keep it by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> ddceaac6
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge ddceaac6
|
Updating e09d034b..ddceaac6
|
Fast-forward
|
config/Mosaic.py | 13 ++++---------
|
config/measureCoaddSources.py | 4 +++-
|
config/processCcd.py | 14 ++++----------
|
3 files changed, 11 insertions(+), 20 deletions(-)
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
|
commit ddceaac6341899e2a1ee6c30cc02ca031617e08d
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Mon Mar 20 13:55:30 2017 -0400
|
|
config: use PS1 reference catalog in LSST format
|
|
This frees us from dependency on astrometry_net, and removes our reliance
|
on eups for setting the configuration.
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git push
|
Counting objects: 6, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (6/6), done.
|
Writing objects: 100% (6/6), 961 bytes | 0 bytes/s, done.
|
Total 6 (delta 5), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (5/5), completed with 4 local objects.
|
To github.com:HyperSuprime-Cam/obs_subaru.git
|
e09d034b..ddceaac6 master -> master
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git rebase -p --onto master f76c79a lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/meas_extensions_psfex ((423c6dd...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
423c6dd Merge branch 'tickets/DM-9810'
|
9b2b692 Add config parameter for BASIS_TYPE
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> 423c6dd
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master=) $ git merge 423c6dd
|
Updating 1b4ab0c..423c6dd
|
Fast-forward
|
python/lsst/meas/extensions/psfex/psfexPsfDeterminer.py | 13 +++++++++++++
|
1 file changed, 13 insertions(+)
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git out
|
commit 423c6dd4409b74f58f625e99ccea30baf5033800
|
Merge: 1b4ab0c 9b2b692
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:38:24 2017 -0400
|
|
Merge branch 'tickets/DM-9810'
|
|
commit 9b2b69271208aaddd3a1aaeb77066c6fac4fecdc
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Fri Mar 17 13:05:18 2017 -0400
|
|
Add config parameter for BASIS_TYPE
|
price@pap-laptop:~/hsc/meas_extensions_psfex (master>) $ git push
|
Counting objects: 9, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (8/8), done.
|
Writing objects: 100% (9/9), 1.05 KiB | 0 bytes/s, done.
|
Total 9 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/meas_extensions_psfex.git
|
1b4ab0c..423c6dd master -> master
|
|
|
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 60c4f2d lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/obs_subaru ((3ec4d6f1...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
3ec4d6f1 Merge branch 'tickets/DM-9810'
|
ee7cef3f Change psfex oversampling options.
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> 3ec4d6f1
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git merge 3ec4d6f1
|
Updating ddceaac6..3ec4d6f1
|
Fast-forward
|
config/processCcd.py | 3 +++
|
1 file changed, 3 insertions(+)
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git out
|
commit 3ec4d6f12002a8396fa636398d5920e0bf56511e
|
Merge: ddceaac6 ee7cef3f
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:42:08 2017 -0400
|
|
Merge branch 'tickets/DM-9810'
|
|
commit ee7cef3ff06583f094a897451d9e6f157f1bc164
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 12:42:32 2017 -0400
|
|
Change psfex oversampling options.
|
|
This commit forces psfex to use the native pixel basis
|
if the FWHM > 3 pixels. Otherwise, it will 2x oversample the
|
PSF model.
|
price@pap-laptop:~/hsc/obs_subaru (master>) $ git push
|
Counting objects: 5, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (5/5), done.
|
Writing objects: 100% (5/5), 661 bytes | 0 bytes/s, done.
|
Total 5 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/obs_subaru.git
|
ddceaac6..3ec4d6f1 master -> master
|
|
|
|
price@pap-laptop:~/hsc/meas_deblender (master=) $ git rebase -p --onto master e6413fc lsst/master
|
Successfully rebased and updated detached HEAD.
|
price@pap-laptop:~/hsc/meas_deblender ((dadfd13...)) $ git co master
|
Warning: you are leaving 2 commits behind, not connected to
|
any of your branches:
|
|
dadfd13 Merge branch 'tickets/DM-9856'
|
042d331 Add medianSmoothTemplate as config parameter
|
|
If you want to keep them by creating a new branch, this may be a good time
|
to do so with:
|
|
git branch <new-branch-name> dadfd13
|
|
Switched to branch 'master'
|
Your branch is up-to-date with 'origin/master'.
|
price@pap-laptop:~/hsc/meas_deblender (master=) $ git merge dadfd13
|
Updating fd75620..dadfd13
|
Fast-forward
|
python/lsst/meas/deblender/deblend.py | 5 ++++-
|
1 file changed, 4 insertions(+), 1 deletion(-)
|
price@pap-laptop:~/hsc/meas_deblender (master>) $ git out
|
commit dadfd1301d4505e3d505e6cf0a5034402a9dfabb
|
Merge: fd75620 042d331
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Tue Mar 21 15:37:14 2017 -0400
|
|
Merge branch 'tickets/DM-9856'
|
|
commit 042d33132eaabeb9e597c994a4af21396c79983c
|
Author: Bob Armstrong <rearmstr@gmail.com>
|
Date: Fri Mar 17 13:06:15 2017 -0400
|
|
Add medianSmoothTemplate as config parameter
|
price@pap-laptop:~/hsc/meas_deblender (master>) $ git push
|
Counting objects: 8, done.
|
Delta compression using up to 8 threads.
|
Compressing objects: 100% (7/7), done.
|
Writing objects: 100% (8/8), 836 bytes | 0 bytes/s, done.
|
Total 8 (delta 4), reused 0 (delta 0)
|
remote: Resolving deltas: 100% (4/4), completed with 3 local objects.
|
To github.com:HyperSuprime-Cam/meas_deblender.git
|
fd75620..dadfd13 master -> master
|
Planning to defer the following "essential" tickets until the next beta release. This will allow us to start verification testing.
DM-9812 done:
price@pap-laptop:~/hsc/obs_base (master=) $ git rebase -p --onto master 7f06ade lsst/master
|
|
price@pap-laptop:~/hsc/meas_astrom (master=) $ git rebase -p --onto master e3a4b3c lsst/master
|
==> Conflict in __init__.py easily resolved
|
==> Added commit to adapt use of SchemaMapper.join
|
|
price@pap-laptop:~/hsc/pipe_tasks (master=) $ git rebase -p --onto master 4569002 lsst/master
|
|
price@pap-laptop:~/hsc/obs_subaru (master=) $ git rebase -p --onto master 7d65835 lsst/master
|
Released:
This new beta release incorporates some features and bugfixes we anticipate needing for the production run:
•
DM-9541: Bug related to MPI pickling when running coaddDriver
•DM-9812: Clean up outputs from CharacterizeImageTask and CalibrateTask
•DM-9438: Switch default reference catalog for HSC to PS1 in LSST format
•DM-9810: Make PSFEx oversampling configurable
•DM-9856: Add option to deblender to turn off smoothing.
•DM-9871: Move wcs and fcr datasets out of pointing directory in obs_subaru
•DM-9874: Increase matcher.numBrightStarsThis should allow some basic verification runs to start so we can iron out any obvious bugs.
Installation instructions are appended below.
Cheers,
Paul.
Installation notes:
1. Do not install the new hscPipe into the same eups stack as previous versions; prepare a new directory that will hold this and subsequent releases of hscPipe.
2. If you have installed the LSST pipeline, you can try installing into that eups stack (e.g., EUPS_PKGROOT=http://master.ipmu.jp/~price/packages-hscPipe5 eups distrib install hscPipe 5.0-beta1). I haven’t tested that mode yet, but I hope that it will work and not corrupt anything...
3. The software now uses the system version of various third-party packages (e.g., python, numpy, scipy) if they are recent enough. You will be given an option to install Miniconda Python (a miniature version of Anaconda Python) that contains the required packages. We recommend that you do choose to install Miniconda, even if you are confident that you have recent packages, because it will simplify the process in the event you have any problems or notice any bugs.
4. The HSC pipeline now has support for python 3, but I've not yet forked the miniconda3 package. You can try using your own python 3 packages if you want...
5. We now support building the HSC pipeline on Mac OSX with the System Integrity Protection (SIP) feature.
6. Make sure you have the necessary pre-requisites installed. In particular, you need a recent C++ compiler that can handle C++11 (e.g., gcc >= 4.8). See the LSST install page (https://pipelines.lsst.io/install/newinstall.html#prerequisites) for details and installation instructions.Installation instructions:
-1. Install the pre-requisites. See https://pipelines.lsst.io/install/newinstall.html#prerequisites .
0. Start off with a fresh shell.
1. Set up the directory that will host the installation.
mkdir /path/to/hscPipe5
cd /path/to/hscPipe5
2. Set the environment: drop any old stack environment, point to the new stack package source (which will likely change for future releases).
unset EUPS_DIR EUPS_SHELL EUPS_PATH EUPS_PKGROOT SETUP_EUPS
export EUPS_PKGROOT=http://master.ipmu.jp/~price/packages-hscPipe5/
3. Ensure you're using a suitable C++ compiler: need gcc >= 4.8 on Linux. If you don't have it in your environment, take the necessary steps to load it.
4. Download the install script:
wget $EUPS_PKGROOT/newinstall.sh
5. Run the install script; we recommend entering "yes" when asked if you want to install Miniconda.
bash newinstall.sh
6. Source the appropriate setup script for your shell. You'll want to do this each time you use the HSC stack; you may want to put it in your shell startup script.
source /path/to/hscPipe5/loadLSST.bash
7. Install the rest of the stack. For future releases, you'll be able to just run this. A rather long pause (5-10 minutes?) after installing the first package (apr) appears to be common — be patient.
eups distrib install hscPipe 5.0-beta1Support:
1. Look at the LSST installation pages: https://pipelines.lsst.io/install/newinstall.html
2. Search the LSST forum for reports of similar problems: https://community.lsst.org
3. Ask a question on the LSST forum
4. Ask a question on this mailing list
DM-9541has been done, and it was a simple fast-forward merge onto HSC master.