Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13261

Create hscPipe 6.0-beta5 release

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Team:
      External

      Description

      hscPipe 6.0-beta5 will include the following important features and bug fixes:

      • DM-12664: PsfMatched slivers of calexps are worthless
      • DM-12995: Improve detection thresholds and background with temporary sky objects
      • DM-13146: Update HSC defects
      • DM-13182: SpanSet.fromMask produces larger spans than image
      • DM-13227: Refine exposure-wide background subtraction
      • DM-13260: Support construction of new HSC calibs
      • DM-7588: Update MPICH
      • DM-13345: Improve template and warp variance for Warp Compare
      • DM-13408: Fix fringe fix
      • DM-13396: Fix mask propagation on coadds
      • DM-13388: Enable visit-level sky subtraction for HSC by default
      • DM-13389: Enable transmission curve attachment for HSC by default

      Additional features/fixes that we might want:

      • DM-13148: Support Suprime-Cam
      • DM-13244: Port parallel ingest from HSC
      • DM-11776: "RuntimeWarning: invalid value encountered in greater/less" in processing HSC data
      • DM-13380: Add base_PixelFlags_flag_inexact_psfCenter

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment - - edited

            For the maybe list:

            • DM-12373: ExposureInfo has a TransmissionCurve
            • DM-12375: Add TransmissionCurve datasets, attach them during ISR

            (shouldn't block the release unless other things delay it enough for them to sneak in)

            Show
            jbosch Jim Bosch added a comment - - edited For the maybe list: DM-12373 : ExposureInfo has a TransmissionCurve DM-12375 : Add TransmissionCurve datasets, attach them during ISR (shouldn't block the release unless other things delay it enough for them to sneak in)
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            + -DM-12184-: Coaddition Tasks cannot assume that N masks can fit in memory

            + DM-13345:  Add config parameters to control variance in template and warps + Writing nImage even if assemble.doWrite=False

            DM-12692 adapt temporal threshold for N > 50

            Show
            yusra Yusra AlSayyad added a comment - - edited + - DM-12184 -: Coaddition Tasks cannot assume that N masks can fit in memory + DM-13345 :  Add config parameters to control variance in template and warps + Writing nImage even if assemble.doWrite=False +  DM-12692  adapt temporal threshold for N > 50
            Hide
            price Paul Price added a comment -

            Release is out.

            hscPipe 6.0-beta5
             
             
            mpich: ff merge
            * DM-7588: upgrade to mpich 3.2.1
             
            Third-party package upgrades I noticed:
            * flake8
            * pyflakes
            * starlink_ast
            * astshim
             
            sconsUtils: ff merge
            * DM-12527
            * DM-10342
            * DM-12956
             
            daf_persistence: reset
            * DM-12450
            * DM-12656
            * DM-12117
             
            afw: reset (or rebase 2ca2e996b 0e8fc950e ; 7de3051c8 150cece59 ; 8bbfd0c49 bc0f584f6 ; 15c72cc64 2d9284d60 ; 5205ac629 3ec8770c8 ; 278b05818 2835ff510)
            * DM-9953
            * DM-13110
            * DM-13182
            * DM-12740
            * DM-12373
            * DM-12995
            * DM-13189
            * DM-13996***
             
            skymap: ff merge
            * DM-12701
             
            obs_base: ff merge
            * DM-12375
             
            pipe_base: reset
            * DM-12450
            * DM-12531
             
            meas_base: ff merge
            * DM-13380
             
            meas_algorithms: ff merge
            * DM-13055
            * DM-12995
             
            meas_astrom: ff merge
            * DM-12430
            * DM-12431
             
            ip_isr: ff merge
            * DM-12375
            * DM-13260
            * DM-13408
             
            ip_diffim: ff merge
            * DM-12664
            * DM-13325
             
            pipe_tasks: reset
            * DM-13110
            * DM-13244
            * DM-12995
            * DM-12184
            * DM-13345
            * DM-13380
            * DM-12692
            * DM-13396***
             
            pipe_drivers: reset
            * DM-13184
            * DM-13244
            * DM-13227
            * DM-13260
            * DM-13396***
             
            obs_subaru: reset
            * DM-13146
            * DM-12375
            * DM-13260
            * DM-13388
             
            obs_cfht
            obs_decam
            obs_lsstSim
             
            meas_mosaic: ff merge
            * DM-13062
             
             
            Well, we can basically do a "reset lsst/master" on all of the repos.
             
             
            for pkg in $(cat packages.txt dependencies.txt) ; do if [[ ! -e $HOME/hsc/$pkg ]] ; then git clone git@github.com:HyperSuprime-Cam/$pkg.git ; git --git-dir $HOME/hsc/$pkg/.git remote add lsst git@github.com:LSST/$pkg.git ; fi ; git --git-dir $HOME/hsc/$pkg/.git fetch --all ; git --git-dir $HOME/hsc/$pkg/.git reset --hard lsst/master ; done
             
            for pkg in $(cat packages.txt dependencies.txt) ; do git --git-dir $HOME/hsc/$pkg/.git push origin master --tags --force ; done
             
            for pkg in $(cat packages.txt) ; do git --git-dir $HOME/hsc/$pkg/.git tag 6.0b5-hsc -m "HSC 6.0-beta5 release, 2018-01-30" ; git --git-dir $HOME/hsc/$pkg/.git push origin 6.0b5-hsc ; done
             
            price@pap-laptop:~/hsc/starlink_ast (lsst-dev *%<>) $ git reset --hard lsst/lsst-dev
            price@pap-laptop:~/hsc/starlink_ast (lsst-dev=) $ git tag 6.0b5-hsc -m "HSC 6.0-beta5 release, 2018-01-30" --force
            price@pap-laptop:~/hsc/starlink_ast (lsst-dev=) $ git push origin 6.0b5-hsc --force
             
            pprice@tiger-sumire:~ $ unset EUPS_DIR EUPS_SHELL EUPS_PATH EUPS_PKGROOT SETUP_EUPS
            pprice@tiger-sumire:~ $ . /tigress/pprice/hsc-release6/lsstsw/bin/setup.sh
            pprice@tiger-sumire:~ $ module load rh/devtoolset/6
            pprice@tiger-sumire:~ $ rebuild hscPipe
             
            pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ setup hscPipe 6.0-beta0+20
            pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ setup -jr .
            pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ scons install declare version=6.0-beta5
             
            pprice@tiger-sumire:~ $ eups list --depend --raw hscPipe 6.0-beta5 | while IFS=\| read product version ; do eups declare $product $version -t current ; done
            pprice@tiger-sumire:~ $ eups distrib create --server-dir=/tigress/pprice/hsc-release6/hsc-server -S REPOSITORY_PATH='git://github.com/HyperSuprime-Cam/$PRODUCT.git' -f generic -d eupspkg hscPipe 6.0-beta5
            pprice@tiger-sumire:~ $ eups distrib declare -t current --server-dir /tigress/pprice/hsc-release6/hsc-server
            pprice@tiger-sumire:~ $ rsync -rvz /tigress/pprice/hsc-release6/hsc-server/ master-noproxy:public_html/packages-hscPipe6
             
             
            -----
             
            pprice@tiger-sumire:~ $ unsetup eups
            pprice@tiger-sumire:~ $ . /tigress/HSC/stack/hsc3-tiger-20171206/loadLSST.bash 
            pprice@tiger-sumire:~ $ module load rh/devtoolset/6
            pprice@tiger-sumire:~ $ eups distrib install hscPipe 6.0-beta5
             
            pprice@perseus:~ $ unsetup eups
            pprice@perseus:~ $ . /tigress/HSC/stack/hsc3-perseus-20171206/loadLSST.bash 
            pprice@perseus:~ $ module load rh/devtoolset/6
            pprice@perseus:~ $ eups distrib install hscPipe 6.0-beta5
            
            

            Show
            price Paul Price added a comment - Release is out. hscPipe 6.0-beta5     mpich: ff merge * DM-7588: upgrade to mpich 3.2.1   Third-party package upgrades I noticed: * flake8 * pyflakes * starlink_ast * astshim   sconsUtils: ff merge * DM-12527 * DM-10342 * DM-12956   daf_persistence: reset * DM-12450 * DM-12656 * DM-12117   afw: reset (or rebase 2ca2e996b 0e8fc950e ; 7de3051c8 150cece59 ; 8bbfd0c49 bc0f584f6 ; 15c72cc64 2d9284d60 ; 5205ac629 3ec8770c8 ; 278b05818 2835ff510) * DM-9953 * DM-13110 * DM-13182 * DM-12740 * DM-12373 * DM-12995 * DM-13189 * DM-13996***   skymap: ff merge * DM-12701   obs_base: ff merge * DM-12375   pipe_base: reset * DM-12450 * DM-12531   meas_base: ff merge * DM-13380   meas_algorithms: ff merge * DM-13055 * DM-12995   meas_astrom: ff merge * DM-12430 * DM-12431   ip_isr: ff merge * DM-12375 * DM-13260 * DM-13408   ip_diffim: ff merge * DM-12664 * DM-13325   pipe_tasks: reset * DM-13110 * DM-13244 * DM-12995 * DM-12184 * DM-13345 * DM-13380 * DM-12692 * DM-13396***   pipe_drivers: reset * DM-13184 * DM-13244 * DM-13227 * DM-13260 * DM-13396***   obs_subaru: reset * DM-13146 * DM-12375 * DM-13260 * DM-13388   obs_cfht obs_decam obs_lsstSim   meas_mosaic: ff merge * DM-13062     Well, we can basically do a "reset lsst/master" on all of the repos.     for pkg in $(cat packages.txt dependencies.txt) ; do if [[ ! -e $HOME/hsc/$pkg ]] ; then git clone git@github.com:HyperSuprime-Cam/$pkg.git ; git --git-dir $HOME/hsc/$pkg/.git remote add lsst git@github.com:LSST/$pkg.git ; fi ; git --git-dir $HOME/hsc/$pkg/.git fetch --all ; git --git-dir $HOME/hsc/$pkg/.git reset --hard lsst/master ; done   for pkg in $(cat packages.txt dependencies.txt) ; do git --git-dir $HOME/hsc/$pkg/.git push origin master --tags --force ; done   for pkg in $(cat packages.txt) ; do git --git-dir $HOME/hsc/$pkg/.git tag 6.0b5-hsc -m "HSC 6.0-beta5 release, 2018-01-30" ; git --git-dir $HOME/hsc/$pkg/.git push origin 6.0b5-hsc ; done   price@pap-laptop:~/hsc/starlink_ast (lsst-dev *%<>) $ git reset --hard lsst/lsst-dev price@pap-laptop:~/hsc/starlink_ast (lsst-dev=) $ git tag 6.0b5-hsc -m "HSC 6.0-beta5 release, 2018-01-30" --force price@pap-laptop:~/hsc/starlink_ast (lsst-dev=) $ git push origin 6.0b5-hsc --force   pprice@tiger-sumire:~ $ unset EUPS_DIR EUPS_SHELL EUPS_PATH EUPS_PKGROOT SETUP_EUPS pprice@tiger-sumire:~ $ . /tigress/pprice/hsc-release6/lsstsw/bin/setup.sh pprice@tiger-sumire:~ $ module load rh/devtoolset/6 pprice@tiger-sumire:~ $ rebuild hscPipe   pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ setup hscPipe 6.0-beta0+20 pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ setup -jr . pprice@tiger-sumire:/tigress/pprice/hsc-release6/lsstsw/build/hscPipe (master %=) $ scons install declare version=6.0-beta5   pprice@tiger-sumire:~ $ eups list --depend --raw hscPipe 6.0-beta5 | while IFS=\| read product version ; do eups declare $product $version -t current ; done pprice@tiger-sumire:~ $ eups distrib create --server-dir=/tigress/pprice/hsc-release6/hsc-server -S REPOSITORY_PATH='git://github.com/HyperSuprime-Cam/$PRODUCT.git' -f generic -d eupspkg hscPipe 6.0-beta5 pprice@tiger-sumire:~ $ eups distrib declare -t current --server-dir /tigress/pprice/hsc-release6/hsc-server pprice@tiger-sumire:~ $ rsync -rvz /tigress/pprice/hsc-release6/hsc-server/ master-noproxy:public_html/packages-hscPipe6     -----   pprice@tiger-sumire:~ $ unsetup eups pprice@tiger-sumire:~ $ . /tigress/HSC/stack/hsc3-tiger-20171206/loadLSST.bash pprice@tiger-sumire:~ $ module load rh/devtoolset/6 pprice@tiger-sumire:~ $ eups distrib install hscPipe 6.0-beta5   pprice@perseus:~ $ unsetup eups pprice@perseus:~ $ . /tigress/HSC/stack/hsc3-perseus-20171206/loadLSST.bash pprice@perseus:~ $ module load rh/devtoolset/6 pprice@perseus:~ $ eups distrib install hscPipe 6.0-beta5

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                Hisanori Furusawa, Hsin-Fang Chiang, Jim Bosch, Masayuki Tanaka, Paul Price, Sogo Mineo, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel