Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14239

understand HSC data

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      5
    • Epic Link:
    • Sprint:
      SUIT Sprint 2018-05, SUIT Sprint 2018-06, SUIT Sprint 2018-07
    • Team:
      Science User Interface

      Description

      Get some HSC sample data to make sure that Firefly can read and display them properly.

      The reprocessed HSC data could be accessed from lsst-dev.ncsa.illinois.edu at 

      /datasets/hsc/repo/rerun/RC/

       

        Attachments

          Issue Links

            Activity

            Hide
            xiuqin Xiuqin Wu [X] (Inactive) added a comment -

            HSC RC reprocess status page

            Show
            xiuqin Xiuqin Wu [X] (Inactive) added a comment - HSC RC reprocess status page
            Hide
            cwang Cindy Wang [X] (Inactive) added a comment - - edited

            HSC data from 'Reprocessing of the HSC RC2 dataset', https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC2+dataset, at the repo path, /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/,  is explored. (data at any other repo path can be read and displayed using the following steps)

            In order to view HSC data from Firefly UI, please try the steps below,

            (the steps and the scripts used are made per instructions and utility file, filetable.py, which uploads table of files matching a pattern to a FireflyCleint,  for file browser application on lsst-dev prepared by David Shupe

            1. login lsst-dev:   ssh lsst-dev.ncsa.illinois.edu
            2. open browser window with tri-view:  https://lsst-demo.ncsa.illinois.edu/firefly/?__wsch=browse-files
            3. run the following scripts on your lsst-dev term: 

                        /home/cwangdev/hscData/enableDTSet6

                        source /home/cwangdev/hscData/hscSetup

                  4. run the script to start the file browser on lsst-dev  (FITS file list will be shown on Firefly UI)

                       /home/cwangdev/hscData/getHSCFiles  <HSC data location>  <fits filename pattern>  ; if no <fits file name pattern> is specified, all fits files at the relevant location are listed. 

                      ex.  /home/cwangdev/hscData/getHSCFiles /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/00814/HSC-Y/corr CORR-*-008

                      (files  with path like /datasets/hsc/repo/rerun/RC/w_2018_18/DM-14243/00814/HSC-Y/corr/CORR-*-008.fits are listed in 'Table-1' as shown in  firefly_browser.png) 

                  From the file browser, you can select and explore HSC FITS file by the following steps, 

            • from column 'path' of Table-1, copy one fits path 
            • go to 'Upload' page, select radio button 'Upload from URL', paste the copied fits path, and click button 'Upload' 
            • FITS file summary is listed and FITS image or table can be further selected and searched.    

            note: step 1-4 provides a way to browse the url path of HSC data at specific location with specific filename pattern. You may skip the steps and enter the url string into Firefly 'Upload' page manually. 

             

             

            Show
            cwang Cindy Wang [X] (Inactive) added a comment - - edited HSC data from 'Reprocessing of the HSC RC2 dataset', https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC2+dataset , at the repo path, /datasets/hsc/repo/rerun/RC/w_2018_18/ DM-14243 /,  is explored. (data at any other repo path can be read and displayed using the following steps) In order to view HSC data from Firefly UI, please try the steps below, (the steps and the scripts used are made per instructions and utility file, filetable.py, which uploads table of files matching a pattern to a FireflyCleint,  for file browser application on lsst-dev prepared by  David Shupe )  login lsst-dev:    ssh  lsst-dev.ncsa.illinois.edu open browser window with tri-view:   https://lsst-demo.ncsa.illinois.edu/firefly/?__wsch=browse-files run the following scripts on your lsst-dev term:              /home/cwangdev/hscData/enableDTSet6             source /home/cwangdev/hscData/hscSetup       4. run the script to start the file browser on lsst-dev  (FITS file list will be shown on Firefly UI)             /home/cwangdev/hscData/getHSCFiles   <HSC data location>    <fits filename pattern>   ; if no <fits file name pattern> is specified, all fits files at the relevant location are listed.            ex.  /home/cwangdev/hscData/getHSCFiles /datasets/hsc/repo/rerun/RC/w_2018_18/ DM-14243 /00814/HSC-Y/corr CORR-*-008           (files  with path like /datasets/hsc/repo/rerun/RC/w_2018_18/ DM-14243 /00814/HSC-Y/corr/CORR-*-008.fits are listed in 'Table-1' as shown in  firefly_browser.png)        From the file browser, you can select and explore HSC FITS file by the following steps,  from column 'path' of Table-1, copy one fits path  go to 'Upload' page, select radio button 'Upload from URL', paste the copied fits path, and click button 'Upload'  FITS file summary is listed and FITS image or table can be further selected and searched.     note: step 1-4 provides a way to browse the url path of HSC data at specific location with specific filename pattern. You may skip the steps and enter the url string into Firefly 'Upload' page manually.     
            Hide
            cwang Cindy Wang [X] (Inactive) added a comment - - edited

            The following HSC data (FITS data) are explored and readable by using Firefly, 

            /datasets/hsc/rerun/repo/RC/w_2018_18/DM-14243/<number for single epoch>/<filter>/corr/CORR*, BKGD, skyCorr*, calexp-*

                                                                                                                                                       /output/ICRS*, SRCMATCH, SRCMATCHFULL-, SRC-*

                                                                                               /deepCoadd/filter/track/patch/psfMatchedWarp-*, warp-*

                                                                                                                                    <patch>, <patch>_nImage

                                                                                               /deepCoadd-results/filter/tract/patch/celexp-*, det-, det-bkgd-, meas-, srcMatch-, forced_src-, srcMatchFull-

                                                                                                                              /merged/tract/patch/ref-*, mergeDet-*

                                                                                               /jointcal-results/tract/fcr-*

                                                                                                                     /filter/tract/jointcal-photoCalib-*, jointcal-wcs-

             

            Show
            cwang Cindy Wang [X] (Inactive) added a comment - - edited The following HSC data (FITS data) are explored and readable by using Firefly,  /datasets/hsc/rerun/repo/RC/w_2018_18/ DM-14243 /<number for single epoch>/<filter>/corr/CORR* , BKGD , skyCorr * , calexp- *                                                                                                                                            /output/ICRS* , SRCMATCH , SRCMATCHFULL- , SRC- *                                                                                    /deepCoadd/filter/track/patch/psfMatchedWarp-* , warp- *                                                                                                                         <patch>, <patch>_nImage                                                                                    /deepCoadd-results/filter/tract/patch/celexp-* , det- , det-bkgd- , meas- , srcMatch- , forced_src- , srcMatchFull-                                                                                                                   /merged/tract/patch/ref-* , mergeDet- *                                                                                    /jointcal-results/tract/fcr-*                                                                                                          /filter/tract/jointcal-photoCalib-* , jointcal-wcs-  
            Hide
            xiuqin Xiuqin Wu [X] (Inactive) added a comment -

            Sporadicly checked the files and they are all displayed properly in Firefly. I created a ticket to improve viewing a list of files (with mixed FITS images and tables). DM-15083

            Show
            xiuqin Xiuqin Wu [X] (Inactive) added a comment - Sporadicly checked the files and they are all displayed properly in Firefly. I created a ticket to improve viewing a list of files (with mixed FITS images and tables). DM-15083
            Hide
            xiuqin Xiuqin Wu [X] (Inactive) added a comment -

            done

            Show
            xiuqin Xiuqin Wu [X] (Inactive) added a comment - done

              People

              Assignee:
              cwang Cindy Wang [X] (Inactive)
              Reporter:
              xiuqin Xiuqin Wu [X] (Inactive)
              Reviewers:
              Xiuqin Wu [X] (Inactive)
              Watchers:
              Cindy Wang [X] (Inactive), Gregory Dubois-Felsmann, Tatiana Goldina, Xiuqin Wu [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.