Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13436

Run pipe_analysis scripts with the RC2 data of w_2018_04

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Run

      1. visitAnalysis
      2. coaddAnalysis
      3. colorAnalysis
      4. compareVisitAnalysis
      5. compareCoaddAnalysis

      with the w_2018_04 RC2 outputs from DM-13256

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The pipe_analysis version is 10a03bd with the w_2018_04 stack. fastparquet has been pip-intalled.

            Following the scripts in /scratch/lauren/DM-11345/slurm/, some example commands are:

            visitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --tract=9615 --id visit=26024
             
            coaddAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=NB0921 -c doWriteParquetTables=True
             
            colorAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y^NB0921  -c doWriteParquetTables=True
             
            compareVisitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/UD_COSMOS_9813  --tract=9813 --id visit=25816
             
            compareCoaddAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/WIDE_GAMA15H_9615  --id tract=9615 filter=HSC-Z 
            
            

            As the HSC RC2 dataset was not included in w_2018_02 DM-13066, the comparison is against Lauren MacArthur's repos from w_2017_51.

            The visit ones need mosaic ouputs and can be executed per visit; the coadd ones need multiband outputs and can be executed per filter/tract/patch. colorAnalysis is per tract/patch.

            Also starting my own notes at https://confluence.lsstcorp.org/display/~hchiang2/Notes+on+pipe_analysis

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The pipe_analysis version is 10a03bd with the w_2018_04 stack. fastparquet has been pip-intalled. Following the scripts in /scratch/lauren/ DM-11345 /slurm/ , some example commands are: visitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --tract=9615 --id visit=26024   coaddAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=NB0921 -c doWriteParquetTables=True   colorAnalysis.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256 --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y^NB0921 -c doWriteParquetTables=True   compareVisitAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/UD_COSMOS_9813 --tract=9813 --id visit=25816   compareCoaddAnalysis.py /datasets/hsc/repo --rerun RC/w_2018_04/DM-13256 --rerun2 private/lauren/DM-11345/w51/WIDE_GAMA15H_9615 --id tract=9615 filter=HSC-Z As the HSC RC2 dataset was not included in w_2018_02 DM-13066 , the comparison is against Lauren MacArthur 's repos from w_2017_51. The visit ones need mosaic ouputs and can be executed per visit; the coadd ones need multiband outputs and can be executed per filter/tract/patch. colorAnalysis is per tract/patch. Also starting my own notes at https://confluence.lsstcorp.org/display/~hchiang2/Notes+on+pipe_analysis
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The outputs are added to /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/
            and logs are at /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs

            visitAnalysis was run for each visit; 432 logs for 432 visits are at /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs/visitAnalysis/. I utilized slurm --multi-prog option to execute multiple visits on each worker node. The visit IDs are not in the log filenames because I didn't find a way (yet) to directly name the logs with the visit IDs.

            coaddAnalysis and compareCoaddAnalysis were run one job for each tract+filter. So in total 16 jobs/logs for coaddAnalysis and 16 for compareCoaddAnalysis.

            colorAnalysis was run for each tract; 3 logs available.

            compareVisitAnalysis was done in a mixed way; some were grouped but some were done per visit. Logs are copied to /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/qaLogs/compareVisitAnalysis/. Sorry for the nonuniform logs; this will be improved next time DM-13523.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The outputs are added to /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 / and logs are at /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 /qaLogs visitAnalysis was run for each visit; 432 logs for 432 visits are at /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 /qaLogs/visitAnalysis/ . I utilized slurm --multi-prog option to execute multiple visits on each worker node. The visit IDs are not in the log filenames because I didn't find a way (yet) to directly name the logs with the visit IDs. coaddAnalysis and compareCoaddAnalysis were run one job for each tract+filter. So in total 16 jobs/logs for coaddAnalysis and 16 for compareCoaddAnalysis. colorAnalysis was run for each tract; 3 logs available. compareVisitAnalysis was done in a mixed way; some were grouped but some were done per visit. Logs are copied to /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 /qaLogs/compareVisitAnalysis/ . Sorry for the nonuniform logs; this will be improved next time DM-13523 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For compareCoaddAnalysis.py at the COSMOS tract, I had to specify the available patches in the command line. Otherwise, process failed at not being able to find non-existent patches. For example

            Traceback (most recent call last):
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+33/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__
                result = task.run(dataRef, **kwargs)
              File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 1059, in run
                repoInfo1 = getRepoInfo(patchRefList1[0], coaddName=self.config.coaddName, coaddDataset=dataset)
              File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/utils.py", line 950, in getRepoInfo
                metadata = butler.get(metaStr, dataId)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+8/python/lsst/daf/persistence/butler.py", line 1410, in get
                raise NoResults("No locations for get:", datasetType, dataId)
            lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:deepCoadd_forced_src dataId:DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '0,0'}, tag=set())
            srun: error: lsst-verify-worker29: task 0: Exited with exit code 1
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For  compareCoaddAnalysis.py at the COSMOS tract, I had to specify the available patches in the command line. Otherwise, process failed at not being able to find non-existent patches. For example Traceback (most recent call last): File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+33/python/lsst/pipe/base/cmdLineTask.py", line 408, in __call__ result = task.run(dataRef, **kwargs) File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/coaddAnalysis.py", line 1059, in run repoInfo1 = getRepoInfo(patchRefList1[0], coaddName=self.config.coaddName, coaddDataset=dataset) File "/home/hchiang2/stack/pipe_analysis/python/lsst/pipe/analysis/utils.py", line 950, in getRepoInfo metadata = butler.get(metaStr, dataId) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+8/python/lsst/daf/persistence/butler.py", line 1410, in get raise NoResults("No locations for get:", datasetType, dataId) lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:deepCoadd_forced_src dataId:DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '0,0'}, tag=set()) srun: error: lsst-verify-worker29: task 0: Exited with exit code 1
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks for noting this.  I’ll try to fix that soon.

            Show
            lauren Lauren MacArthur added a comment - Thanks for noting this.  I’ll try to fix that soon.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            All runs have finished and logs have been copied.

            Many calculateSysError were seen in the logs of compareVisitAnalysis and compareCoaddAnalysis. In compareVisitAnalysis logs, there are many warnings of

            Warning: sysErr calculation failed: The iteration is not making good progress, as measured by the
              improvement from the last ten iterations.
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - All runs have finished and logs have been copied. Many calculateSysError were seen in the logs of compareVisitAnalysis and compareCoaddAnalysis . In compareVisitAnalysis logs, there are many warnings of Warning: sysErr calculation failed: The iteration is not making good progress, as measured by the improvement from the last ten iterations.
            Hide
            lauren Lauren MacArthur added a comment -

            Those warnings are actually indicating close-to-perfect agreement in the matched source measurement being fit (no progress on the iterative fitting can be made on zero variance).  I will think about trying to get rid of them in this case (but I think they are coming from scipy...) 

            Show
            lauren Lauren MacArthur added a comment - Those warnings are actually indicating close-to-perfect agreement in the matched source measurement being fit (no progress on the iterative fitting can be made on zero variance).  I will think about trying to get rid of them in this case (but I think they are coming from scipy...) 

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.