Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14616

Run colorAnalysis for the S18 PDR1 outputs

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Team:
      Data Facility

      Description

      Run colorAnalysis.py script on all coadds from the S18 PDR1 outputs (DM-13666).

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            While DM-13519 is in progress, use the tickets/DM-13519 branch (obs_subaru and pipe_analysis). It currently needs sims_catUtils and its dependencies.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - While DM-13519 is in progress, use the tickets/ DM-13519 branch (obs_subaru and pipe_analysis). It currently needs sims_catUtils and its dependencies.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I've used Lauren's installation of sims_catUtils (-t sims), the tickets/DM-13519 branches (baf46a9 and 7a430b3), and lsst_distrib at w_2018_21.

            In an earlier attempt I got 8 wide tracts failed with AssertionError on assert all(len(cat) == num for cat in catalogs.values()). I reran them and wasn't able to reproduce; all other attempts were successful and gave identical statistics numbers. The failed ones had smaller "num" in this "statistics" log message e.g. in

            colorAnalysis INFO: Statistics from DataId(initialdata={'tract': 8525, 'filter'
            : 'HSC-I', 'patch': '0,0'}, tag=set()) of Galactic Extinction:  A_HSC-Y (mag): {'
            all': Stats(mean=0.0287; stdev=0.0025; num=1357959; total=1414473; median=0.0285;
             clip=0.0090; forcedMean=None)}
            

            The guess is somehow the deepCoadd_forced_src catalogs for all patches or bands didn’t get read in properly. That could have happened if the multiband processing hadn't finished, but that's not the case here. We don't know what happened but haven't seen the error again.. New attempts completed fine.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I've used Lauren's installation of sims_catUtils ( -t sims ), the tickets/ DM-13519 branches (baf46a9 and 7a430b3), and lsst_distrib at w_2018_21 . In an earlier attempt I got 8 wide tracts failed with AssertionError on assert all(len(cat) == num for cat in catalogs.values()) . I reran them and wasn't able to reproduce; all other attempts were successful and gave identical statistics numbers. The failed ones had smaller "num" in this "statistics" log message e.g. in colorAnalysis INFO: Statistics from DataId(initialdata={'tract': 8525, 'filter' : 'HSC-I', 'patch': '0,0'}, tag=set()) of Galactic Extinction: A_HSC-Y (mag): {' all': Stats(mean=0.0287; stdev=0.0025; num=1357959; total=1414473; median=0.0285; clip=0.0090; forcedMean=None)} The guess is somehow the deepCoadd_forced_src catalogs for all patches or bands didn’t get read in properly. That could have happened if the multiband processing hadn't finished, but that's not the case here. We don't know what happened but haven't seen the error again.. New attempts completed fine.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In DEEP, tract=8282 and tract=8284 failed by

            TaskError: Flux filter from config not found (neither HSC-I nor the generic i
            

             
            (from here I guess)
            In either of the two tracts, there are no data in the HSC-I band (default of config.fluxFilter). Either tract has only 1 patch of data actually, so we probably don't care for now? Or, should the config fluxFilter or fluxFilterGeneric use a different filer?   I also noticed that colorAnalysis does not store config files like typical tasks; I assume that will be taken care of in the upcoming pipe_analysis clean-up. 

            Lauren MacArthur  The results of all wide/deep/udeep tracts are linked at https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In DEEP, tract=8282 and tract=8284 failed by TaskError: Flux filter from config not found (neither HSC-I nor the generic i   (from here I guess) In either of the two tracts, there are no data in the HSC-I band (default of config.fluxFilter). Either tract has only 1 patch of data actually, so we probably don't care for now? Or, should the config fluxFilter or fluxFilterGeneric use a different filer?   I also noticed that colorAnalysis does not store config files like typical tasks; I assume that will be taken care of in the upcoming pipe_analysis clean-up.  Lauren MacArthur  The results of all wide/deep/udeep tracts are linked at https://lsst-web.ncsa.illinois.edu/~hchiang2/lauren_DM-13519/
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks so much for doing this and providing feedback.  I think I will leave that error for now as we don't really have any deeply meaningful color analysis that doesn't involve i-band (and I imagine most science users will be ignoring those tracts until the I band gets filled in...probably already has been in the not-yet-public SSP data!)

             

            Thanks for the note about persisting configs.  I will add it to DM-14559.

            Show
            lauren Lauren MacArthur added a comment - Thanks so much for doing this and providing feedback.  I think I will leave that error for now as we don't really have any deeply meaningful color analysis that doesn't involve i-band (and I imagine most science users will be ignoring those tracts until the I band gets filled in...probably already has been in the not-yet-public SSP data!)   Thanks for the note about persisting configs.  I will add it to DM-14559 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm marking this as Won't Fix though I believe the main goal was meant with the previous attempt. Let me know if more needs to be done with these data from last year.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm marking this as Won't Fix though I believe the main goal was meant with the previous attempt. Let me know if more needs to be done with these data from last year.

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel