Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17421

Define monthly ImSim DC2 reprocessing dataset subset

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP F19-5, DRP F19-6 (Nov), DRP S20-2 (Jan)
    • Team:
      Data Release Production

      Description

      Consulting with DESC and Hsin-Fang Chiang, choose a tract or two of ImSim DC2 data from the subset Hsin-Fang Chiang already copied to /datasets/ to reprocess as part of the new monthly DRP reprocessing, that meet DRP's QA needs.

      End product of this tickets is tract number(s) and visits listed in the comments.

        Attachments

          Activity

          Hide
          lauren Lauren MacArthur added a comment - - edited

          Quick update: Hsin-Fang Chiang is already processing DC2 on DM-17555.  Based on the following comments from James Chiang:

          For the initial Run1.2i processing, if you were to start off with one tract, I'd recommend using either 4849, 4850, 4639, or 4638. These are the four that are fully enclosed by the 5x5 sq degree protoDC2 simulation region. Of those, 4849 overlaps with the DDF so that one may be of the most general interest to people.

          we agreed that indeed 4849 would be the best tract to start with. That processing has already revealed some issues (see DM-17731 & DM-17737) which may need resolving first, but once that track has been satisfactorily processed, I will have a look and summarize any relevant characteristics/QA analysis.

          Show
          lauren Lauren MacArthur added a comment - - edited Quick update: Hsin-Fang Chiang is already processing DC2 on DM-17555 .  Based on the following comments from  James Chiang : For the initial Run1.2i processing, if you were to start off with one tract, I'd recommend using either 4849, 4850, 4639, or 4638. These are the four that are fully enclosed by the 5x5 sq degree protoDC2 simulation region. Of those, 4849 overlaps with the DDF so that one may be of the most general interest to people. we agreed that indeed 4849 would be the best tract to start with. That processing has already revealed some issues (see DM-17731 & DM-17737 ) which may need resolving first, but once that track has been satisfactorily processed, I will have a look and summarize any relevant characteristics/QA analysis.
          Hide
          yusra Yusra AlSayyad added a comment - - edited

          Lauren MacArthur, DM-17731 & DM-17737 should not hold up the QA analysis of those tracts. They indicate bad WCS's in ~1 visit. I'm going to look into what went wrong with the WCS in those visits, but the coadd's should still be fine, since those visits didn't make it through the warping stage..

          Show
          yusra Yusra AlSayyad added a comment - - edited Lauren MacArthur , DM-17731 & DM-17737 should not hold up the QA analysis of those tracts. They indicate bad WCS's in ~1 visit. I'm going to look into what went wrong with the WCS in those visits, but the coadd's should still be fine, since those visits didn't make it through the warping stage..
          Hide
          jbosch Jim Bosch added a comment -

          Michael Wood-Vasey has said that he'd be happy to help with this project; I'll let those already familiar with what's going on chime in on what shape that might take.

          Show
          jbosch Jim Bosch added a comment - Michael Wood-Vasey has said that he'd be happy to help with this project; I'll let those already familiar with what's going on chime in on what shape that might take.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Thanks, Jim Bosch. I just got up to speed, and of course Hsin-Fang Chiang has it all figured out already

          Tract 4849 is a totally reasonable and good choice. I agree with just picking one tract. I'll follow along on these tickets, but please ping me if you have questions.

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Thanks, Jim Bosch . I just got up to speed, and of course Hsin-Fang Chiang has it all figured out already Tract 4849 is a totally reasonable and good choice. I agree with just picking one tract. I'll follow along on these tickets, but please ping me if you have questions.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Oh, one specific thing. I have the DESC processings of these condensed into pseudo-DPDD Parquet files. Where should I put them on lsst-dev for convenience of comparison?

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Oh, one specific thing. I have the DESC processings of these condensed into pseudo-DPDD Parquet files. Where should I put them on lsst-dev for convenience of comparison?
          Hide
          lauren Lauren MacArthur added a comment -

          Great, thanks Michael Wood-Vasey...your input here will certainly be of great value!  As for the DESC processing output, my typical QA-ing with the pipe_analysis repo makes use of butler-ized outputs for direct comparisons of different reruns of the same dataset, so having a standard output repo would be ideal for me at this point if we want to run my comparison scripts of the new Hsin-Fang Chiang processing against any DESC processing (however, my first step will be to get the scripts to run on DESC outputs since there are still some lingering HSC-isms baked in, especially in the colorAnalysis.py script...)

          Show
          lauren Lauren MacArthur added a comment - Great, thanks Michael Wood-Vasey ...your input here will certainly be of great value!  As for the DESC processing output, my typical QA-ing with the pipe_analysis repo makes use of butler-ized outputs for direct comparisons of different reruns of the same dataset, so having a standard output repo would be ideal for me at this point if we want to run my comparison scripts of the new Hsin-Fang Chiang processing against any DESC processing (however, my first step will be to get the scripts to run on DESC outputs since there are still some lingering HSC-isms baked in, especially in the colorAnalysis.py script...)
          Hide
          lauren Lauren MacArthur added a comment -

          Marking this as invalid as it has been superseded by DM-22954, which defines two tracts of DC2 Run2.2i (y1+y2) to be included in validation runs.

          These data were processed on lsst_dev on DM-22816 and I have cerated DM-23250 to run the visitAnalysis.py and coaddAnalysis.py pipe_analysis scripts on the outputs for some basic QA & characterization.

          Show
          lauren Lauren MacArthur added a comment - Marking this as invalid as it has been superseded by DM-22954 , which defines two tracts of DC2 Run2.2i (y1+y2) to be included in validation runs. These data were processed on lsst_dev on DM-22816 and I have cerated DM-23250 to run the visitAnalysis.py and coaddAnalysis.py pipe_analysis scripts on the outputs for some basic QA & characterization.

            People

            Assignee:
            lauren Lauren MacArthur
            Reporter:
            yusra Yusra AlSayyad
            Watchers:
            Hsin-Fang Chiang, Jim Bosch, Lauren MacArthur, Michael Wood-Vasey, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.