Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11345

Redefine HSC "RC" dataset for bi-weeklies processing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP F17-5, DRP F17-6, DRP S18-1, DRP S18-2
    • Team:
      Data Release Production

      Description

      We are currently processing what is known as the HSC RC dataset, where RC ==> Release Candidate, on a bi-weekly basis (for all even numbered weeklies). See https://confluence.lsstcorp.org/display/DM/Reprocessing+of+the+HSC+RC+dataset for full details. The purpose of this processing is to have a fully processed dataset as the pipeline evolves and be able track any quality/performance changes (for better or for worse - to properly praise/bless the former and catch & fix/revert the latter). Clearly, the dataset involved should be representative of the widest range of data quality/observing conditions expected, and is currently available to us through public HSC data. The RC dataset we have been using is not satisfactory in this respect. This issue is to remedy this situation by redefining (and perhaps renaming? RC ==> CI ??) the set of HSC tracts/visits to include in our bi-weeklies processing.

      A list of known data "characteristics" that this dataset should cover include tracts that suffer from the known issues to date, including:

      • a very well-behaved tract
      • a tract that includes visits that have proven challenging for astrometric fitting (see DM-11090)
      • a tract that includes visits that have proven challenging for photometric fitting (see DM-11927)
      • a tract including "too-good-seeing" visits (see DM-9855)
      • a tract including a wide range in seeing conditions (to test PSF-matching, see DM-8289)
      • tracts covering a range of "challenges" for deblending (possibly those identified in DM-9168)
      • a tract at the survey's edge (to check for issues arising due to patchy coverage - verify if 8766 WIDE still fills this bill)
      • a tract including galaxies suitable for assessing total magnitudes for "big" galaxies (list of examples along with "best" total magnitude estimates to be compared against being compiled by HSC member Song Huang)
      • a tract with observations in both the old i and new i2 HSC filters (see RFC-390)
      • PLEASE ADD YOUR FAVORITE PATHOLOGY HERE

      We will have to keep this dataset to a reasonable size (perhaps Hsin-Fang Chiang could comment on what "reasonable" means to her in numbers?) While we will certainly want to add pathological areas as we find/gain access to them (e.g. crowded fields), the set we define and agree upon here will be considered the canonical one until a need to expand/redefine is identified.

        Attachments

        1. PsfDist_UD_COSMOS_9813.png
          PsfDist_UD_COSMOS_9813.png
          21 kB
        2. PsfDist_WIDE_GAMMA15H_9615.png
          PsfDist_WIDE_GAMMA15H_9615.png
          19 kB
        3. PsfDist_WIDE_VVDS_9697.png
          PsfDist_WIDE_VVDS_9697.png
          18 kB
        4. tracts_patches_UD_COSMOS_HSC-I.png
          tracts_patches_UD_COSMOS_HSC-I.png
          97 kB
        5. visits_UD_COSMOS_9813_I.png
          visits_UD_COSMOS_9813_I.png
          270 kB
        6. visits_W_VVDS_9697_I.png
          visits_W_VVDS_9697_I.png
          317 kB
        7. visits_WIDE_GAMA15H_9615_I.png
          visits_WIDE_GAMA15H_9615_I.png
          286 kB
        8. W-GAMA15H_i_area.png
          W-GAMA15H_i_area.png
          53 kB
        9. WIDE_GAMA15H_9615_1,5_1,6-gri.png
          WIDE_GAMA15H_9615_1,5_1,6-gri.png
          2.96 MB
        10. W-VVDS_i_area.png
          W-VVDS_i_area.png
          51 kB

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Reductions from the w_2017_51 weekly live in /datasets/hsc/repo/rerun/private/lauren/DM-11345/w51/ on lsat-dev

            The coadd and color QA scripts have been run and parquet tables created, so those can be perused (most easily with the tools of Tim Morton). The visit level scripts are running now.

            Show
            lauren Lauren MacArthur added a comment - Reductions from the w_2017_51 weekly live in /datasets/hsc/repo/rerun/private/lauren/ DM-11345 /w51/ on lsat-dev The coadd and color QA scripts have been run and parquet tables created, so those can be perused (most easily with the tools of Tim Morton ). The visit level scripts are running now.
            Hide
            lauren Lauren MacArthur added a comment -

            I've attached the survey footprints (taken from https://hsc-release.mtk.nao.ac.jp/doc/index.php/data/).  Note that WIDE_GAMA15H_9615 satisfies our "survey's edge" criterion.

             

            Show
            lauren Lauren MacArthur added a comment - I've attached the survey footprints (taken from https://hsc-release.mtk.nao.ac.jp/doc/index.php/data/ ).  Note that WIDE_GAMA15H_9615 satisfies our "survey's edge" criterion.  
            Hide
            lauren Lauren MacArthur added a comment -

            I have included as much info about the proposed "RC2" dataset to ensure all our criteria (delineated above) are met. Jim, would you mind having a look to see if you're satisfied or would like to see any more characteristics (or clarifications of what I've presented) before signing off. I image Robert Lupton, Paul Price, and Bob Armstrong might want to weigh in as well.

            Also, I've had a discussion with Hsin-Fang Chiang about changing from the current RC dataset she is running as the bi-weekly dataset to the one defined here (as well as adding the running of the QA scripts to her workflow). If there are no major concerns or issues, this switch-over could happen as early as the w_2018_06 release bi-weekly run.

            Show
            lauren Lauren MacArthur added a comment - I have included as much info about the proposed "RC2" dataset to ensure all our criteria (delineated above) are met. Jim, would you mind having a look to see if you're satisfied or would like to see any more characteristics (or clarifications of what I've presented) before signing off. I image Robert Lupton , Paul Price , and Bob Armstrong might want to weigh in as well. Also, I've had a discussion with Hsin-Fang Chiang about changing from the current RC dataset she is running as the bi-weekly dataset to the one defined here (as well as adding the running of the QA scripts to her workflow). If there are no major concerns or issues, this switch-over could happen as early as the w_2018_06 release bi-weekly run.
            Hide
            jbosch Jim Bosch added a comment -

            Looks great; you've certainly done due diligence on these!

            I think the two remaining questions (which should not block closing this ticket) are:

            • What do we call the new dataset?
            • When do we start using it.

            I'll (re-)start those discussions on #dm-hsc-reprocessing in Slack to see if we can get some resolution on those issues.

            Show
            jbosch Jim Bosch added a comment - Looks great; you've certainly done due diligence on these! I think the two remaining questions (which should not block closing this ticket) are: What do we call the new dataset? When do we start using it. I'll (re-)start those discussions on #dm-hsc-reprocessing in Slack to see if we can get some resolution on those issues.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks, Jim!

            Show
            lauren Lauren MacArthur added a comment - Thanks, Jim!

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Jim Bosch
                Watchers:
                Bob Armstrong, Dino Bektesevic, Hsin-Fang Chiang, Ian Sullivan, Jeffrey Carlin, Jim Bosch, John Swinbank, Lauren MacArthur, Paul Price, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel