Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10129

Process HSC RC dataset using Stack version chosen for the full S17B reprocessing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Process the RC dataset using the stack version that the full S17B HSC reprocessing will be based on. Presumably this version will be effectively the same or comparable to the version HSC chooses for their internal data processing.

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton added a comment -

            The output repo looks fine to me; in that I was able to successfully produce the full set of QA plots (DM-10044). Bob Armstrong will look at those to make sure there's nothing catastrophically wrong, but I don't expect there to be, and from our discussion at the group meeting yesterday, the consensus is that you can go ahead with the processing with your current set of configs, as the goal of this exercise is the mechanics of the large-scale exercise more than producing science-quality data results (in fact, there do seem to be some residual bugs in meas_mosaic). Thus, I will go ahead and mark this reviewed so you can get started, bugs and all.

            Show
            tmorton Tim Morton added a comment - The output repo looks fine to me; in that I was able to successfully produce the full set of QA plots ( DM-10044 ). Bob Armstrong will look at those to make sure there's nothing catastrophically wrong, but I don't expect there to be, and from our discussion at the group meeting yesterday, the consensus is that you can go ahead with the processing with your current set of configs, as the goal of this exercise is the mechanics of the large-scale exercise more than producing science-quality data results (in fact, there do seem to be some residual bugs in meas_mosaic). Thus, I will go ahead and mark this reviewed so you can get started, bugs and all.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for the review.

            If meas_mosaic has known important bugs, should I not run mosaic? Or is it worse not running mosaic?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for the review. If meas_mosaic has known important bugs, should I not run mosaic? Or is it worse not running mosaic?
            Hide
            tmorton Tim Morton added a comment -

            I can't give a good answer for this... As far as I can tell from the discussion on #subaru-hsc on slack, people aren't yet sure whether the problems are in mosaic or not. If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

            Show
            tmorton Tim Morton added a comment - I can't give a good answer for this... As far as I can tell from the discussion on #subaru-hsc on slack, people aren't yet sure whether the problems are in mosaic or not. If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?
            Hide
            swinbank John Swinbank added a comment -

            If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

            Tim guesses correctly.

            While we know there are some outstanding bugs that will affect the quality of the final results, my hope & expectation is that this is just the first of many such large-scale runs. As such, the primary aim here is to demonstrate that as much of the machinery is working as possible and to understand the issues we'll face running at scale on LSST hardware. Waiting until all the bugs are fixed will delay us hitting that goal — but after we've shown we can do this once, doing it N more times with progressively less buggy stacks should be "easy".

            Show
            swinbank John Swinbank added a comment - If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word? Tim guesses correctly. While we know there are some outstanding bugs that will affect the quality of the final results, my hope & expectation is that this is just the first of many such large-scale runs. As such, the primary aim here is to demonstrate that as much of the machinery is working as possible and to understand the issues we'll face running at scale on LSST hardware. Waiting until all the bugs are fixed will delay us hitting that goal — but after we've shown we can do this once, doing it N more times with progressively less buggy stacks should be "easy".
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'll close this ticket and continue the final tweaks in DM-9800. The repo of processed data will stay in /project until I'm ask to remove it (likely when we are short of space).
            Butler repo path: /datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10129

            Logs are stored in /project/hsc_rc/DM-10129/logs/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'll close this ticket and continue the final tweaks in DM-9800 . The repo of processed data will stay in /project until I'm ask to remove it (likely when we are short of space). Butler repo path: /datasets/hsc/repo/rerun/private/hchiang2/RC/ DM-10129 Logs are stored in /project/hsc_rc/ DM-10129 /logs/

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Tim Morton
                Watchers:
                Hsin-Fang Chiang, John Swinbank, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: