Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2915

Port safe coadd clipping from HSC

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw, meas_base, pipe_tasks
    • Labels:
      None

      Description

      We have an algorithm on the HSC fork that modifies AssembleCoaddTask to clip outliers in a much safer way, based on detecting contiguous regions in the difference between a non-clipped coadd and an aggressively-clipped coadd, and only rejecting pixels that are outliers in a single epoch.

      One complication for this code transfer is that some of the coadd code has been refactored on the HSC side, and there may be code in hscPipe that duplicates much of what's in pipe_tasks. We may need help from Paul Price to resolve those inconsistencies before tackling this issue.

      Once that's done, HSC code can be transferred from the following issues:

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Nate Lust The revised code on meas_base is better, thanks. There are still a couple of issues which I think need attention: you should eliminate the duplicate `typedefs` and take another crack at the comment in the `fail()` function.

            Beyond that, there are a lot of minor spacing and formatting issues. I feel almost bad about pulling you up on these trivial issues, but they do need fixing. I wonder if there are linters that could help you track down these issues in C++ code automatically – might be worth investigating. (There are certainly code formatters, like clang-format, but you want something that'll point out mistakes rather than attempt to reformat the whole file.)

            Show
            swinbank John Swinbank added a comment - Nate Lust The revised code on meas_base is better, thanks. There are still a couple of issues which I think need attention: you should eliminate the duplicate `typedefs` and take another crack at the comment in the `fail()` function. Beyond that, there are a lot of minor spacing and formatting issues. I feel almost bad about pulling you up on these trivial issues, but they do need fixing. I wonder if there are linters that could help you track down these issues in C++ code automatically – might be worth investigating. (There are certainly code formatters, like clang-format , but you want something that'll point out mistakes rather than attempt to reformat the whole file.)
            Hide
            swinbank John Swinbank added a comment -

            By the way, I'm assuming DM-4201 was a response to my request for a tech debt tickets covering this work? It's a start, but could you make it much more explicit about what actually needs doing, please.

            Show
            swinbank John Swinbank added a comment - By the way, I'm assuming DM-4201 was a response to my request for a tech debt tickets covering this work? It's a start, but could you make it much more explicit about what actually needs doing, please.
            Hide
            swinbank John Swinbank added a comment -

            Nate Lust Three tiny whitespace tweaks on your latest meas_base iteration – almost perfect. I suggest you just take care of them when squashing history. I am happy for you to merge, but equally willing to take a look at your cleaned up history first if you like.

            Show
            swinbank John Swinbank added a comment - Nate Lust Three tiny whitespace tweaks on your latest meas_base iteration – almost perfect. I suggest you just take care of them when squashing history. I am happy for you to merge, but equally willing to take a look at your cleaned up history first if you like.
            Hide
            nlust Nate Lust added a comment -

            This work has been completed and merged to master in: meas_base, pipe_tasks, and obs_subaru.

            Show
            nlust Nate Lust added a comment - This work has been completed and merged to master in: meas_base, pipe_tasks, and obs_subaru.
            Hide
            swinbank John Swinbank added a comment -

            Thanks Nate Lust. As a final step, could you please write a brief (no more than a couple of sentences) summary of the new functionality you have enabled in the LSST stack and add it to https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+W16+release+notes ? Thanks!

            Show
            swinbank John Swinbank added a comment - Thanks Nate Lust . As a final step, could you please write a brief (no more than a couple of sentences) summary of the new functionality you have enabled in the LSST stack and add it to https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+W16+release+notes ? Thanks!

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Bob Armstrong, Jim Bosch, John Swinbank
                Watchers:
                Bob Armstrong, Jim Bosch, John Swinbank, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: