Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2674

Get meas_mosaic working on HSC data with LSST stack

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_mosaic
    • Labels:
      None
    • Story Points:
      10
    • Sprint:
      Science Pipelines DM-W16-6, DRP X16-2, DRP X16-3, DRP F16-1, DRP F16-2
    • Team:
      Data Release Production

      Description

      We have an old, bitrotted version of meas_mosaic on the LSST side, created in a failed attempt to get it running on LSST PhoSim data.

      Now that we're making a serious effort to get HSC data running through the LSST pipeline, we'll need to get it running with the LSST pipeline at least on HSC data, which will probably involve just merging everything from the HSC side over, and then fixing it until it builds and runs.

      For this issue, we'll assume that we're going to use the Eigen backend for the matrix solver, rather than the MKL one we use on the HSC side. That will make it much slower (since MKL is multithreaded and we can't make Eigen multithreaded for just meas_mosaic), but hopefully still usable.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            John, would you be able to give this a look? The meas_mosaic work is on the tickets/DM-2674 branch and you'll need the u/lauren/DM-2674 branch on obs_subaru to run mosaic.py.

            Show
            lauren Lauren MacArthur added a comment - - edited John, would you be able to give this a look? The meas_mosaic work is on the tickets/ DM-2674 branch and you'll need the u/lauren/ DM-2674 branch on obs_subaru to run mosaic.py .
            Hide
            swinbank John Swinbank added a comment -

            At the sprint planning meeting of 2016-06-29 we (incl Robert Lupton, Bob Armstrong, Jim Bosch, Pim Schellart [X]) eyeballed the figures above. The consensus was that the quality of these results is adequate to regard this ticket as done, but we'll carefully examine the results from meas_mosaic in the upcoming RC dataset processing (DM-6817, DM-6818).

            Remaining review here (to follow) will focus on the code rather than the output.

            Show
            swinbank John Swinbank added a comment - At the sprint planning meeting of 2016-06-29 we (incl Robert Lupton , Bob Armstrong , Jim Bosch , Pim Schellart [X] ) eyeballed the figures above. The consensus was that the quality of these results is adequate to regard this ticket as done, but we'll carefully examine the results from meas_mosaic in the upcoming RC dataset processing ( DM-6817 , DM-6818 ). Remaining review here (to follow) will focus on the code rather than the output.
            Hide
            swinbank John Swinbank added a comment -

            Looks good. I've pushed a number of minor cleanups to the code and commit structure to u/swinbank/DM-2674a. Please check if you are happy with them and, if so, adopt them.

            Here are a couple of minor comments:

            • You check in a few places for "HSC stack outputs". While I know what this means, to future generations there will be no distinction between the "HSC stack" and the "LSST stack". Can you add another couple of words of commentary to explain what this means and why it's important?
            • Similarly, I'd like a little bit more on srcSchemaMap. Can you provide an example of using it? Is it really necessary except for as a debugging aid during porting? (Being useful as a debugging aid doesn't mean we should get rid of it, but we should probably flag it as such.)
            • Some of the cleanups are a little gratuitous (does it really help to turn all the 's into "s?!), but since they're done now I don't propose to revert them.
            • I have not carefully checked all of the code changes: clean code is not the priority here and proof of pudding is in eating, so I'm more interested in good results. I have eyeballed most of it (the main exception being all the stuff that got copied to utils.py) and I did not see anything I was very unhappy about.
            • Have you compared the performance of this code vs. the HSC version? Yasuda-san was worried that the changes to catalogue loading would cause issues. Perhaps we won't see that until we try running it at scale, though.
            • I've edited the table file to drop the setupOptional of MKL and OpenBLAS. setupOptional is treated as a requirement by Jenkins and EUPS distrib, and we don't want dependencies on these products.
            • This now builds on Jenkins.
            Show
            swinbank John Swinbank added a comment - Looks good. I've pushed a number of minor cleanups to the code and commit structure to u/swinbank/DM-2674a . Please check if you are happy with them and, if so, adopt them. Here are a couple of minor comments: You check in a few places for "HSC stack outputs". While I know what this means, to future generations there will be no distinction between the "HSC stack" and the "LSST stack". Can you add another couple of words of commentary to explain what this means and why it's important? Similarly, I'd like a little bit more on srcSchemaMap . Can you provide an example of using it? Is it really necessary except for as a debugging aid during porting? (Being useful as a debugging aid doesn't mean we should get rid of it, but we should probably flag it as such.) Some of the cleanups are a little gratuitous (does it really help to turn all the 's into "s?!), but since they're done now I don't propose to revert them. I have not carefully checked all of the code changes: clean code is not the priority here and proof of pudding is in eating, so I'm more interested in good results. I have eyeballed most of it (the main exception being all the stuff that got copied to utils.py ) and I did not see anything I was very unhappy about. Have you compared the performance of this code vs. the HSC version? Yasuda-san was worried that the changes to catalogue loading would cause issues. Perhaps we won't see that until we try running it at scale, though. I've edited the table file to drop the setupOptional of MKL and OpenBLAS. setupOptional is treated as a requirement by Jenkins and EUPS distrib, and we don't want dependencies on these products. This now builds on Jenkins .
            Hide
            swinbank John Swinbank added a comment -

            Now also looked at obs_subaru: sorry for missing that previously. Minor comments on the pull request. I guess my most significant concern is whether we could renormalize config/Mosaic.py, config/hsc/Mosaic.py, config/filterMap.py, config/hsc/filterMap.py, etc – it seems like there's a lot of overrides in here that are probably confusingly redundant. Maybe spent a few minutes checking?

            Other than that, good to go.

            Show
            swinbank John Swinbank added a comment - Now also looked at obs_subaru: sorry for missing that previously. Minor comments on the pull request . I guess my most significant concern is whether we could renormalize config/Mosaic.py , config/hsc/Mosaic.py , config/filterMap.py , config/hsc/filterMap.py , etc – it seems like there's a lot of overrides in here that are probably confusingly redundant. Maybe spent a few minutes checking? Other than that, good to go.
            Hide
            lauren Lauren MacArthur added a comment -

            I've addressed the obs_subaru comments. In meas_mosaic I've added comments in commit messages and in the checkHscStack() function in utils.py to emphasize what is meant by the "HSC stack" and that it will soon become obsolete. I've also pointed to the code that demonstrates the use of the srcSchemaMap.

            I have not done any performance testing of this code vs. the HSC version. We can keep that in mind in the context of DM-6816.

            A final Jenkins run was successful.

            Merged to master.

            Show
            lauren Lauren MacArthur added a comment - I've addressed the obs_subaru comments. In meas_mosaic I've added comments in commit messages and in the checkHscStack() function in utils.py to emphasize what is meant by the "HSC stack" and that it will soon become obsolete. I've also pointed to the code that demonstrates the use of the srcSchemaMap . I have not done any performance testing of this code vs. the HSC version. We can keep that in mind in the context of DM-6816 . A final Jenkins run was successful. Merged to master.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                John Swinbank
                Watchers:
                Jim Bosch, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel