Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-285

Revise front-end for warping

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      The warping implementation (currently MakeCoaddTempExpTask) relies on a SelectImagesTask to sort out the inputs. Unfortunately, this selection is performed inside the MakeCoaddTempExp.run method, which means MakeCoaddTempExpTask gets parallelised over patches rather than over individual warps (unless the user has some additional mechanics for setting the command-lines, essentially doing the selection themselves), which is inefficient. The SelectImagesTask hierarchy grew up as a MySQL database query before other things got tacked on to support butler-based selection, and could use simplification.

      I propose to move the selection of images into the TaskRunner.getTargetList method, which will allow for parallelisation over individual warps. I will also refactor the SelectImagesTask hierarchy and add a sub-class to perform selection of images using the spatial database introduced in DM-3472.

      There is a SelectSdssImagesTask in obs_sdss that I propose to drop because it relies on a particular database that I don't have access to and cannot recreate, and that code probably hasn't been used in a while. If someone objects to dropping it, I will attempt to modify it to conform with the SelectImagesTask changes, but testing and fixing it will have to be the responsibility of whoever wants to keep it.

      While I'm modifying the warping front-end, I also propose to move the config parameters that are operational in nature (e.g., doWrite, doOverwrite) out of the Config. Their presence in the Config causes trouble because there are legitimate reasons for changing their values between command-line incantations, but doing so requires adding --clobber-config which should only be required for changes of an algorithmic nature.

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Without reading the comment history too carefully, I can probably do this on DM-15236 which is the ticket to "refactor makeCoaddTempExp in prep for Gen 3 middleware."

            Show
            yusra Yusra AlSayyad added a comment - - edited Without reading the comment history too carefully, I can probably do this on DM-15236 which is the ticket to "refactor makeCoaddTempExp in prep for Gen 3 middleware."
            Hide
            tjenness Tim Jenness added a comment -

            Ok, I've attached that ticket.

            Show
            tjenness Tim Jenness added a comment - Ok, I've attached that ticket.
            Hide
            yusra Yusra AlSayyad added a comment -

            I just marked DM-15236 as invalid because the the work was actually done on DM-17028https://github.com/lsst/pipe_tasks/commit/62e839c2e8686beb4b9883a710c8e7fb8af9ab5b

            Jim Bosch's original comment, "I expect SuperTask will help with this in the future," has been realized.  There's still some work to be done this cycle to retire the Gen 2 version.  I'll attach the appropriate ticket when we get there. 

             

            Show
            yusra Yusra AlSayyad added a comment - I just marked DM-15236 as invalid because the the work was actually done on DM-17028 :  https://github.com/lsst/pipe_tasks/commit/62e839c2e8686beb4b9883a710c8e7fb8af9ab5b Jim Bosch 's original comment, "I expect SuperTask will help with this in the future," has been realized.  There's still some work to be done this cycle to retire the Gen 2 version.  I'll attach the appropriate ticket when we get there.   
            Hide
            tjenness Tim Jenness added a comment -

            Yusra AlSayyad RFCs that have no work to be done keep popping up at the DMCCB for discussion. If you need to do more work on this RFC implementation would it be possible to create a ticket for it (it doesn't have to cover all the work, just some work to allow the RFC to keep on hiding).

            Show
            tjenness Tim Jenness added a comment - Yusra AlSayyad RFCs that have no work to be done keep popping up at the DMCCB for discussion. If you need to do more work on this RFC implementation would it be possible to create a ticket for it (it doesn't have to cover all the work, just some work to allow the RFC to keep on hiding).
            Hide
            yusra Yusra AlSayyad added a comment -

            I'm comfortable calling this implemented now. I expect deprecation of the Gen 2 version this Fall. 

            Show
            yusra Yusra AlSayyad added a comment - I'm comfortable calling this implemented now. I expect deprecation of the Gen 2 version this Fall. 

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Watchers:
              Andy Salnikov, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Kian-Tat Lim, Paul Price, Russell Owen, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.