Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7716

Fix dataset lookup in multibandDriver.py

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      0.5
    • Epic Link:
    • Sprint:
      DRP F16-4, DRP F16-5
    • Team:
      Data Release Production

      Description

      MultibandDriver builds a patch reference list by checking for the existence of self.config.coaddName + "Coadd". However, when coadd assembly is performed by running coaddDriver, this coadd is not persisted in favor of only persisting the updated calexp version. Thus, currently MultibandDriver ends up with and empty patch list and exits quietly. The existence check should be on the self.config.coaddName + "Coadd_calexp".

        Attachments

          Issue Links

            Activity

            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue blocks DM-6816 [ DM-6816 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            lauren Lauren MacArthur added a comment -

            Eli Rykoff, would you mind having a look at this? In particular, can you check if you get farther with the tutorial instructions with this fix. I ran into a subsequent issue with the deblender (which I will work on tomorrow), but perhaps you won't hit it with the data your running with as it was in an exception.

            Show
            lauren Lauren MacArthur added a comment - Eli Rykoff , would you mind having a look at this? In particular, can you check if you get farther with the tutorial instructions with this fix. I ran into a subsequent issue with the deblender (which I will work on tomorrow), but perhaps you won't hit it with the data your running with as it was in an exception.
            lauren Lauren MacArthur made changes -
            Reviewers Eli Rykoff [ erykoff ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            erykoff Eli Rykoff added a comment -

            multiBandDriver.py makes it further than before, but died on me with "AttributeError in mapNoBalance: 'HscMapper' object has no attribute 'map_deepCoadd_srcMatch'"

            Show
            erykoff Eli Rykoff added a comment - multiBandDriver.py makes it further than before, but died on me with "AttributeError in mapNoBalance: 'HscMapper' object has no attribute 'map_deepCoadd_srcMatch'"
            Hide
            jbosch Jim Bosch added a comment -

            I believe that error is something that recently broke and was fixed (in obs_subaru and daf_butlerUtils). Getting more recent versions of those ought to fix it.

            Show
            jbosch Jim Bosch added a comment - I believe that error is something that recently broke and was fixed (in obs_subaru and daf_butlerUtils). Getting more recent versions of those ought to fix it.
            Hide
            erykoff Eli Rykoff added a comment -

            Oh, okay. I had built this on Friday then had to run home to pick up a sick kid before it was done. This morning I didn't want to tell it to rebuild because of the time taken, but I guess that was a mistake? (Or did I also need to rebuild obs_subaru explicitly as well as pipe_drivers?)

            Show
            erykoff Eli Rykoff added a comment - Oh, okay. I had built this on Friday then had to run home to pick up a sick kid before it was done. This morning I didn't want to tell it to rebuild because of the time taken, but I guess that was a mistake? (Or did I also need to rebuild obs_subaru explicitly as well as pipe_drivers?)
            Hide
            lauren Lauren MacArthur added a comment -

            Indeed, you need to be updated to Perry Gee's fix from Sept 21 in obs_subaru: https://github.com/lsst/obs_subaru/commit/a2a63407b44f86e1e75ce99c171abd6ad1ac7614

            Show
            lauren Lauren MacArthur added a comment - Indeed, you need to be updated to Perry Gee 's fix from Sept 21 in obs_subaru : https://github.com/lsst/obs_subaru/commit/a2a63407b44f86e1e75ce99c171abd6ad1ac7614
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            Hide
            jbosch Jim Bosch added a comment -

            Sorry, modified the wrong ticket. I blame too many browser tabs.

            Show
            jbosch Jim Bosch added a comment - Sorry, modified the wrong ticket. I blame too many browser tabs.
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Done [ 10002 ] In Review [ 10004 ]
            Hide
            erykoff Eli Rykoff added a comment -

            Okay, everything ran smoothly now that I got the setups ordered properly. Looking at the files that are produced they look reasonable, though I don't have a "truth" to check against.

            Do I now select "Review Complete"?

            Show
            erykoff Eli Rykoff added a comment - Okay, everything ran smoothly now that I got the setups ordered properly. Looking at the files that are produced they look reasonable, though I don't have a "truth" to check against. Do I now select "Review Complete"?
            Hide
            lauren Lauren MacArthur added a comment -

            As detailed here, after running multiBandDriver you should see files corresponding to:

            • deepCoadd_mergeDet
            • deepCoadd_meas
            • deepCoadd_ref
            • deepCoadd_forced_src

            You hit the Review Complete button when you are satisfied that the work described has been accomplished and in accordance with our standards (coding, commit messages, passes Jenkins, etc.) and is suitable to be merged to master.

            Show
            lauren Lauren MacArthur added a comment - As detailed here , after running multiBandDriver you should see files corresponding to: deepCoadd_mergeDet deepCoadd_meas deepCoadd_ref deepCoadd_forced_src You hit the Review Complete button when you are satisfied that the work described has been accomplished and in accordance with our standards (coding, commit messages, passes Jenkins, etc.) and is suitable to be merged to master.
            Hide
            erykoff Eli Rykoff added a comment -

            Okay, everything looks hunky dory on the outputs, coding, commit message. How do I do the jenkins test? (Thanks for taking the time to help a newbie here!)

            Show
            erykoff Eli Rykoff added a comment - Okay, everything looks hunky dory on the outputs, coding, commit message. How do I do the jenkins test? (Thanks for taking the time to help a newbie here!)
            Hide
            lauren Lauren MacArthur added a comment -

            No problem! Actually, I don't think there is currently anything in Jenkins that exercises the tasks in pipe_drivers, so it doesn't really apply in this case, but we can at least make sure it builds.

            The instructions for Jenkins are here.

            I did a run against branch tickets/DM-7716 with just pipe_drivers in the product list. Console output is here. You can see it picked up the branch specified in every package where one exists (in this case, only pipe_drivers), all others build against current master.

            Show
            lauren Lauren MacArthur added a comment - No problem! Actually, I don't think there is currently anything in Jenkins that exercises the tasks in pipe_drivers , so it doesn't really apply in this case, but we can at least make sure it builds. The instructions for Jenkins are here . I did a run against branch tickets/ DM-7716 with just pipe_drivers in the product list. Console output is here . You can see it picked up the branch specified in every package where one exists (in this case, only pipe_drivers ), all others build against current master.
            Hide
            erykoff Eli Rykoff added a comment -

            I'm trying to replicate that output on Jenkins, and it only gives me the start: https://ci.lsst.codes/job/stack-os-matrix/16410/console Just a general checkout and no build, nothing about tickets/DM-7716, no tests, etc.

            Show
            erykoff Eli Rykoff added a comment - I'm trying to replicate that output on Jenkins, and it only gives me the start: https://ci.lsst.codes/job/stack-os-matrix/16410/console Just a general checkout and no build, nothing about tickets/ DM-7716 , no tests, etc.
            Hide
            tjenness Tim Jenness added a comment -

            https://ci.lsst.codes/job/stack-os-matrix/label=centos-7,python=py2/16410/console

            Takes a few clicks to get from the link you used (the root of the submission) to the specific output for each sub job.

            Show
            tjenness Tim Jenness added a comment - https://ci.lsst.codes/job/stack-os-matrix/label=centos-7,python=py2/16410/console Takes a few clicks to get from the link you used (the root of the submission) to the specific output for each sub job.
            Hide
            erykoff Eli Rykoff added a comment -

            Thanks, but where are those links? I've done everything but run a fuzz-test to click on all the links on the page, and I just keep going in circles.

            Show
            erykoff Eli Rykoff added a comment - Thanks, but where are those links? I've done everything but run a fuzz-test to click on all the links on the page, and I just keep going in circles.
            Hide
            tjenness Tim Jenness added a comment -

            Jonathan Sick this needs to go in the developer guide:

            The main problem is that the links at the bottom of that page don't take you directly to the triggered jobs, but take you to a root page for the collection of related triggered jobs.

            From the link you included scroll down to the "Triggering" line. Click on one of those. Then you are taken to a page of all the jobs for centos7/py2 or whatever. (Joshua Hoblitt is there any way for the link here to take you to the actual numbered job? Makes no sense as implemented). Your job was 16410 so click on that link. Now you are on the page with information about the centos7/py2 job associated with your submission. Click on "Console Output" and you'll get the information I put in my link.

            Show
            tjenness Tim Jenness added a comment - Jonathan Sick this needs to go in the developer guide: The main problem is that the links at the bottom of that page don't take you directly to the triggered jobs, but take you to a root page for the collection of related triggered jobs. From the link you included scroll down to the "Triggering" line. Click on one of those. Then you are taken to a page of all the jobs for centos7/py2 or whatever. ( Joshua Hoblitt is there any way for the link here to take you to the actual numbered job? Makes no sense as implemented). Your job was 16410 so click on that link. Now you are on the page with information about the centos7/py2 job associated with your submission. Click on "Console Output" and you'll get the information I put in my link.
            Hide
            erykoff Eli Rykoff added a comment -

            A-ha. Okay, from the main page https://ci.lsst.codes/job/stack-os-matrix/16410/ if you click on one of the configurations it takes you to a page that looks almost identical but isn't! In fact it's https://ci.lsst.codes/job/stack-os-matrix/16410/label=centos-6,python=py2/ and doesn't have the "Build Configurations" links. And from here, when you click on the Console Output it shows up.

            Therefore, this passes Jenkins, and I will click "Review Completed" (at long last!)

            Show
            erykoff Eli Rykoff added a comment - A-ha. Okay, from the main page https://ci.lsst.codes/job/stack-os-matrix/16410/ if you click on one of the configurations it takes you to a page that looks almost identical but isn't! In fact it's https://ci.lsst.codes/job/stack-os-matrix/16410/label=centos-6,python=py2/ and doesn't have the "Build Configurations" links. And from here , when you click on the Console Output it shows up. Therefore, this passes Jenkins, and I will click "Review Completed" (at long last!)
            Hide
            erykoff Eli Rykoff added a comment -

            Code produces expected output, passes all tests.

            Show
            erykoff Eli Rykoff added a comment - Code produces expected output, passes all tests.
            erykoff Eli Rykoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            jsick Jonathan Sick added a comment - - edited

            Jonathan Sick this needs to go in the developer guide:

            Thanks Tim. I agree that ci.lsst.codes needs to be documented, especially given the UX. I'll bring it up at SQuaRE's standup to see how we can schedule it.

            Show
            jsick Jonathan Sick added a comment - - edited Jonathan Sick this needs to go in the developer guide: Thanks Tim. I agree that ci.lsst.codes needs to be documented, especially given the UX. I'll bring it up at SQuaRE's standup to see how we can schedule it.
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-7603 [ DM-7603 ]
            swinbank John Swinbank made changes -
            Epic Link DM-6172 [ 24685 ]
            swinbank John Swinbank made changes -
            Sprint DRP F16-4 [ 246 ] DRP F16-4, DRP F16-5 [ 246, 252 ]
            swinbank John Swinbank made changes -
            Rank Ranked higher
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            swinbank John Swinbank added a comment -

            Lauren MacArthur – looks like something that should go in the release notes, right? Could you add a brief mention to https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+F16+Release+Notes please?

            Show
            swinbank John Swinbank added a comment - Lauren MacArthur – looks like something that should go in the release notes, right? Could you add a brief mention to https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+F16+Release+Notes please?
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ]
            Hide
            lauren Lauren MacArthur added a comment -

            Done.

            Show
            lauren Lauren MacArthur added a comment - Done.
            Hide
            swinbank John Swinbank added a comment -

            thank you!

            Show
            swinbank John Swinbank added a comment - thank you!
            rhl Robert Lupton made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            rhl Robert Lupton made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            price Paul Price made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            pschella Pim Schellart [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            rhl Robert Lupton made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            krzys Krzysztof Findeisen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14330 ] This issue links to "Page (Confluence)" [ 14330 ]

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Eli Rykoff
              Watchers:
              Eli Rykoff, Jim Bosch, John Swinbank, Jonathan Sick, Lauren MacArthur, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.