Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27062

Experiment with making a simple PNG plot browser for pipe-analysis plots

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Explore using panel/holoviews to serve static PNG plots produced by pipe_analysis scripts in a simple browser interface.

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            Browser-based interactive PNG explorer for pipe_analysis plots here: https://github.com/timothydmorton/pipe-analysis-navigator

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - Browser-based interactive PNG explorer for pipe_analysis plots here:  https://github.com/timothydmorton/pipe-analysis-navigator
            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            Check this out, let me know if it works for you, and let me know if you have any suggestions.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - Check this out, let me know if it works for you, and let me know if you have any suggestions.
            Hide
            lauren Lauren MacArthur added a comment -

            Sorry for the delay getting to this... I went to have a look shortly after it came in, but was expecting to be able to just point to a URL from the browser on my iPad to peruse the interface.  When I discovered that wasn’t possible, I stalled out and let it slip down my inbox (so sorry!). I will give the port forwarding dance a try, but am very keen to know if direct access with out that is going to be possible?

            Show
            lauren Lauren MacArthur added a comment - Sorry for the delay getting to this... I went to have a look shortly after it came in, but was expecting to be able to just point to a URL from the browser on my iPad to peruse the interface.  When I discovered that wasn’t possible, I stalled out and let it slip down my inbox (so sorry!). I will give the port forwarding dance a try, but am very keen to know if direct access with out that is going to be possible?
            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            No worries-- It should in principle be possible to have this running somewhere that can access the data, accessible by URL, but I'm not sure whom to ask about that.  Yusra AlSayyad or Jim Bosch do you know whom we should approach about this possibility?  Lauren, correct me if I'm wrong, but what we're going for here is that we want Lauren to be able to point her iPad browser to a url and access an interface like this.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - No worries-- It should in principle be possible to have this running somewhere that can access the data, accessible by URL, but I'm not sure whom to ask about that.  Yusra AlSayyad or Jim Bosch do you know whom we should approach about this possibility?  Lauren, correct me if I'm wrong, but what we're going for here is that we want Lauren to be able to point her iPad browser to a url and access an interface like this.
            Hide
            jbosch Jim Bosch added a comment -

            I'm not sure who to ask - probably someone at NCSA, but we may want to loop in others to make sure we also have a similar setup in the Interim Data Facility (Google Cloud), where this will be even more important.  Maybe just bring it up in #dm on slack?

            Show
            jbosch Jim Bosch added a comment - I'm not sure who to ask - probably someone at NCSA, but we may want to loop in others to make sure we also have a similar setup in the Interim Data Facility (Google Cloud), where this will be even more important.  Maybe just bring it up in #dm on slack?
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Ok, finally got around to this...and it looks really nice!  A few comments to consider:

            • it would definitely be helpful to have a way to change the plot sizes (even just a "fit to window" of sorts would help...as is, the two plot columns already extend beyond the browser window size I usually default to).
            • along those lines, is it possible to make the "Plots" window horizontally scrollable to see the full filenames? That way we could perhaps save on some "real estate" (a term I learned during our calls with the folks at Quansight!) on the leftmost column.  Also, if the full names could "pop up" on hovering, that would be really nice too.
            • there was some lag after hitting submit, so it would be nice to have a way to know it's "thinking"
            • I can only see the GalacticExtinction colorAnalysis plots (I assume because the others don't have a "filter" in their filename?). Perhaps it would make most sense to have "color" as one of the filters and they all appear if that is selected as the "filter"?
            • I often like to blink between two repos. I was able to to that by opening another dashboard, pointing it at a different repo, and blinking the tabs having made the same selection of plots in both (which is totally fine, but just thought I'd point out another use case in case you have even better ideas!)
            • Big caveat: at I first thought things weren't working when I hit "submit" having pointed to my recently made /datasets/hsc/repo/rerun/private/lauren/DM-22266/parq/plots and nothing appeared. Then looking at the traceback I see:

                File "/home/lauren/LSST/pipe-analysis-navigator/navigator/parse.py", line 77, in build_df
                  raise RuntimeError(f'Please add the following plot types to {category_path} and run again: {missing_content}')
              RuntimeError: Please add the following plot types to /home/lauren/LSST/pipe-analysis-navigator/navigator/categories.csv and run again: {'footArea_calib_psf_used', 'matches_CircApRad12pix', 'diff_footArea_calib_psf_used', 'diff_footArea', 'footArea', 'matches_PSF_mag_calib_astrometry_used', 'matches_PSF', 'matches_CircApRad12pix_mag_calib_astrometry_used'}
              

              So, yep, I have made some updates to the plots produced and their names. I get that the plot names probably need to be hard-coded in order to sort them into their appropriate categories (which you did a nice job of!  Oh, except I might have put the "pStar" plots into the "shape" category), so perhaps we just have to be aware of each other and that updates to the plots produced (in name or addition) should be accompanied by an associated update to the dashboard.  Maybe we could also make it not a hard fail mode and just put the unrecognized plots in a catchall category (but keeping a WARNING message in the traceback)?  Also, having pasted the failed repo into the "Repository" and hitting "Submit", the plots from the old repo were retained, so one might be mislead into thinking the new repo had loaded.

            • Nit-pick: it feels all-too-natural to expect hitting a carriage return in the "Repository" box after entering a location.  I keep getting caught by the need to actually hit the "Submit" button (but I can get over that!)
            • Ok, one more nit-pick:  the ordering of the plots displayed seems to be determined by the order in which they are listed in the "Plots" window, so when ones adds another to the selection, they all get rearranged.  Could that be changed to the order in which they were selected (I get that may be tricky for a simultaneous/shift selection of multiple plots).
            • I'm not sure if this is possible, but it would be super cool if the "Repository" could turn into a dropbox of all repos available given a "ROOT" path (e.g. all the RC2 runs would show up as selections if one entered datasets/hsc/repo/rerun/RC).
            Show
            lauren Lauren MacArthur added a comment - - edited Ok, finally got around to this...and it looks really nice!  A few comments to consider: it would definitely be helpful to have a way to change the plot sizes (even just a "fit to window" of sorts would help...as is, the two plot columns already extend beyond the browser window size I usually default to). along those lines, is it possible to make the "Plots" window horizontally scrollable to see the full filenames? That way we could perhaps save on some "real estate" (a term I learned during our calls with the folks at Quansight!) on the leftmost column.  Also, if the full names could "pop up" on hovering, that would be really nice too. there was some lag after hitting submit, so it would be nice to have a way to know it's "thinking" I can only see the GalacticExtinction colorAnalysis plots (I assume because the others don't have a "filter" in their filename?). Perhaps it would make most sense to have "color" as one of the filters and they all appear if that is selected as the "filter"? I often like to blink between two repos. I was able to to that by opening another dashboard, pointing it at a different repo, and blinking the tabs having made the same selection of plots in both (which is totally fine, but just thought I'd point out another use case in case you have even better ideas!) Big caveat: at I first thought things weren't working when I hit "submit" having pointed to my recently made /datasets/hsc/repo/rerun/private/lauren/ DM-22266 /parq/plots and nothing appeared. Then looking at the traceback I see: File "/home/lauren/LSST/pipe-analysis-navigator/navigator/parse.py" , line 77 , in build_df raise RuntimeError(f 'Please add the following plot types to {category_path} and run again: {missing_content}' ) RuntimeError: Please add the following plot types to / home / lauren / LSST / pipe - analysis - navigator / navigator / categories.csv and run again: { 'footArea_calib_psf_used' , 'matches_CircApRad12pix' , 'diff_footArea_calib_psf_used' , 'diff_footArea' , 'footArea' , 'matches_PSF_mag_calib_astrometry_used' , 'matches_PSF' , 'matches_CircApRad12pix_mag_calib_astrometry_used' } So, yep, I have made some updates to the plots produced and their names. I get that the plot names probably need to be hard-coded in order to sort them into their appropriate categories (which you did a nice job of!  Oh, except I might have put the "pStar" plots into the "shape" category), so perhaps we just have to be aware of each other and that updates to the plots produced (in name or addition) should be accompanied by an associated update to the dashboard.  Maybe we could also make it not a hard fail mode and just put the unrecognized plots in a catchall category (but keeping a WARNING message in the traceback)?  Also, having pasted the failed repo into the "Repository" and hitting "Submit", the plots from the old repo were retained, so one might be mislead into thinking the new repo had loaded. Nit-pick: it feels all-too-natural to expect hitting a carriage return in the "Repository" box after entering a location.  I keep getting caught by the need to actually hit the "Submit" button (but I can get over that!) Ok, one more nit-pick:  the ordering of the plots displayed seems to be determined by the order in which they are listed in the "Plots" window, so when ones adds another to the selection, they all get rearranged.  Could that be changed to the order in which they were selected (I get that may be tricky for a simultaneous/shift selection of multiple plots). I'm not sure if this is possible, but it would be super cool if the "Repository" could turn into a dropbox of all repos available given a "ROOT" path (e.g. all the RC2 runs would show up as selections if one entered datasets/hsc/repo/rerun/RC ).
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks, Tim!  Hopefully the access issue can be resolved as this does indeed provide a nice plot perusing experience 

            Show
            lauren Lauren MacArthur added a comment - Thanks, Tim!  Hopefully the access issue can be resolved as this does indeed provide a nice plot perusing experience 
            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            Thanks, Lauren!  I should be able to implement some (though not all) of these suggestions.  I will also be in touch with Sophie Reed to work on a gen3 backend for this as well.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - Thanks, Lauren!  I should be able to implement some (though not all) of these suggestions.  I will also be in touch with Sophie Reed to work on a gen3 backend for this as well.

              People

              Assignee:
              tmorton Tim Morton [X] (Inactive)
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Reviewers:
              Lauren MacArthur
              Watchers:
              Jim Bosch, Lauren MacArthur, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.