Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10619

Build a prototype bokeh server implementation to demonstrate desired interactive QA plots

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      10
    • Epic Link:
    • Sprint:
      DRP F17-1, DRP F17-2
    • Team:
      Data Release Production

      Description

      Following the plan made in DM-10045, make an example demonstration of the type of plots we will want for interactive QA plotting. Implement data computations to mirror the current structure of the pipe_analysis plotting code, in order to simplify the future process of porting more of its functionality into Bokeh.

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton added a comment - - edited

            Attached are a few screen shots from the prototype QA dashboard. The elements are (1) a scatter plot of [something] vs. [some magnitude] (upper left); (2) an ra-dec sky plot, colored according to the y-axis on the top-left plot; (3) histograms of the x- and y-values of plot (1); (4) drop-down menus; (5) slider bars; and (6) a table listing selected points. (1) and (2) are pan/zoomable in both dimensions with the correct tools selected, and (3) pans/zooms on the x-axis without any options to change tools. (2) and (6) have separate tabs for each "label" (currently only star/galaxy in this demo). Points may be selected in (1) or (2), and they are linked. Changing points selection updates the histograms in (3) and the rows listed in (6). (4) changes what the x- and y-values of (1) are, and (5) adjusts the point size and alpha for (1) and (2). Legends for (1) and (3) are clickable, allowing for toggling whether the different categories are visible or not in the plots.

            The data for this demo comes from running coaddAnalysis.py on a single patch of an HSC rerun, from which I dumped one of the created tables (that are not currently being officially persisted in any way). The code for this demo is currently living at https://github.com/timothydmorton/bokeh-explore.

            Show
            tmorton Tim Morton added a comment - - edited Attached are a few screen shots from the prototype QA dashboard. The elements are (1) a scatter plot of [something] vs. [some magnitude] (upper left); (2) an ra-dec sky plot, colored according to the y-axis on the top-left plot; (3) histograms of the x- and y-values of plot (1); (4) drop-down menus; (5) slider bars; and (6) a table listing selected points. (1) and (2) are pan/zoomable in both dimensions with the correct tools selected, and (3) pans/zooms on the x-axis without any options to change tools. (2) and (6) have separate tabs for each "label" (currently only star/galaxy in this demo). Points may be selected in (1) or (2), and they are linked. Changing points selection updates the histograms in (3) and the rows listed in (6). (4) changes what the x- and y-values of (1) are, and (5) adjusts the point size and alpha for (1) and (2). Legends for (1) and (3) are clickable, allowing for toggling whether the different categories are visible or not in the plots. The data for this demo comes from running coaddAnalysis.py on a single patch of an HSC rerun, from which I dumped one of the created tables (that are not currently being officially persisted in any way). The code for this demo is currently living at https://github.com/timothydmorton/bokeh-explore .
            Hide
            tmorton Tim Morton added a comment - - edited

            I presented this at the DRP group meeting on Monday, 7/10, where I received the following feedback:

            • Robert Lupton requested there be the ability to have useful (ideally custom-defined in some way) callbacks attached to actions such as clicking on points in the scatter plot. Currently the demo can display basic info using a hover tool.
            • A request for text-based queries on the catalog being explored, to enable more nuanced data exploration.
            • A request for the ability to choose custom columns (and/or linear combinations thereof), rather than simply pre-established computations on the data.
            • There was a desire to see an 'all' category in the histograms, rather than having categories be individually normalized.
            • Several were curious about integrating and using these plots within JupyterLab, in order to allow more custom interaction.

            John Swinbank should I mark you as the reviewer for this?

            Show
            tmorton Tim Morton added a comment - - edited I presented this at the DRP group meeting on Monday, 7/10, where I received the following feedback: Robert Lupton requested there be the ability to have useful (ideally custom-defined in some way) callbacks attached to actions such as clicking on points in the scatter plot. Currently the demo can display basic info using a hover tool. A request for text-based queries on the catalog being explored, to enable more nuanced data exploration. A request for the ability to choose custom columns (and/or linear combinations thereof), rather than simply pre-established computations on the data. There was a desire to see an 'all' category in the histograms, rather than having categories be individually normalized. Several were curious about integrating and using these plots within JupyterLab, in order to allow more custom interaction. John Swinbank should I mark you as the reviewer for this?
            Hide
            tmorton Tim Morton added a comment -

            I'll also mention that the plots currently in this demo dashboard reproduce most of the features of the plots created by coaddAnalysis.py, but is only a first step toward replacing the QA information generated by the pipe_analysis scripts. In particular, the figures produced by colorAnalysis.py should be integrated as well. I have also been using only one patch of data. There may be significant programmatic changes required when the data are extended to use an entire patch, for example.

            Show
            tmorton Tim Morton added a comment - I'll also mention that the plots currently in this demo dashboard reproduce most of the features of the plots created by coaddAnalysis.py , but is only a first step toward replacing the QA information generated by the pipe_analysis scripts. In particular, the figures produced by colorAnalysis.py should be integrated as well. I have also been using only one patch of data. There may be significant programmatic changes required when the data are extended to use an entire patch, for example.
            Hide
            swinbank John Swinbank added a comment -

            Thanks Tim Morton; this is great.

            The only thing I'd request is that you ticket the various feature requests you've received and link them back to this issue. Other than that, good to go!

            Show
            swinbank John Swinbank added a comment - Thanks Tim Morton ; this is great. The only thing I'd request is that you ticket the various feature requests you've received and link them back to this issue. Other than that, good to go!
            Hide
            afausti Angelo Fausti added a comment -

            Tim Morton nice work with Bokeh plots looking forward to interact more with you in the context of the Bokeh and JupyterLab integration.

            Show
            afausti Angelo Fausti added a comment - Tim Morton nice work with Bokeh plots looking forward to interact more with you in the context of the Bokeh and JupyterLab integration.

              People

              • Assignee:
                tmorton Tim Morton
                Reporter:
                tmorton Tim Morton
                Reviewers:
                John Swinbank
                Watchers:
                Angelo Fausti, John Swinbank, Tim Morton, Xiuqin Wu [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel