Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10832

Heatmap (density) plot in multi-trace chart



    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: SUIT
    • Labels:
    • Story Points:
    • Sprint:
      SUIT Sprint 2017-6, SUIT Sprint 2017-7
    • Team:
      Science User Interface


      We need to migrate Firefly density plot (Heatmap with the data calculated on server side) to multi-trace chart architecture. This includes options and table connections.

      This ticket is dealing with the options for heatmap. It will not change the server-side algorithm. Currently I am planning to include the options for number of bins in X and Y direction and add an option for Plotly color scale. (If needed we can also use the discretized color scale as we did before.)

      07/05/2017 TG

      Per Xiuqin’s request, I am adding current server side implementation details for the binning algorithm, which is supposed to split XY space into rectangles and record how many points are falling into each bin. (xUnit and yUnit below are the dimensions of each bin.)


        xColumnOrExpr and yColumnOrExpr – columns or column expressions for x and y
        maxPoints – maximum number of bins
        xyRatio (default 1, aspect ratio of the plot – if we want bins to be more square-like)
        xMin, xMax, yMin, yMax (if NaN, use min or max value of the corresponding column/column expression)


      1. “decimated" table with the following columns:
              x, y, rowidx, weight, decimateKey  
              x, y – x and Y values of a “representative” point – a random point from the bin
              rowidx – row index of the “representative” point in the original table
              weight – total number of points in the bin
              decimateKey – a string "<xIndex>:<yIndex>”, which identifies the bin
      2. decimate key information, which contains xMin, xMax, xUnit, and yUnit
                  this information allows to calculate center of the bin from x and y values above


      1. Calculate min and max of x and y columns OR accept them as parameters.
      2. Calculate xUnit and yUnit:
          nXbins = (int)sqrt(maxPoints*xyRatio)
          nYbins = (int)sqrt(maxPoints/xyRatio)
          xUnit = (xMax - xMin)/nXs
          yUnit = (yMax - yMin)/nYs;
          // increase cell size a bit to include max values into grid
          xUnit += xUnit/1000.0/nXs;
          yUnit += yUnit/1000.0/nYs;
      3. Create a map from bin key (decimateKey) to sample (“representative”) point.
          for each row
             get x and y values (xval and yval) for this row
             if either x or y and NaN or the point is not within limits, continue to the next row
             calculate decimateKey:
                xIndex = (int)((xval-xMin)/xUnit)
                yIndex = (int)((yval-yMin)/yUnit)
                decimateKey = xIndex+”:”+yIndex
            if decimateKey is already present in the map:
                increase the number of represented rows  weight for this bin by one           
                randomly decide if the current row should be chosen as a “representative”:
                   if (Math.random < 1.0/weight):
                      make current row representative
               add current point to the map with weight 1
      4. Create an output table from the map created in step 3

      The algorithm is implemented in doDecimation method of QueryUtil.java


          Issue Links


            There are no comments yet on this issue.


              tatianag Tatiana Goldina
              tatianag Tatiana Goldina
              Loi Ly
              Emmanuel Joliet, Loi Ly, Tatiana Goldina, Xiuqin Wu [X] (Inactive)
              0 Vote for this issue
              4 Start watching this issue