Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15089

Display format for numerical values for LSST data

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      SUIT Sprint 2018-09, SUIT Sprint 2018-10, SUIT Sprint 2018-11, SUIT Sprint 2018-12, SUIT Sprint 2019-01, SUIT Sprint 2019-02, SUIT Sprint 2019-03, SUIT Sprint 2019-04
    • Team:
      Science User Interface

      Description

      This ticket is to gather information for numerical data display for LSST data in SUIT. 

      Current Firefly displays 6 digits after decimal point by default. This may not be best for some data. We need to have a plan either specify the precision for each column in table data, or have a guideline for precision for different type of data, i.e ra, dec, magnitude, flux, error, ...

        Attachments

          Issue Links

            Activity

            Hide
            tatianag Tatiana Goldina added a comment - - edited

            Currently Firefly uses %9g for FITS tables, guessing with minimum of 8 decimal digits - for VO Tables.  If the first value is in scientific format with 1 digit precision, all values will be formatted like this. 
             
            To Illustrate the problem, consider the following result table:

            <TABLE name="results">     
              <FIELD name="Float Field"  datatype="float"/>     
              <FIELD name="Double Field"  datatype="double"/>     
              <DATA>       
                <TABLEDATA>          
                  <TR><TD>10.68</TD><TD>10.68</TD></TR>
                  <TR><TD>287.3</TD><TD>287.3</TD></TR>
                  <TR><TD>1.2e-9</TD><TD>1.2e-9</TD></TR>
                </TABLEDATA>
              </DATA> 
            </TABLE> 

             Displayed in Firefly, the table will look like this:

            Float Column Double Column
            10.68000031 10.68000000
            287.29998779 287.30000000
            0.00000000 0.00000000

             I see at least 3 issues here:

            1. Float precision is exceeded
            2. Trailing zeroes are making wrong statement about the source data precision
            3. Guessing takes into account only the first non-null value. If the first value has 8 digit precision, all values will be formatted with this precision.
            Show
            tatianag Tatiana Goldina added a comment - - edited Currently Firefly uses %9g for FITS tables, guessing with minimum of 8 decimal digits - for VO Tables.  If the first value is in scientific format with 1 digit precision, all values will be formatted like this.    To Illustrate the problem, consider the following result table: <TABLE name="results">     <FIELD name="Float Field"  datatype="float"/>     <FIELD name="Double Field"  datatype="double"/>     <DATA>       <TABLEDATA>           <TR><TD>10.68</TD><TD>10.68</TD></TR> <TR><TD>287.3</TD><TD>287.3</TD></TR> <TR><TD>1.2e-9</TD><TD>1.2e-9</TD></TR> </TABLEDATA> </DATA> </TABLE>   Displayed in Firefly, the table will look like this: Float Column Double Column 10.68000031 10.68000000 287.29998779 287.30000000 0.00000000 0.00000000  I see at least 3 issues here: Float precision is exceeded Trailing zeroes are making wrong statement about the source data precision Guessing takes into account only the first non-null value. If the first value has 8 digit precision, all values will be formatted with this precision.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            We discussed this ticket a bit in the FireflyCCB meeting today.  We are going to talk about this in the FireflyCCB-D advisory group now, possibly under a new ticket in IPAC Jira (which we'll post here).

            Show
            gpdf Gregory Dubois-Felsmann added a comment - We discussed this ticket a bit in the FireflyCCB meeting today.  We are going to talk about this in the FireflyCCB-D advisory group now, possibly under a new ticket in IPAC Jira (which we'll post here).
            Hide
            tatianag Tatiana Goldina added a comment -

            When deciding how to be user friendly, we should understand, that whatever algorithm we choose to limit the precision of the displayed numbers, there will be a scenario (a set of numbers), when it fails. Hence if we are limiting the precision, we should allow to change it. And we should allow users to see the original data from the data provider. Hence, preserving the original precision in the absence of other information is the first step toward being user friendly.

            We should avoid displaying wrong data or misrepresenting precision.

            Loi has suggested using Java’s toString(). There is no loss of data and no trailing zeros. The disadvantage is that the formatting of a column might be inconsistent: since the trailing zeros are omitted, the numbers in a column might appear with the different number of decimal places. Also, choosing scientific or decimal format depends on the data value, see https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#toString(double)

            Using Java’s toString() works very well when the input data came as strings. With binary data (like binary FITS table), we’d display the original number. For example, if 0.1 is represented by 0.0999999999999659, we’d display 0.0999999999999659. This is not user friendly, but as long as we are not displaying wrong data — and we are displaying exactly what we have received — the rounding issues can be addressed later by allowing users to adjust the format.

            Show
            tatianag Tatiana Goldina added a comment - When deciding how to be user friendly, we should understand, that whatever algorithm we choose to limit the precision of the displayed numbers, there will be a scenario (a set of numbers), when it fails. Hence if we are limiting the precision, we should allow to change it. And we should allow users to see the original data from the data provider. Hence, preserving the original precision in the absence of other information is the first step toward being user friendly. We should avoid displaying wrong data or misrepresenting precision. Loi has suggested using Java’s toString(). There is no loss of data and no trailing zeros. The disadvantage is that the formatting of a column might be inconsistent: since the trailing zeros are omitted, the numbers in a column might appear with the different number of decimal places. Also, choosing scientific or decimal format depends on the data value, see https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#toString(double ) Using Java’s toString() works very well when the input data came as strings. With binary data (like binary FITS table), we’d display the original number. For example, if 0.1 is represented by 0.0999999999999659, we’d display 0.0999999999999659. This is not user friendly, but as long as we are not displaying wrong data — and we are displaying exactly what we have received — the rounding issues can be addressed later by allowing users to adjust the format.
            Hide
            ejoliet Emmanuel Joliet added a comment -

            Gregory Dubois-Felsmann could you post here the CCB ticket. I can't find it. Thanks.

            Show
            ejoliet Emmanuel Joliet added a comment - Gregory Dubois-Felsmann could you post here the CCB ticket. I can't find it. Thanks.
            Hide
            xiuqin Xiuqin Wu [X] (Inactive) added a comment -

            DM-15826 recognizes the VOTable precision for values and Firefly will respect that attribute. Specifically the following was in that ticket:

            Note: values of attributes precision and width are concatenated like 'wwEn' or 'wwFn'  and stored as 'precision' in DataType, where 'ww' stands for the value from attribute width and  'En' or 'Fn' stands for value from  attribute precision. If the original precision value contains numeric digit only like '2', it is converted to be 'F2'. 

            Show
            xiuqin Xiuqin Wu [X] (Inactive) added a comment - DM-15826 recognizes the VOTable precision for values and Firefly will respect that attribute. Specifically the following was in that ticket: Note: values of attributes precision and width are concatenated like ' wwEn'  or ' wwFn'   and stored as ' precision ' in  DataType,  where 'ww' stands for the value from attribute width and  'En' or 'Fn' stands for value from  attribute precision. If the original precision value contains numeric digit only like '2', it is converted to be 'F2'. 

              People

              • Assignee:
                gpdf Gregory Dubois-Felsmann
                Reporter:
                xiuqin Xiuqin Wu [X] (Inactive)
                Watchers:
                Emmanuel Joliet, Gregory Dubois-Felsmann, Loi Ly, Tatiana Goldina, Vandana Desai, Xiuqin Wu [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel