Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18690

Implement maximum record limit in TAP query screens in Firefly

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Firefly, SUIT
    • Labels:

      Description

      04/03/2019

      https://github.com/Caltech-IPAC/firefly/pull/783

      "Row Limit" field is displayed at the bottom of the TAP Search, along with "Search" and "Cancel" buttons. It is propagated to the server and is translated into MAXREC parameter of the search request. We allow to disable it by clearing.

      tap.maxrec.hardlimit property (default 10000000) is a config property. If MAXREC is set, it can not exceed this value.
      defaultMaxrec is an application property (default 50000 for Firefly). It is the default value for MAXREC parameter.

      Other changes:

      • Filters are now shown by default in the TAP result tables
      • Removed 'tables.showInfoButton' app property and made it a table option instead.
      • Fixed Service URL select box style: the text was truncated a bit on the bottom and selected item was shown in white, which was hard to see on light green.

      03/28/2019

      In the Firefly TAP search UI, we need a way to specify an upper limit on the size of the query result.

      This limit should be independent of any "TOP (nnn)" clause that may be included in the user's ADQL query text. If both are present, the desired effect is that the smaller of the two limits will determine the size of the query result.

      After extensive discussion in the IPAC LSST group, we propose to put the UI element for this limit in the "grey bar" at the bottom of the TAP search screens in which the "Search" and "Cancel" buttons already appear. The UI element must appear in both "single table" mode and in the type-my-own-ADQL mode, and should be in exactly the same location in the grey bar so that it does not "move" when changing modes. (This emphasizes that it is a global property of TAP searches via Firefly.)

      The UI element should allow the user to type in a non-negative integer (yes, I think 0 should be allowed - see the IVOA standards' discussion of MAXREC=0 for the reason), as well as to select suggested numbers from a menu. The menu must always contain at least one entry: a "hard limit" that cannot be exceeded even by typing. The value here should be the lesser of a "Firefly hard limit" (which should be a Firefly application-level configuration parameter) designed to preserve correct, if perhaps sluggish, performance, and any TAP-service-specific hard record limit (which might, for instance, have been returned from a .../capabilities endpoint query or from the Registry).

      The native Firefly hard limit on a good-sized server with the current implementation might be somewhere in the 2M-5M range.

      A default value should also be provided in the menu, and pre-selected in a new search session. This should also be controlled by a configuration parameter so that it can be application-specific, and this should be coded so that it can be overridden by a number obtained from the Registry. (A defaultMaxRecords parameter does exist.) A number like 5000 may be appropriate here.

      This ticket does not ask for the work of actually querying the Registry or the .../capabilities endpoint. The work on this ticket is only intended to facilitate adding those features at a later date.

      The above describes the UI.

      The implementation should ultimately be based on the use of the MAXREC DALI parameter to the underlying TAP service. However, we first need to determine whether LSST's TAP service supports this (and check IRSA and NED as well). It may be that we need a fallback to editing the ADQL query to add, or, if necessary, modify a user-provided TOP (nnn) clause.

        Attachments

          Issue Links

            Activity

            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            The equivalent UI element in TOPCAT is worth looking at for inspiration. It behaves mostly like what I'm looking for, though it does not enforce the upper limit on manually-entered text.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - The equivalent UI element in TOPCAT is worth looking at for inspiration. It behaves mostly like what I'm looking for, though it does not enforce the upper limit on manually-entered text.
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            We should still consider whether we need some way to turn this off completely, as there may be useful TAP services out there that support neither DALI MAXREC nor ADQL TOP.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - We should still consider whether we need some way to turn this off completely, as there may be useful TAP services out there that support neither DALI MAXREC nor ADQL TOP .
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            Based on discussions today about the experience with using this with the current LSST services, it seems clear that for now it will have to be possible for the user to turn off the limit. It may be reasonable for the UI to display some sort of warning in this situation, e.g., "Disabling the record limit is not recommended".

            Show
            gpdf Gregory Dubois-Felsmann added a comment - Based on discussions today about the experience with using this with the current LSST services, it seems clear that for now it will have to be possible for the user to turn off the limit. It may be reasonable for the UI to display some sort of warning in this situation, e.g., "Disabling the record limit is not recommended".

              People

              Assignee:
              tatianag Tatiana Goldina
              Reporter:
              gpdf Gregory Dubois-Felsmann
              Watchers:
              Gregory Dubois-Felsmann, Loi Ly
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.