Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24773

Input tract information for previous RC2 runs in the SQuaSH database

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      After DM-18529, tract ID is added as metadata to the verification jobs and that allows us to upload metrics for multiple tracts to SQuaSH and use the tract tag in InfluxDB to create dashboards filtering the results based on the track ID.

      However, currently a query like:

      SELECT "AM1" FROM "squash-demo"."autogen"."validate_drp" WHERE  "dataset"='HSC RC2' AND "filter"='HSC-R' AND "tract"='9813'

      returns only the last data point, because the previous data points don't have the tract information. That was expected and we need to add the InfluxDB tag tract=9813 to all previous data points to fix this.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24669 Update RC2 dashboards in squash now that we're ingesting 3 tracts.

          • Done

            Activity

            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            InfluxDB does not have a mechanism to UPDATE tags for existing values, see for example https://github.com/influxdata/influxdb/issues/3904

            A workaround is to use an InfluxDB client, read the data for example as a Pandas Dataframe, add or modify the new column in Pandas and write it back to InfluxDB. This notebook demonstrate that using the aioinflux Python client to input the "tract" tag value for previous runs of the HSC RC2 reprocessing.

            https://github.com/lsst-sqre/squash-restful-api/pull/47

             

            Show
            afausti Angelo Fausti added a comment - - edited InfluxDB does not have a mechanism to UPDATE tags for existing values, see for example  https://github.com/influxdata/influxdb/issues/3904 A workaround is to use an InfluxDB client, read the data for example as a Pandas Dataframe, add or modify the new column in Pandas and write it back to InfluxDB. This notebook demonstrate that using the aioinflux Python client to input the "tract" tag value for previous runs of the HSC RC2 reprocessing. https://github.com/lsst-sqre/squash-restful-api/pull/47  

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.