Angelo Fausti added a comment - 23/Dec/16 12:53 PM - edited Jonathan Sick currently the job JSON has a structure like this   "measurements" : [ { "metric" : "AM1" , "value" : 7.15136555363356 }, { "metric" : "AM2" , "value" : 6.80681963522785 }, { "metric" : "PA1" , "value" : 14.9064428565398 } ]   I imagine replacing the scalar measurement by the new measurement JSON: data[ 'measurements' ][ 0 ].keys()   [u 'blobs' , u 'parameters' , u 'metric' , u 'value' , u 'extras' , u 'spec_name' , u 'filter_name' , u 'identifier' , u 'unit' ] For measurements that are done in different filters or depend on those we can have multiple measurements for the same metric. That means we should have a list of measurements for each metric, e.g   "measurements" : [ { "metric" : "AM1" , ---> "measurement" : [] <--- }, { "metric" : "AM2" , "measurement" : [] }, { "metric" : "PA1" , "measurement" : [] } ]   For the datasets produced for each job we also need the blob JSON, example:   data[ 'blobs' ][ 0 ].keys()   [u 'identifier' , u 'data' , u 'name' ]     { "ci_id" : "2" , "ci_name" : "demo" , "ci_dataset" : "cfht" , "ci_label" : "centos-7" , "date" : "2016-06-02T05:21:57.298935Z" , "ci_url" : "https://ci.lsst.codes/job/validate_drp/dataset=cfht,label=centos-7/2/" , "status" : 0 , ---> "blobs" : {} <--- "measurements" : [ { "metric" : "AM1" , "measurement" : [ ] }, { "metric" : "AM2" , "measurement" : [ ] }, { "metric" : "PA1" , "value" : [ ] } ],   if it sounds reasonable I can mock that to continue development. The important thing for me now is to be able to retrieve the measurement JSON from the SQUASH API given the ci_id, ci_dataset and the metric and then call an URL to load the corresponding bokeh app https: //angelo-squash-bokeh.lsst.codes/photometry?metric=PA1&ci_dataset=cfht&ci_id=1

Jonathan Sick added a comment - 23/Dec/16 12:57 PM in the final JSON sample for the full REST json, blobs is an object/dict. Do you want the keys of this dict to be the blob identifier ? this would make it easy to look up from a measurement. If so, the actions that post-qa needs to do to shim it's native format to the SQUASH format is: Convert the blobs array to an object keyed by identifier . Convert the measurements array to an an array of objects with fields: 1) metric name, and 2) array of corresponding measurements. Thinking of the last one, it may make more sense to simply make measurements an object keyed by metric names, { "measurements": { {"AM1": [], <- array of measurement objects ...} } }

Angelo Fausti added a comment - 23/Dec/16 1:05 PM - edited I agree having blobs object keyed by identifier will make it easier to look up. Also, making measurements object keyed by metric name is better. Looks good, thanks for the suggestions. NOTE: the above conversation predates what is currently implemented. The current proposal for the model and serializes changes is slightly different.

Angelo Fausti added a comment - 28/Dec/16 12:49 PM - edited New schema proposed for the SQuaSH database, see corresponding changes in the models and serializers. new field data in the Job table to store datasets produced by the job (json blob) new field metadata in the Measurement table (json blob) renamed condition to operator in the Metric table to conform with validate_base metric definition renamed units fields to unit in the Metric table accept empty string in unit field (some metrics may not have unit) new fields in the Metric table: parameters, specs and reference to conform with validade_base metric definition (json blobs)

Angelo Fausti added a comment - 18/Jan/17 1:47 PM - edited Here is how the API endpoints look like after changing the model and serializers. Notice that measurements metadata is not present in the Metrics App endpoint. We kept just the fields required for this app minimizing the amount of data returned by this endpoint.

Angelo Fausti added a comment - 18/Jan/17 2:12 PM See also PR https://github.com/lsst-sqre/qa-dashboard/pull/31

Jonathan Sick added a comment - 19/Jan/17 3:10 PM This looks fine. Some comments are in the PR. The main need I see is documenting the schema of these JSON fields (maybe in SQR-008) so that it's clear what data is in them (there's a slight difference between the json schema made by validate_base and what finally ends up in these JSON fields and it's good to make it clear).

Angelo Fausti added a comment - 19/Jan/17 3:30 PM Thanks, yes I will document that properly in the SQuaSH technote, basically the mapping of validade_base Job to SQUASH API is the following 1) the content of blobs is ingested into the Job.data field (make sense to change the field name to blobs ) 2) the content of measurements is ingested in the Measurement.metadata field, with the exception of the metric specification which is ingested in the Metric table for normalization 3) the measurement value is kept in a separate field in the Measurement table for convenience 4) the metrics specification has unit , description , operator , specs and reference which are all individual fields in the Metric table.

Angelo Fausti added a comment - 19/Jan/17 4:11 PM Applied PR comments