Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31131

lsst.verify outputs non-standard NaN in JSON

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • verify
    • None

    Description

      lsst.verify outputs NaN to its JSON as it uses Python's json.dump() with allow_nan=True. The resulting JSON is non-standard. requests 2.26.0 has changed to enforce the standard, returning an error when attempting to use HTTP POST on such a JSON object.

      In the short term, we can disable validate_drp_gen3, which is generating these NaNs. (ap_verify does not appear to do so at present.)

      In the long term, we will either need to convince requests to enable "relaxed enforcement" or change our representation of NaN to something standards-compliant. This is the responsibility of lsst.verify.

      Attachments

        Issue Links

          Activity

            I would like a unit test for the case where a Measurement has a NaN value, but other than that, this looks like an improvement.

            krughoff Simon Krughoff (Inactive) added a comment - I would like a unit test for the case where a Measurement has a NaN value, but other than that, this looks like an improvement.
            afausti Angelo Fausti added a comment - - edited

            Thanks krughoff I've added a test in verify for the (de)serialization of NaN and Inf values. I agree that deserializing JSON null always back to NaN breaks the symmetry but I don't have a better idea here.

            afausti Angelo Fausti added a comment - - edited Thanks krughoff I've added a test in verify for the (de)serialization of NaN and Inf values. I agree that deserializing JSON null always back to NaN breaks the symmetry but I don't have a better idea here.

            Looks great.  Thanks for adding the test.

            krughoff Simon Krughoff (Inactive) added a comment - Looks great.  Thanks for adding the test.

            The SQuaSH API was updated and verify change was merged.

            afausti Angelo Fausti added a comment - The SQuaSH API was updated and verify change was merged.

            Verifying the results from the verify_drp_metrics CI run #21

            YAML output

             
            !<lsst.verify.Measurement>
            blobs:
            - !<lsst.verify.Blob>
              data: {}
              identifier: 43ba92ab3eb6487ea3c8c7ef7457823a
              name: AD3_design
            identifier: beadc1d7ef3f443fbb58bc84e9fba0e4
            metric: AD3_design
            notes: {}
            unit: marcsec
            value: .nan
            

            JSON output

            { 
                  "blob_refs": [],
                  "identifier": "beadc1d7ef3f443fbb58bc84e9fba0e4",
                  "metric": "validate_drp.AD3_design",
                  "unit": "marcsec",
                  "value": null
             }
            

            And from the SQuaSH API https://squash-restful-api.lsst.codes/job/51761

            {"value": null, "unit": "marcsec", "metric": "validate_drp.AD3_design"}
            

            afausti Angelo Fausti added a comment - Verifying the results from the verify_drp_metrics CI run #21 YAML output !<lsst.verify.Measurement> blobs: - !<lsst.verify.Blob> data: {} identifier: 43ba92ab3eb6487ea3c8c7ef7457823a name: AD3_design identifier: beadc1d7ef3f443fbb58bc84e9fba0e4 metric: AD3_design notes: {} unit: marcsec value: .nan JSON output { "blob_refs": [], "identifier": "beadc1d7ef3f443fbb58bc84e9fba0e4", "metric": "validate_drp.AD3_design", "unit": "marcsec", "value": null } And from the SQuaSH API https://squash-restful-api.lsst.codes/job/51761 {"value": null, "unit": "marcsec", "metric": "validate_drp.AD3_design"}

            People

              afausti Angelo Fausti
              ktl Kian-Tat Lim
              Simon Krughoff (Inactive)
              Angelo Fausti, Colin Slater, Jeffrey Carlin, Kian-Tat Lim, Leanne Guy, Simon Krughoff (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.