Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22389

verify fails with Astropy 4

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: verify
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      AP F19-6 (November)
    • Team:
      Alert Production

      Description

      Doing a build with Astropy4 I now get failures from verify package relating to YAML serialization.

      This works fine with the same version of YAML on Astropy 3 but fails on Astropy 4.

      self = <yaml.loader.SafeLoader object at 0x7fbb0063aba8>
      node = SequenceNode(tag='tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar', value=[MappingNode(tag='tag:yam...Node(tag='tag:yaml.org,2002:int', value='0')]))]), ScalarNode(tag='tag:yaml.org,2002:binary', value='AAAAAAAAFEA=\n')])
       
          def construct_undefined(self, node):
              raise ConstructorError(None, None,
                      "could not determine a constructor for the tag %r" % node.tag,
      >               node.start_mark)
      E       yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:numpy.core.multiarray.scalar'
      E         in "<unicode string>", line 9, column 14:
      E                 value: !!python/object/apply:numpy.core ... 
      E                        ^
       
      ../../stack/DarwinX86/pyyaml/5.1+2/lib/python/yaml/constructor.py:420: ConstructorError
      

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue is triggered by DM-22388 [ DM-22388 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-21875 [ DM-21875 ]
            Hide
            tjenness Tim Jenness added a comment -

            On Astropy 4 for the blob test:

            !<lsst.verify.Blob>
            data:
              mag1: !<lsst.verify.Datum>
                description: Magnitude
                label: mag1
                unit: mag
                value: !!python/object/apply:numpy.core.multiarray.scalar
                - &id001 !!python/object/apply:numpy.dtype
                  args:
                  - f8
                  - 0
                  - 1
                  state: !!python/tuple
                  - 3
                  - <
                  - null
                  - null
                  - null
                  - -1
                  - -1
                  - 0
                - !!binary |
                  AAAAAAAAFEA=
              mag2: !<lsst.verify.Datum>
                description: Magnitude
                label: mag2
                unit: mag
                value: !!python/object/apply:numpy.core.multiarray.scalar
                - *id001
                - !!binary |
                  AAAAAAAAJEA=
            identifier: 81b88b52a20b47bda3a11cb74db9f41b
            name: demo
            

            For the working test with astropy 3:

            !<lsst.verify.Blob>
            data:
              mag1: !<lsst.verify.Datum>
                description: Magnitude
                label: mag1
                unit: mag
                value: 5.0
              mag2: !<lsst.verify.Datum>
                description: Magnitude
                label: mag2
                unit: mag
                value: 10.0
            identifier: 78dfeee927cf4962bc2b1143b0e75184
            name: demo
            

            Show
            tjenness Tim Jenness added a comment - On Astropy 4 for the blob test: !<lsst.verify.Blob> data: mag1: !<lsst.verify.Datum> description: Magnitude label: mag1 unit: mag value: !!python/object/apply:numpy.core.multiarray.scalar - &id001 !!python/object/apply:numpy.dtype args: - f8 - 0 - 1 state: !!python/tuple - 3 - < - null - null - null - -1 - -1 - 0 - !!binary | AAAAAAAAFEA= mag2: !<lsst.verify.Datum> description: Magnitude label: mag2 unit: mag value: !!python/object/apply:numpy.core.multiarray.scalar - *id001 - !!binary | AAAAAAAAJEA= identifier: 81b88b52a20b47bda3a11cb74db9f41b name: demo For the working test with astropy 3: !<lsst.verify.Blob> data: mag1: !<lsst.verify.Datum> description: Magnitude label: mag1 unit: mag value: 5.0 mag2: !<lsst.verify.Datum> description: Magnitude label: mag2 unit: mag value: 10.0 identifier: 78dfeee927cf4962bc2b1143b0e75184 name: demo
            Hide
            tjenness Tim Jenness added a comment - - edited

            The difference seems to be in datum_representer where .value returns a float in astropy3 but numpy.float64 in astropy4.

            Show
            tjenness Tim Jenness added a comment - - edited The difference seems to be in datum_representer where .value returns a float in astropy3 but numpy.float64 in astropy4.
            krzys Krzysztof Findeisen made changes -
            Epic Link DM-21441 [ 423048 ]
            krzys Krzysztof Findeisen made changes -
            Sprint AP F19-6 (November) [ 958 ]
            Story Points 2
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            tjenness Tim Jenness added a comment -

            Astropy 4:

            >>> import astropy.units as u
            >>> import numpy
            >>> a = 5 * u.s
            >>> type(a.value)
            <class 'numpy.float64'>
            

            Astropy 3:

            >>> import astropy.units as u
            >>> import numpy
            >>> a = 5 * u.s
            >>> type(a.value)
            <class 'float'>
            

            Show
            tjenness Tim Jenness added a comment - Astropy 4: >>> import astropy.units as u >>> import numpy >>> a = 5 * u.s >>> type(a.value) <class 'numpy.float64'> Astropy 3: >>> import astropy.units as u >>> import numpy >>> a = 5 * u.s >>> type(a.value) <class 'float'>
            krzys Krzysztof Findeisen made changes -
            Comment [ This code looks suspicious: https://github.com/lsst/verify/blob/master/python/lsst/verify/datum.py#L109-L111. I can't figure out where it's called, though. ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "GitHub issue (Web Link)" [ 22582 ]
            Hide
            krzys Krzysztof Findeisen added a comment -

            Tim Jenness, can you test this on your system? I don't have Astropy 4 myself.

            Show
            krzys Krzysztof Findeisen added a comment - Tim Jenness , can you test this on your system? I don't have Astropy 4 myself.
            krzys Krzysztof Findeisen made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            krzys Krzysztof Findeisen made changes -
            Issue Type Story [ 10001 ] Bug [ 1 ]
            Hide
            tjenness Tim Jenness added a comment -

            Tests pass with this. Thank you for the quick fix.

            Show
            tjenness Tim Jenness added a comment - Tests pass with this. Thank you for the quick fix.
            krzys Krzysztof Findeisen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Tim Jenness
                Watchers:
                John Swinbank, Krzysztof Findeisen, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel