Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15462

Failure in lsst_ci with numpy1.15

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: lsst_ci, validate_drp
    • Labels:
      None
    • Team:
      External

      Description

      Testing bin/deploy -b with DM-15385, DM-15419, DM-15394, DM-15389, and DM-15443 (Python 3.7.0, numpy 1.15.0, and astropy 3.0.4) I get the following error from lsst_ci:

      [2018-08-18T00:45:21.894199Z] Traceback (most recent call last):
      [2018-08-18T00:45:21.894202Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
      [2018-08-18T00:45:21.894204Z]     validate.run(args.repo, **kwargs)
      [2018-08-18T00:45:21.894206Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
      [2018-08-18T00:45:21.894208Z]     metrics_package=metrics_package, **kwargs)
      [2018-08-18T00:45:21.894210Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
      [2018-08-18T00:45:21.894212Z]     metrics_package=metrics_package, **kwargs)
      [2018-08-18T00:45:21.894214Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 350, in runOneFilter
      [2018-08-18T00:45:21.894216Z]     pa2 = measurePA2(metrics[pa2_spec.metric_name], pa1, pf1_spec.threshold)
      [2018-08-18T00:45:21.894218Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/calcsrd/pa2.py", line 62, in measurePA2
      [2018-08-18T00:45:21.894220Z]     return Measurement(metric, np.percentile(np.abs(magDiffs), pf1Percentile) * magDiffs.unit,
      [2018-08-18T00:45:21.894222Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
      [2018-08-18T00:45:21.894224Z]     a, q, axis, out, overwrite_input, interpolation, keepdims)
      [2018-08-18T00:45:21.894226Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
      [2018-08-18T00:45:21.894229Z]     interpolation=interpolation)
      [2018-08-18T00:45:21.894231Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
      [2018-08-18T00:45:21.894233Z]     r = func(a, **kwargs)
      [2018-08-18T00:45:21.894235Z]   File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func
      [2018-08-18T00:45:21.894237Z]     ap.partition(concatenate((indices_below, indices_above)), axis=axis)
      [2018-08-18T00:45:21.894239Z] TypeError: Partition index must be integer
      [2018-08-18T00:45:21.894241Z] ===================== 2 failed, 1 passed in 409.12 seconds =====================
      

      The fix is to edit lsst.validate.drp.calcsrd.measurePA2 to cast pf1Percentile to a float before calling np.percentile. I have built this without problem with numpy1.14.5, astropy3.0.3 and python 3.6.6.

      The problem is numpy 1.15. Using python 3.6.6, astropy 3.0.4 and numpy 1.15 also fails in the same way.

        Attachments

          Issue Links

            Activity

            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Risk Score 0
            tjenness Tim Jenness made changes -
            Issue Type Story [ 10001 ] Bug [ 1 ]
            tjenness Tim Jenness made changes -
            Description Testing {{bin/deploy -b}} with DM-15385, DM-15419, DM-15394, DM-15389, and DM-15443 (Python 3.7.0, numpy 1.15.0, and astropy 3.0.4) I get the following error from lsst_ci:

            {code}
            [2018-08-18T00:45:21.894199Z] Traceback (most recent call last):
            [2018-08-18T00:45:21.894202Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
            [2018-08-18T00:45:21.894204Z] validate.run(args.repo, **kwargs)
            [2018-08-18T00:45:21.894206Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
            [2018-08-18T00:45:21.894208Z] metrics_package=metrics_package, **kwargs)
            [2018-08-18T00:45:21.894210Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
            [2018-08-18T00:45:21.894212Z] metrics_package=metrics_package, **kwargs)
            [2018-08-18T00:45:21.894214Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 350, in runOneFilter
            [2018-08-18T00:45:21.894216Z] pa2 = measurePA2(metrics[pa2_spec.metric_name], pa1, pf1_spec.threshold)
            [2018-08-18T00:45:21.894218Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/calcsrd/pa2.py", line 62, in measurePA2
            [2018-08-18T00:45:21.894220Z] return Measurement(metric, np.percentile(np.abs(magDiffs), pf1Percentile) * magDiffs.unit,
            [2018-08-18T00:45:21.894222Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
            [2018-08-18T00:45:21.894224Z] a, q, axis, out, overwrite_input, interpolation, keepdims)
            [2018-08-18T00:45:21.894226Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
            [2018-08-18T00:45:21.894229Z] interpolation=interpolation)
            [2018-08-18T00:45:21.894231Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
            [2018-08-18T00:45:21.894233Z] r = func(a, **kwargs)
            [2018-08-18T00:45:21.894235Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func
            [2018-08-18T00:45:21.894237Z] ap.partition(concatenate((indices_below, indices_above)), axis=axis)
            [2018-08-18T00:45:21.894239Z] TypeError: Partition index must be integer
            [2018-08-18T00:45:21.894241Z] ===================== 2 failed, 1 passed in 409.12 seconds =====================
            {code}

            The fix is to edit {{lsst.validate.drp.calcsrd.measurePA2}} to cast {{pf1Percentile}} to a {{float}} before calling {{np.percentile}}. I have built this without problem with numpy1.14.5, astropy3.0.3 and python 3.6.6. I will now try to do a build with python 3.6 and update numpy/astropy.
            Testing {{bin/deploy -b}} with DM-15385, DM-15419, DM-15394, DM-15389, and DM-15443 (Python 3.7.0, numpy 1.15.0, and astropy 3.0.4) I get the following error from lsst_ci:

            {code}
            [2018-08-18T00:45:21.894199Z] Traceback (most recent call last):
            [2018-08-18T00:45:21.894202Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/bin/validateDrp.py", line 104, in <module>
            [2018-08-18T00:45:21.894204Z] validate.run(args.repo, **kwargs)
            [2018-08-18T00:45:21.894206Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 152, in run
            [2018-08-18T00:45:21.894208Z] metrics_package=metrics_package, **kwargs)
            [2018-08-18T00:45:21.894210Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 243, in runOneRepo
            [2018-08-18T00:45:21.894212Z] metrics_package=metrics_package, **kwargs)
            [2018-08-18T00:45:21.894214Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/validate.py", line 350, in runOneFilter
            [2018-08-18T00:45:21.894216Z] pa2 = measurePA2(metrics[pa2_spec.metric_name], pa1, pf1_spec.threshold)
            [2018-08-18T00:45:21.894218Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/stack/DarwinX86/validate_drp/16.0-9-gf51232b+1/python/lsst/validate/drp/calcsrd/pa2.py", line 62, in measurePA2
            [2018-08-18T00:45:21.894220Z] return Measurement(metric, np.percentile(np.abs(magDiffs), pf1Percentile) * magDiffs.unit,
            [2018-08-18T00:45:21.894222Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
            [2018-08-18T00:45:21.894224Z] a, q, axis, out, overwrite_input, interpolation, keepdims)
            [2018-08-18T00:45:21.894226Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
            [2018-08-18T00:45:21.894229Z] interpolation=interpolation)
            [2018-08-18T00:45:21.894231Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
            [2018-08-18T00:45:21.894233Z] r = func(a, **kwargs)
            [2018-08-18T00:45:21.894235Z] File "/Volumes/ExternalSSD/Users/timj/work/lsstsw37/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3754, in _quantile_ureduce_func
            [2018-08-18T00:45:21.894237Z] ap.partition(concatenate((indices_below, indices_above)), axis=axis)
            [2018-08-18T00:45:21.894239Z] TypeError: Partition index must be integer
            [2018-08-18T00:45:21.894241Z] ===================== 2 failed, 1 passed in 409.12 seconds =====================
            {code}

            The fix is to edit {{lsst.validate.drp.calcsrd.measurePA2}} to cast {{pf1Percentile}} to a {{float}} before calling {{np.percentile}}. I have built this without problem with numpy1.14.5, astropy3.0.3 and python 3.6.6.

            The problem is numpy 1.15. Using python 3.6.6, astropy 3.0.4 and numpy 1.15 also fails in the same way.
            tjenness Tim Jenness made changes -
            Summary Failure in lsst_ci with python3.7/numpy1.15/astropy3.0.4 Failure in lsst_ci with numpy1.15
            tjenness Tim Jenness made changes -
            Priority Undefined [ 10000 ] Major [ 3 ]
            jcarlin Jeffrey Carlin made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status To Do [ 10001 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            tjenness Tim Jenness made changes -
            Assignee Michael Wood-Vasey [ wmwood-vasey ] Jeffrey Carlin [ jcarlin ]
            jcarlin Jeffrey Carlin made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-16289 [ DM-16289 ]
            swinbank John Swinbank made changes -
            Team SQuaRE [ 10302 ] External [ 12117 ]

              People

              Assignee:
              jcarlin Jeffrey Carlin
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Tim Jenness
              Watchers:
              Jeffrey Carlin, Jonathan Sick, Joshua Hoblitt, Michael Wood-Vasey, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.