The test
FAILED pandas/tests/frame/test_ufunc.py::test_alignment_deprecation_many_inputs
is failing.
Also we have some typing problems
Performing static analysis using mypy
pandas/core/internals/ops.py:130: error: Unused "type: ignore" comment
pandas/core/dtypes/common.py:539: error: Unused "type: ignore" comment
pandas/core/dtypes/cast.py:1127: error: Unused "type: ignore" comment
pandas/core/dtypes/cast.py:1149: error: Unused "type: ignore" comment
pandas/core/missing.py:337: error: Argument 1 to "apply_along_axis" has incompatible type "Callable[[ndarray[Any, Any]], None]"; expected "Callable[..., Union[_SupportsArray[dtype[<nothing>]], Sequence[_SupportsArray[dtype[<nothing>]]], Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]]]]]" [arg-type]
pandas/core/missing.py:776: error: Argument 1 to "apply_along_axis" has incompatible type "partial[None]"; expected "Callable[..., Union[_SupportsArray[dtype[<nothing>]], Sequence[_SupportsArray[dtype[<nothing>]]], Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]]]]]" [arg-type]
pandas/core/missing.py:776: note: "partial[None].__call__" has type "Callable[[VarArg(Any), KwArg(Any)], None]"
pandas/core/missing.py:782: error: Argument 2 to "apply_along_axis" has incompatible type "Union[str, int]"; expected "SupportsIndex" [arg-type]
pandas/core/arraylike.py:271: error: Unused "type: ignore" comment
pandas/core/algorithms.py:772: error: Argument 2 to "append" has incompatible type "List[Union[str, float, Period, Timestamp, Timedelta, Any]]"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/algorithms.py:1071: error: Argument 1 to "__call__" of "_UFunc_Nin1_Nout1" has incompatible type "Optional[ndarray[Any, dtype[bool_]]]"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/strings/accessor.py:585: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[Any]]", variable has type "Series") [assignment]
pandas/core/indexes/base.py:611: error: Unused "type: ignore" comment
pandas/core/indexes/base.py:4527: error: Argument 3 to "putmask" has incompatible type "Union[ExtensionArray, ndarray[Any, Any]]"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/indexes/base.py:5029: error: Item "ExtensionArray" of "Union[ExtensionArray, ndarray[Any, Any]]" has no attribute "_ndarray" [union-attr]
pandas/core/indexes/base.py:5029: error: Item "ndarray[Any, Any]" of "Union[ExtensionArray, ndarray[Any, Any]]" has no attribute "_ndarray" [union-attr]
pandas/core/indexes/base.py:6520: error: Incompatible types in assignment (expression has type "ExtensionArray", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/core/indexes/base.py:6571: error: No overload variant of "insert" matches argument types "ndarray[Any, Any]", "int", "None" [call-overload]
pandas/core/indexes/base.py:6571: note: Possible overload variants:
pandas/core/indexes/base.py:6571: note: def [_SCT <: generic] insert(arr: Union[_SupportsArray[dtype[_SCT]], Sequence[_SupportsArray[dtype[_SCT]]], Sequence[Sequence[_SupportsArray[dtype[_SCT]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[_SCT]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[_SCT]]]]]]], obj: Union[slice, Union[_SupportsArray[dtype[Union[bool_, integer[Any]]]], _NestedSequence[_SupportsArray[dtype[Union[bool_, integer[Any]]]]], bool, int, _NestedSequence[Union[bool, int]]]], values: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], axis: Optional[SupportsIndex] = ...) -> ndarray[Any, dtype[_SCT]]
pandas/core/indexes/base.py:6571: note: def insert(arr: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], obj: Union[slice, Union[_SupportsArray[dtype[Union[bool_, integer[Any]]]], _NestedSequence[_SupportsArray[dtype[Union[bool_, integer[Any]]]]], bool, int, _NestedSequence[Union[bool, int]]]], values: Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]], axis: Optional[SupportsIndex] = ...) -> ndarray[Any, dtype[Any]]
pandas/core/arrays/masked.py:770: error: Need type annotation for "npvalues" [var-annotated]
pandas/core/arrays/masked.py:946: error: Argument 3 to "putmask" has incompatible type "object"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/arrays/masked.py:1022: error: Argument 3 to "putmask" has incompatible type "object"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/arrays/interval.py:1642: error: Argument 1 to "in1d" has incompatible type "Union[ExtensionArray, ndarray[Any, Any], ndarray[Any, dtype[Any]]]"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/arrays/sparse/array.py:775: error: Need type annotation for "new_values" [var-annotated]
pandas/core/arrays/sparse/array.py:927: error: Invalid index type "Tuple[Union[int, ellipsis], ...]" for "ndarray[Any, Any]"; expected type "Union[SupportsIndex, _SupportsArray[dtype[Union[bool_, integer[Any]]]], _NestedSequence[_SupportsArray[dtype[Union[bool_, integer[Any]]]]], _NestedSequence[Union[bool, int]], Tuple[Union[SupportsIndex, _SupportsArray[dtype[Union[bool_, integer[Any]]]], _NestedSequence[_SupportsArray[dtype[Union[bool_, integer[Any]]]]], _NestedSequence[Union[bool, int]]], ...]]" [index]
pandas/core/arrays/sparse/array.py:1167: error: Argument 2 to "IntIndex" has incompatible type "ndarray[Any, dtype[signedinteger[_32Bit]]]"; expected "Sequence[int]" [arg-type]
pandas/core/arrays/sparse/array.py:1345: error: Need type annotation for "sparse_values" [var-annotated]
pandas/core/indexes/multi.py:368: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[Any]]", variable has type "List[Any]") [assignment]
pandas/core/indexes/multi.py:1091: error: Argument 1 to "concatenate" has incompatible type "List[object]"; expected "Union[_SupportsArray[dtype[<nothing>]], Sequence[_SupportsArray[dtype[<nothing>]]], Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]]]]" [arg-type]
pandas/core/indexes/multi.py:1569: error: Argument 1 to "lexsort" has incompatible type "List[Union[ExtensionArray, ndarray[Any, Any]]]"; expected "Union[_SupportsArray[dtype[Any]], _NestedSequence[_SupportsArray[dtype[Any]]], bool, int, float, complex, str, bytes, _NestedSequence[Union[bool, int, float, complex, str, bytes]]]" [arg-type]
pandas/core/indexing.py:2018: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[signedinteger[Any]]]", variable has type "List[Any]") [assignment]
pandas/core/indexing.py:2080: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[Any]]", variable has type "Series") [assignment]
pandas/io/stata.py:602: error: Value of type variable "_IntType" of "iinfo" cannot be "object" [type-var]
pandas/io/stata.py:685: error: Need type annotation for "off" [var-annotated]
pandas/io/stata.py:687: error: Need type annotation for "val" [var-annotated]
pandas/io/stata.py:1698: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1700: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1701: error: "ndarray[Any, dtype[Any]]" has no attribute "columns" [attr-defined]
pandas/io/stata.py:1707: error: "ndarray[Any, dtype[Any]]" has no attribute "index" [attr-defined]
pandas/io/stata.py:1711: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1711: error: Argument 1 to "_do_select_columns" of "StataReader" has incompatible type "ndarray[Any, dtype[Any]]"; expected "DataFrame" [arg-type]
pandas/io/stata.py:1721: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1721: error: Argument 1 to "_insert_strls" of "StataReader" has incompatible type "ndarray[Any, dtype[Any]]"; expected "DataFrame" [arg-type]
pandas/io/stata.py:1725: error: "ndarray[Any, dtype[Any]]" has no attribute "index" [attr-defined]
pandas/io/stata.py:1730: error: "ndarray[Any, dtype[Any]]" has no attribute "columns" [attr-defined]
pandas/io/stata.py:1740: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1743: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1743: error: Argument 1 to "_do_convert_missing" of "StataReader" has incompatible type "ndarray[Any, dtype[Any]]"; expected "DataFrame" [arg-type]
pandas/io/stata.py:1752: error: "ndarray[Any, dtype[Any]]" has no attribute "columns" [attr-defined]
pandas/io/stata.py:1762: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1763: error: Argument 1 to "_do_convert_categoricals" of "StataReader" has incompatible type "ndarray[Any, dtype[Any]]"; expected "DataFrame" [arg-type]
pandas/io/stata.py:1783: error: Incompatible types in assignment (expression has type "DataFrame", variable has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/stata.py:1786: error: "ndarray[Any, dtype[Any]]" has no attribute "set_index" [attr-defined]
pandas/io/stata.py:1786: error: "ndarray[Any, dtype[Any]]" has no attribute "pop" [attr-defined]
pandas/io/stata.py:1788: error: Incompatible return value type (got "ndarray[Any, dtype[Any]]", expected "DataFrame") [return-value]
pandas/io/parsers/c_parser_wrapper.py:385: error: Unused "type: ignore" comment
pandas/io/parsers/c_parser_wrapper.py:402: error: Argument 1 to "concatenate" has incompatible type "List[Union[ExtensionArray, ndarray[Any, Any]]]"; expected "Union[_SupportsArray[dtype[Any]], Sequence[_SupportsArray[dtype[Any]]], Sequence[Sequence[_SupportsArray[dtype[Any]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[Any]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[Any]]]]]]]" [arg-type]
pandas/core/series.py:1983: error: Argument "minlength" to "bincount" has incompatible type "Optional[int]"; expected "SupportsIndex" [arg-type]
pandas/core/generic.py:6983: error: Incompatible types in assignment (expression has type "Index", variable has type "ndarray[Any, dtype[signedinteger[Any]]]") [assignment]
pandas/core/frame.py:2431: error: Argument 1 to "append" of "list" has incompatible type "Union[type, dtype[Any], str]"; expected "dtype[_SCT]" [arg-type]
pandas/core/window/rolling.py:359: error: Unused "type: ignore" comment
pandas/core/reshape/reshape.py:1028: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[signedinteger[Any]]]", variable has type "Index") [assignment]
pandas/core/reshape/reshape.py:1038: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[Any]]", variable has type "Index") [assignment]
pandas/core/reshape/melt.py:136: error: Incompatible types in assignment (expression has type "ndarray[Any, dtype[Any]]", variable has type "Series") [assignment]
pandas/core/internals/managers.py:1094: error: No overload variant of "__getitem__" of "tuple" matches argument type "ndarray[Any, dtype[signedinteger[Any]]]" [call-overload]
pandas/core/internals/managers.py:1094: note: Possible overload variants:
pandas/core/internals/managers.py:1094: note: def __getitem__(self, int) -> Block
pandas/core/internals/managers.py:1094: note: def __getitem__(self, slice) -> Tuple[Block, ...]
pandas/core/internals/managers.py:1103: error: Argument "blkno" to "_iset_single" of "BlockManager" has incompatible type "ndarray[Any, dtype[signedinteger[Any]]]"; expected "int" [arg-type]
pandas/core/internals/managers.py:1130: error: Incompatible types in assignment (expression has type "int", variable has type "ndarray[Any, dtype[signedinteger[Any]]]") [assignment]
pandas/core/internals/managers.py:1131: error: No overload variant of "__getitem__" of "tuple" matches argument type "ndarray[Any, dtype[signedinteger[Any]]]" [call-overload]
pandas/core/internals/managers.py:1131: note: Possible overload variants:
pandas/core/internals/managers.py:1131: note: def __getitem__(self, int) -> Block
pandas/core/internals/managers.py:1131: note: def __getitem__(self, slice) -> Tuple[Block, ...]
pandas/core/internals/managers.py:1156: error: Non-overlapping container check (element type: "int", container item type: "ndarray[Any, dtype[signedinteger[Any]]]") [comparison-overlap]
pandas/core/internals/blocks.py:376: error: Argument 1 to "delete" has incompatible type "Union[ndarray[Any, Any], ExtensionArray]"; expected "Union[_SupportsArray[dtype[Any]], Sequence[_SupportsArray[dtype[Any]]], Sequence[Sequence[_SupportsArray[dtype[Any]]]], Sequence[Sequence[Sequence[_SupportsArray[dtype[Any]]]]], Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[Any]]]]]]]" [arg-type]
pandas/core/internals/array_manager.py:1206: error: Need type annotation for "array" [var-annotated]
pandas/core/groupby/ops.py:163: error: Unused "type: ignore" comment
pandas/core/groupby/ops.py:813: error: Incompatible types in assignment (expression has type "List[<nothing>]", variable has type "ndarray[Any, dtype[signedinteger[Any]]]") [assignment]
pandas/io/sas/sas7bdat.py:802: error: Incompatible types in assignment (expression has type "Series", target has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/sas/sas7bdat.py:805: error: Incompatible types in assignment (expression has type "Series", target has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/sas/sas7bdat.py:805: error: Argument 1 to "_convert_datetimes" has incompatible type "ndarray[Any, dtype[Any]]"; expected "Series" [arg-type]
pandas/io/sas/sas7bdat.py:807: error: Incompatible types in assignment (expression has type "Series", target has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/sas/sas7bdat.py:807: error: Argument 1 to "_convert_datetimes" has incompatible type "ndarray[Any, dtype[Any]]"; expected "Series" [arg-type]
pandas/io/sas/sas7bdat.py:810: error: Incompatible types in assignment (expression has type "Series", target has type "ndarray[Any, dtype[Any]]") [assignment]
pandas/io/sas/sas7bdat.py:812: error: "ndarray[Any, dtype[Any]]" has no attribute "str" [attr-defined]
pandas/io/sas/sas7bdat.py:816: error: "ndarray[Any, dtype[Any]]" has no attribute "str" [attr-defined]
pandas/tests/extension/date/array.py:112: error: "object_" object is not iterable [misc]
Found 80 errors in 27 files (checked 1340 source files)
Comment From: seberg
In the other PR/issue, this:
E numpy.core._exceptions._UFuncNoLoopError: ufunc 'my_ufunc' did not contain a loop with signature matching types (<class 'numpy.dtype[int64]'>, <class 'numpy.dtype[int64]'>, <class 'numpy.dtype[int64]'>) -> None
was said to be the only "real" issue (not typing related). I think this is a duplicate of https://github.com/numpy/numpy/issues/20699#issuecomment-1004471654 probably, I will look into it (I guess there may be non-nativ byte-order involved, although maybe not necessarily always)
Comment From: attack68
fyi: dev-requirements.txt issues numba>=0.50.1, which resolves to the latest version: 0.54.1, which has the specific dependency of numpy<1.21.0, so any package resolver should yield numpy==1.20.3 for the dev environment, which doesn't run mypy without failures. I, at least, need numpy==1.21.0 to get mypy success on all files.
Comment From: padrepitufo
not sure why but versions of numpy below 1.22 are causing whitesource security vulnerabilities triggers to go off per https://www.whitesourcesoftware.com/vulnerability-database/CVE-2021-41496 . It would seem that 1.21.5 would solve it as the fix (https://github.com/numpy/numpy/pull/18989/commits/16f7824b4d935b6aee98298ca4123d57174a6f2e) shows it's present in 1.21.5
Comment From: seberg
@padrepitufo we have been getting nonsense CVEs for NumPy, this is one of them. I am pretty disappointed by these CVEs appearing out of no-where and being assigned ridiculous severity levels.
Comment From: phofl
I‘ll prepare a pr adressing the typing issues
Comment From: rhshadrach
For the NumPy issue, not sure if this will be fixed by https://github.com/numpy/numpy/issues/20735
Comment From: seberg
@rhshadrach there was just a new numpy release that includes a fix for some reductions (reducelikes) failing with a weird type resolution error. I would hope that fixes the issue here. The issue you linked to is very specific to numba's dynamic ufuncs, pandas should not care about it.
Please make sure to ping me if pandas still has any problems with the 1.22.1 release!
Comment From: rhshadrach
Thanks @seberg - I am still seeing this with 1.22.1. Here is the test that's failing:
https://github.com/pandas-dev/pandas/blob/96f2f2aff10666aa2b8118174a3f39a2e88a0bf4/pandas/tests/frame/test_ufunc.py#L277-L286
and the error message I'm seeing is
numpy.core._exceptions.UFuncTypeError: ufunc 'my_ufunc' did not contain a loop with signature matching types (<class 'numpy.dtype[int64]'>, <class 'numpy.dtype[int64]'>, <class 'numpy.dtype[int64]'>) -> None
on
>>> import numpy as np; np.__version__
'1.22.1'
Comment From: rhshadrach
I was previously running with numba 0.53.0; updating to 0.55.0 I'm no longer seeing a failure.
Comment From: seberg
Numba was broken for a long time for NumPy 1.21+ (they rely on the Python calling convention not changing, but Python optimized things by changing the calling convention, and I used that in NumPy :boom:).
So I expect if you were on old numba, it is just a numba bug. If you use a dynamic numba vectorization (the types are not specified ahead of time), then there is still a problem on 1.22.1. I am planning to fix it soon, but do not feel like rushing it considering that Numba support for NumPy 1.21.x support took 3 months or so.
Comment From: simonjayhawkins
moving to 1.4.1 and removing blocker label
Comment From: rhshadrach
Not sure what the right course of action here is - would we want to bump the minimum version of Numba? It would need to be bumped to a recent version, which seems undesirable. We could always just document this somewhere - a warning in the whatsnew as well as in the Numba section of the docs.
Comment From: simonjayhawkins
removing milestone.
Comment From: mroeschke
Looks like these have been addressed so closing