Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
>>> df = pd.DataFrame([[0,1,2,3,4],[5,6,7,8,9]])
>>> df.iloc[:, pd.Series([0,1,2,3], dtype = pd.Int64Dtype())]
IndexError: .iloc requires numeric indexers, got [0 1 2 3]
# From stack trace
# 1475 # check that the key has a numeric dtype
# 1476 if not is_numeric_dtype(arr.dtype):
# -> 1477 raise IndexError(f".iloc requires numeric indexers, got {arr}")
# 1479 # check that the key does not exceed the maximum size of the index
# 1480 if len(arr) and (arr.max() >= len_axis or arr.min() < -len_axis):
>>> df.iloc[:, pd.Series([0,1,2,3])]
# Works
# Check dtypes. Note int64 vs Int64Dtype
>>> pd.Series([0,1,2,3]).dtype
dtype('int64')
>>> pd.Series([0,1,2,3], dtype = pd.Int64Dtype()).dtype
Int64Dtype()
# Series.iloc works
pd.Series([1,2,3,4,5,6]).iloc[pd.Series([0,1,2,3], dtype = pd.Int64Dtype())]
Issue Description
Pandas.DataFrame.iloc doesn't understand that Pandas pd.Int64Dtype is numeric, and refuses to index with it. Pandas.Series.iloc works.
Expected Behavior
That pd.DataFrame.iloc would recognise that pd.Int64Dtype is numeric and correctly index with it.
Installed Versions
Comment From: mzeitlin11
Thanks for reporting this @anbjork! Looks like the same issue as #39133?
Comment From: phofl
I think this is a bit different. The op should work already
Comment From: topper-123
.iloconly accepts integers, but IntegerArray can hold pd.NA and supplying pd.NA to .iloc is supposed to raise.
>>> import pandas as pd
>>> df = pd.DataFrame([[0,1,2,3,4],[5,6,7,8,9]])
>>> iarr = pd.array([0,1,2,pd.NA], dtype = pd.Int64Dtype())
>>> df.iloc[:, pd.Series(iarr)]
IndexError: ...
>>> df.iloc[:, iarr)
IndexError: ...
So IntegerArray cannot be used inplace for np.ndarray with an int dtype currently. Also notice that iarr._data is array([1, 2, 1] which would definitely give a wrong result.
Also notice that the docs don't mention integer indexing with IntegerArray, thought it insinuates that is does here. The docs should be explicit about if they accept IntegerArray in .iloc and, if accepted, how it deals with arrays with pd.NA values.
I lean toward not allowing integer indexing with IntegerArray and that the current behavior is correct, but the error message and docs should be improved. Maybe someone could make an explicit proposal where it's decided how pd.NA should be treated in .iloc operations to see if that makes sense.
Comment From: phofl
No, as long as numpy dtype Series are supposed to work, the ea series should work as well as long as no missing values are present. "Int64" without missing values are integers as well, so no reason to disallow. This gets more important since we are able to return nullable dtypes from I/O where everything is nullable, not only if nulls are present.
Btw using Int64 as row indexer worked already.
Comment From: topper-123
Ok, Didn't know that. Maybe it could be more explicit in the docs, for example here?
Comment From: phofl
Can you make a suggestion? Personally, the sentence
Operations involving an integer array will behave similar to NumPy arrays
tells me, that I can use both interchangeable as long as no missing values are present
Comment From: topper-123
Yeah, to me e.g. this is unclear: "...and the data will be coerced to another dtype if needed." In the case of .iloc indexing, it was not clear to me what coerced meant (or could mean).
I think a warning at the end of this paragraph about using Int64 for .iloc indexing, given that it's up to the user to ensure the array doesn't have a pd.Na?