Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame([[-1, 2], [3, -4]], index=list("ab"), columns=["one", "two"])
expected = pd.DataFrame([[0, 2], [3, 0]], index=list("ab"), columns=["one", "two"])
# Should be the same as (df < 0) but with boolean dtype
boolean_indexer = pd.DataFrame(
{"one": pd.Series([True, False], dtype="boolean", index=list("ab")),
"two": pd.Series([False, True], dtype="boolean", index=list("ab"))}
)
result = df.copy()
print((df < 0).dtypes)
result[df < 0] = 0
print(result == expected)
one bool
two bool
dtype: object
one two
a True True
b True True
result2 = df.copy()
print(boolean_indexer.dtypes)
result2[boolean_indexer] = 0
print(result2 == expected)
one boolean
two boolean
dtype: object
Traceback (most recent call last):
File "/opt/anaconda3/envs/pandas-dev/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3397, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-5-f53c306060b3>", line 3, in <cell line: 3>
result2[boolean_indexer] = 0
File "/Users/marius/Documents/GitHub/pandas/pandas/core/frame.py", line 3724, in __setitem__
self._setitem_frame(key, value)
File "/Users/marius/Documents/GitHub/pandas/pandas/core/frame.py", line 3841, in _setitem_frame
raise TypeError(
TypeError: Must pass DataFrame or 2-d ndarray with boolean values only
Issue Description
Trying to assign to a dataframe, indexing with a boolean dataframe, fails if the boolean dataframe has the nullable boolean dtype rather than the bool dtype.
It looks like this happens partly because is_bool_dtype() returns False for the boolean dataframe - either that check should be changed in _setitem_frame, or is_bool_dtype() should return True?
Expected Behavior
I would still expect boolean indexing to work with nullable booleans, when I came across this issue I was not deliberately trying to use the boolean dtype, it was just the output from a logical check I did. If logical operations can give either bool or boolean output, both outputs should behave the same way when used for indexing.
Installed Versions
Comment From: marius-mather
An example test case:
def test_dataframe_boolean_indexing_assign(self):
df = DataFrame([[-1, 2], [3, -4]], index=list("ab"), columns=["one", "two"])
expected = DataFrame([[0, 2], [3, 0]], index=list("ab"), columns=["one", "two"])
boolean_indexer = DataFrame(
{"one": pd.Series([True, False], dtype="boolean", index=list("ab")),
"two": pd.Series([False, True], dtype="boolean", index=list("ab"))}
)
result = df.copy()
result[df < 0] = 0
tm.assert_frame_equal(result, expected)
result2 = df.copy()
result2[boolean_indexer] = 0
tm.assert_frame_equal(result2, expected)
Comment From: simonjayhawkins
Thanks @marius-mather for the report.
I would still expect boolean indexing to work with nullable booleans
The pd.NA type is still experimental. If using nullable boolean without any nulls then the behavior should be the same as nullable boolean with nulls as behavior should not be value dependent.
so if pd.NA (nulls in a boolean array) is not treated as False, then the boolean indexing behavior of pd.NA is undefined. and therefore boolean indexing using a nullable boolean without nulls should probably match that.
so i'm not so sure that TypeError: Must pass DataFrame or 2-d ndarray with boolean values only is not correct here.
however, in the spirit of dynamically typed Python and user convenience, then a boolean EA dataframe without nulls could equally be allowed input for indexing.