Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
>>> df = pd.DataFrame({'a': [True, True, False, False], 'b': [True, False, True, False], 'c': [1,2,3,4]})
>>> df
a b c
0 True True 1
1 True False 2
2 False True 3
3 False False 4
>>> df.set_index(['a']).loc[True]
b c
a
True True 1
True False 2
>>> df.set_index(['a', 'b']).loc[True]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kale/hacking/forks/pandas/pandas/core/indexing.py", line 1071, in __getitem__
return self._getitem_axis(maybe_callable, axis=axis)
File "/home/kale/hacking/forks/pandas/pandas/core/indexing.py", line 1306, in _getitem_axis
self._validate_key(key, axis)
File "/home/kale/hacking/forks/pandas/pandas/core/indexing.py", line 1116, in _validate_key
raise KeyError(
KeyError: 'True: boolean label can not be used without a boolean index'
Issue Description
The above snippet creates a data frame with two boolean columns. When just one of those columns is set as the index, .loc[] can select rows based on a boolean value. When both columns are set as the index, though, the same value causes .loc[] to raise a KeyError. The error message makes it seems like pandas doesn't regard a multi-index as a "boolean index", even if the columns making up the index are all boolean.
In pandas 1.4.3, I can work around this by indexing with 1 and 0 instead of True and False. But in 1.5.0 it seems to be an error to provide a integer value to a boolean index.
Expected Behavior
I would expect to be able to use boolean index values even when those values are part of a multi-index:
>>> df.set_index(['a', 'b']).loc[True]
c
b
True 1
False 2
Installed Versions
Comment From: dicristina
As per the user guide, in the MultiIndex case the notation .loc[True] is shorthand for .loc[(True,)]. As a workaround use the tuple notation instead:
import pandas as pd
df = pd.DataFrame({'a': [True, True, False, False], 'b': [True, False, True, False], 'c': [1,2,3,4]})
df.set_index(['a', 'b']).loc[(True,)]
This gives the expected result in 1.4.3:
>>> df.set_index(['a', 'b']).loc[(True,)]
<stdin>:1: PerformanceWarning: indexing past lexsort depth may impact performance.
c
b
True 1
False 2
Comment From: kalekundert
Ah, ok, I suppose I shouldn't be surprised that integrating .loc[] with multi-indexing is complicated and that there are edge cases like this. Thanks for providing a work-around, and feel free to close the issue if this isn't something you think is worth fixing.
Comment From: phofl
This should work. Thanks for the report.
Comment From: Zahlii
Piggy-backing on this, the following simple scenario also fails:
import pandas as pd
x = pd.DataFrame({True: [1,2,3]})
print(x)
print(x.loc[:, True])
# KeyError: 'True: boolean label can not be used without a boolean index'
This is because validate key doesn't use the axis argument.
Comment From: batterseapower
Probably obvious, but this also doesn't work with when you try to select multiple boolean items i.e.:
df = pd.DataFrame({'a': [True, True, False, False], 'b': [True, False, True, False], 'c': [1,2,3,4]}).set_index(['a', 'b'])
df.loc[[True]]
Fails with:
IndexError: Boolean index has wrong length: 1 instead of 4
Even more surprisingly, this also fails when the thing you are indexing with is explicitly an Index -- where it feels like the intention is clearly to select by key rather than to mask the array:
df.loc[pd.Index([True])]
You can work around this by using Index.get_indexer_non_unique directly:
indexer, _missing = pd.Index([True]).get_indexer_non_unique(df.index.get_level_values(0))
df.iloc[indexer >= 0]
Comment From: phofl
@batterseapower could you open another issue about this? this is different than the op, not sure how I am leaning there yet