Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[ ] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
values = {
False :
{'A' : range(2), 'B' : range(2)},
True :
{'A' : range(2), 'B' : range(2)}
}
values = pd.concat({k : pd.DataFrame(values[k]) for k in values})
values.loc[False]
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-40-cca3b469b9a2> in <module>
----> 1 values.loc[False]
~/.local/lib/python3.9/site-packages/pandas/core/indexing.py in __getitem__(self, key)
965
966 maybe_callable = com.apply_if_callable(key, self.obj)
--> 967 return self._getitem_axis(maybe_callable, axis=axis)
968
969 def _is_scalar_access(self, key: tuple):
~/.local/lib/python3.9/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
1199
1200 # fall thru to straight lookup
-> 1201 self._validate_key(key, axis)
1202 return self._get_label(key, axis=axis)
1203
~/.local/lib/python3.9/site-packages/pandas/core/indexing.py in _validate_key(self, key, axis)
1009 or self.obj._get_axis(axis).dtype.name == "boolean"
1010 ):
-> 1011 raise KeyError(
1012 f"{key}: boolean label can not be used without a boolean index"
1013 )
KeyError: 'False: boolean label can not be used without a boolean index'
Issue Description
Issue https://github.com/pandas-dev/pandas/issues/47687 was fixed in https://github.com/pandas-dev/pandas/pull/49766 by checking that if the index is a MultiIndex its first level must be of type bool.
However, this still seems inconsistent with other dtypes, and code paths - for instance values.loc[(False, ), :] works flawlessly with the example I am providing above.
I argue that the original sin is in
https://github.com/pandas-dev/pandas/pull/40726/files#diff-07282871a235456baecd3a55a43339b0da374ca270228b5530f5449aabdca2e1R968
That PR was meant to solve a specific issue, that is, that in some cases a boolean label is casted to int, and from there a series of unexpected behaviors occur.
But there is no reason on earth why Boolean labels shouldn't just follow the rules for labels of other dtypes, we just want to ensure that any undesired casting is avoided across any code path.
By the way, I checked that neither of the two problematic examples in https://github.com/pandas-dev/pandas/issues/20432 work any more, even removing all code from _LocIndexer._validate_key().
So I think we should do it.
Thoughts @mroeschke ?
Expected Behavior
Same as values.loc[(False, ), :]
Installed Versions
Comment From: mroeschke
This seems to work on main (e3143f6119c4745472d8291f72fc06121e72c699)?
In [1]: values = {
...: False :
...: {'A' : range(2), 'B' : range(2)},
...: True :
...: {'A' : range(2), 'B' : range(2)}
...: }
...:
...: values = pd.concat({k : pd.DataFrame(values[k]) for k in values})
...:
...: values.loc[False]
Out[1]:
A B
0 0 0
1 1 1
In [2]: values.loc[(False, ), :]
Out[2]:
A B
0 0 0
1 1 1
In [3]: pd.__version__
Out[3]: '2.0.0.dev0+895.ge3143f6119'
I guess there should be a test where the other MultiIndex level isn't bool necessarily?
Comment From: toobaz
This seems to work on main (e3143f6)?
Wow, that was quick!
... except that what changed is the DataFrame index level data type - not the bug itself, which (as of 5ee4dac ) still emerges with:
In [2]: values = {
...: False :
...: {'A' : range(2), 'B' : range(2)},
...: 42 :
...: {'A' : range(2), 'B' : range(2)}
...: }
...:
...: values = pd.concat({k : pd.DataFrame(values[k]) for k in values})
...:
...: values.loc[False]
So I still think getting rid of the content of _LocIndexer._validate_key() is the way to go.
Comment From: mroeschke
Ah okay yeah I see the bug still exists when the inferred dtype of the object column is more strictly mixed