Code Sample, a copy-pastable example if possible
import pandas as pd
s1 = pd.Series(data = [1,2,3,4,5], index = [1,2,3,4,4])
s2 = pd.Series(data = [1,2,3,4,5], index = ['1', '2', '3', '4', '4'])
s1[0] # KeyError
s2[0] # KeyError
s1.iloc[:4][0] # KeyError
s1.iloc[:5][0] # KeyError
s2.iloc[:4][0] # Returns 1!
s2.iloc[:5][0] # TypeError & KeyError
Problem description
s2[0]
throws a KeyError because the key, 0, is not in the index. This behavior is expected. s2.iloc[:5][0]
also has this correct behavior. However, once the duplicate index has been removed, s2.iloc[:4][0]
, this error is not thrown and instead the value at position 0 is returned.
This error is present with string type indexes, but does not occur with integer indexes (see examples involving s1
above). I have not tried other index types.
Issue is present in a fresh conda environment with python version 3.6.2 and pandas version 0.21.1 (via conda-forge).
Expected Output
s2.iloc[:4][0]
should fail with TypeError and KeyError.
Output of pd.show_versions()
Comment From: chris-b1
The fundamental problem here is #9595, namely []
indexing sometimes has fallback behavior to location based indexing. My general recommendation (and I think at least soft community consensus) is to only use []
indexing for DataFrame column selection and iloc
/ loc
everywhere else.
@jorisvandenbossche's summary does a good job, although doesn't have this particular corner (duplicate vs non-duplicate)
Comment From: jreback
indexing with duplicated indices is quite tricky. you have to pay attention, so pretty much you shouldn't avoid this. coupled with fallback things get tricky. use .loc
for label selection and this problem goes away.