Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
# This gives the warning message: "pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead."
test_index_warning_thrown = pd.Int64Index([1,2,3])
# Creates a dataframe with a column that gets the default type, that is Int64Index
a=pd.DataFrame([20, 30, 15, 2], index=[0, 1, 2, 3], columns=[1])
# Creates a dataframe that gets the explicit Index type.
b=pd.DataFrame([20, 30, 15, 2], index=[0, 1, 2, 3], columns=pd.Index([1], dtype=pd.Int64Dtype()))
# This throws an error that does not seem reasonable. Something with slicing? See below.
a-b
Issue Description
Relates to: https://github.com/pandas-dev/pandas/pull/49560 .
- When we create explicit pd.Int64Index we get deprecation warnings. When we just set the columns from a list we still get Int64Index, however.
- When we explicity create pd.Index with dtype set to int64, to avoid the deprecation warning, we instead get the situation that we can't perform operations with other dataframes were pd.Int64Index is used.
Actual result is an error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File /opt/conda/lib/python3.9/site-packages/pandas/core/indexing.py:873, in _LocationIndexer._validate_tuple_indexer(self, key)
872 try:
--> 873 self._validate_key(k, i)
874 except ValueError as err:
File /opt/conda/lib/python3.9/site-packages/pandas/core/indexing.py:1483, in _iLocIndexer._validate_key(self, key, axis)
1482 else:
-> 1483 raise ValueError(f"Can only index by location with a [{self._valid_types}]")
ValueError: Can only index by location with a [integer, integer slice (START point is INCLUDED, END point is EXCLUDED), listlike of integers, boolean array]
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
Cell In [9], line 1
----> 1 a-b
Expected Behavior
Add, subtract, div etc. should work as normal, the types seem compatible. The docs tell you to create pd.Index instances, and these are all ints.
Installed Versions
Comment From: topper-123
Hi @MikaelUmaN. Thanks for the error report.
I can't reproduce errors or a deprecation warning from the example you posted. Can you give a more explicit example that fails in the decribed way, so I can look at it.
More generally, Int64Index is a legacy index type and only work using numpy integer dtypes, while Int64Dtype is a pandas dtype (nullable integer type), that is not supposed to work with Int64Index, but with Index only. Of course, this is not to say there isn't a bug in there, but I do need a reproducible example, so I can take a look at it.
Comment From: MikaelUmaN
Hey @topper-123 . Thank you for the answer.
I am aware it is a legacy type. The problem is that when I try to migrate away from pd.Int64Index I get a bunch of failed calculations because pd.Int64Index seems to be the default index used when e.g. just passing ints as columns.
That is partly an inconvenience. I tried amending my original post a bit to clarify what I think is strange here. Essentially I think the deprecation notice encourages you to use a type that is then not compatible with the default type used by pandas itself, even though the columns are exactly the same. Somehow, it does not seem logical.
I've reverted my code locally to only use pd.Int64Index because that is what works with legacy code.
Comment From: topper-123
dtype Int64Dtype is not the same as dtype int64, and they have and are expected to have different behavior. Your example (a - b) is probably a bug though (or a feature that hasn't been developed yet, as Int64Dtype in indexes is quite new).
Please notice that while Int64Index is deprecated, the dtype int64 is not and will continue to be available. If you want to keep the old behavior, I recommend to do: pd.Index(data, dtype="int64"). This keeps everything working the same way as it always has (and if not, we're open to bug reports about that).
I'll change the title of this issue to highlight the ops issue. The deprecation is to guide users to the newer recommendedc way to instantiate using int64 (i.e. pd.Index(data, dtype="int64") and should to stay to minimize migration issues when pandas 2.0 gets released.
Comment From: MikaelUmaN
Thanks. Yes that is my main concern; sorry for the confusion.
Comment From: jbrockmendel
The subtraction here now works as expected, closing.