Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of pandas.

  • [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [1]: pdf = pd.DataFrame({"idx": [0, 1, 2], "A": [1, 2, 3]}).set_index('idx')
[PYFLYBY] import pandas as pd

In [2]: pdf
Out[2]:
     A
idx
0    1
1    2
2    3

In [3]: pdf.to_parquet("~/tmp/test.parquet")

In [4]: with pd.option_context("mode.dtype_backend", "pyarrow"):
   ...:     df = pd.read_parquet("~/tmp/test.parquet")
   ...:

In [5]: df
Out[5]:
   A  idx
0  1    0
1  2    1
2  3    2

In [6]: pd.__version__
Out[6]: '2.1.0.dev0+93.g6bb8f73e75'

Issue Description

Roundtripping a DataFrame with an index to parquet and back under the options mode mode.type_backend='pyarrow' causes indexes to be dropped.

Expected Behavior

I think pd.read_parquet should read from the schema (in particular pandas_metadata['index_columns'] to set the index. I don't think this is a bug in a lower level library like pyarrow since I think (?) indexes are a Pandas-specific abstraction. I assume there's some reason that setting the option mode skips pandas_compat._reconstruct_index, but I can't figure it out.

This is what I expect to see instead:

In [16]: df = df.set_index('idx')

In [17]: df.info()
<class 'pandas.core.frame.DataFrame'>
Index: 3 entries, 0 to 2
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   A       3 non-null      int64[pyarrow]
dtypes: int64[pyarrow](1)
memory usage: 50.0 bytes

In [18]: df.index
Out[18]: Index([0, 1, 2], dtype='int64[pyarrow]', name='idx')

Thanks for the help!

Installed Versions

INSTALLED VERSIONS ------------------ commit : 6bb8f73e75bb5d25ca752f34f54705cdc533da54 python : 3.10.4.final.0 python-bits : 64 OS : Linux OS-release : 5.15.90.1-microsoft-standard-WSL2 Version : #1 SMP Fri Jan 27 02:56:13 UTC 2023 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.1.0.dev0+93.g6bb8f73e75 numpy : 1.23.3 pytz : 2022.2.1 dateutil : 2.8.2 setuptools : 58.1.0 pip : 22.3 Cython : 0.29.32 pytest : 7.1.3 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.1 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.5.0 pandas_datareader: 0.10.0 bs4 : 4.11.1 bottleneck : 1.3.6 brotli : None fastparquet : 0.8.3 fsspec : 2022.8.2 gcsfs : None matplotlib : 3.5.3 numba : 0.56.4 numexpr : 2.8.4 odfpy : None openpyxl : 3.0.10 pandas_gbq : None pyarrow : 9.0.0 pyreadstat : None pyxlsb : None s3fs : 2022.8.2 scipy : 1.9.1 snappy : None sqlalchemy : 1.4.41 tables : 3.8.0 tabulate : None xarray : 2023.2.0 xlrd : 2.0.1 zstandard : 0.18.0 tzdata : None qtpy : 2.3.0 pyqt5 : None

Comment From: phofl

Hi, this will be fixed with pyarrow 12.0, so I don't think we should do anything here.

Comment From: mroeschke

Agreed, closing as an upstream issue