Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame({"x": ["foo", "bar", "baz"], "y": ["a", "b", "c"], "z": [1, 2, 3]})
df = df.astype({"x": "string[pyarrow]", "y": "string[python]", "z": "int64[pyarrow]"})
print(f"df.dtypes: \n{df.dtypes}")
print(f"{df.x.dtypes = }")
print(f"{df.y.dtypes = }")
Issue Description
The output of DataFrame.dtypes doesn't include [pyarrow] or [python] when using string extension types. This makes it unclear what dtypes are in a DataFrame. It's also not consistent with Series.dtypes, which includes [pyarrow] / [python] information, and other pyarrow-backed dtypes like int64[pyarrow]
The above code snippet outputs
df.dtypes:
x string
y string
z int64[pyarrow]
dtype: object
df.x.dtypes = string[pyarrow]
df.y.dtypes = string[python]
Expected Behavior
I'd expect string[pyarrow] and string[python] to be in output of DataFrame.dtypes
Installed Versions
INSTALLED VERSIONS
------------------
commit : 8dab54d6573f7186ff0c3b6364d5e4dd635ff3e7
python : 3.9.15.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.5.2
numpy : 1.21.6
pytz : 2022.6
dateutil : 2.8.2
setuptools : 59.8.0
pip : 22.3.1
Cython : None
pytest : 7.2.0
hypothesis : None
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.7.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : 2022.11.0
fsspec : 2022.11.0
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : 2.8.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : 2022.11.0
scipy : 1.9.3
snappy :
sqlalchemy : 1.4.44
tables : 3.7.0
tabulate : None
xarray : 2022.11.0
xlrd : None
xlwt : None
zstandard : None
tzdata : None
Comment From: mroeschke
I think the Series repr uses __str__ of each element which is why this may be happening
In [6]: str(df.x.dtype)
Out[6]: 'string'
In [7]: repr(df.x.dtype)
Out[7]: 'string[pyarrow]'
Comment From: jorisvandenbossche
I would like us to reconsider the change made in #50151. This difference between the repr and str was actually done intentionally AFAIR (although I can't directly find where we discussed this).
The rationale is that most people shouldn't need to be aware of this (what does "string[python]" mean?). You just have columns that are "string" dtype, and the detail about how this is stored under the hood is only something that advanced users need to know. We should still make this information easily available, but IMO this doesn't need to be in the default .dtypes view.
Comment From: jrbourbeau
Thanks @jorisvandenbossche, I see where you're coming from. In that case, maybe we go back to string for string[python] but keep string[pyarrow] when using pyarrow-backed strings? That would keep the numpy-backed extension dtypes simple but still allow users to differentiate between string[pyarrow] and string[python]. That would also be consistent with all the other pyarrow-backed dtypes like int64[pyarrow], float64[pyarrow], etc. which are already including the [pyarrow] portion in DataFrame.dtypes.