Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame({"x": [1, 2, 3], "y": ["foo", "bar", "baz"]})
df_empty = df.iloc[:0]
print(f"{df.convert_dtypes().dtypes = }") # works as expected (i.e. dtype is `string[python]`)
print(f"{df_empty.convert_dtypes().dtypes = }") # `y` has dtype `object` even though `convert_string=True`
with pd.option_context("mode.dtype_backend", "pyarrow"):
print(f"{df.convert_dtypes().dtypes = }") # works as expected (i.e. dtype is `string[pyarrow]`)
print(f"{df_empty.convert_dtypes().dtypes = }") # This errors with `pyarrow.lib.ArrowNotImplementedError: Unsupported numpy type 17`
Issue Description
When calling DataFrame.convert_dtypes() on an empty DataFrame I noticed that issues with handling object dtypes with both the pandas and pyarrow dtype backends.
When using the pandas dtype backend, object columns aren't converted to string[python], they instead remain object (even though convert_string=True). When using the pyarrow backend, we get the following error:
Traceback (most recent call last):
File "/Users/james/projects/dask/dask/test-convert-dtypes.py", line 11, in <module>
print(f"{df_empty.convert_dtypes().dtypes = }") # `y` has dtype `object` even though `convert_string=True`
File "/Users/james/projects/pandas-dev/pandas/pandas/core/generic.py", line 6660, in convert_dtypes
results = [
File "/Users/james/projects/pandas-dev/pandas/pandas/core/generic.py", line 6661, in <listcomp>
col._convert_dtypes(
File "/Users/james/projects/pandas-dev/pandas/pandas/core/series.py", line 5469, in _convert_dtypes
inferred_dtype = convert_dtypes(
File "/Users/james/projects/pandas-dev/pandas/pandas/core/dtypes/cast.py", line 1124, in convert_dtypes
pa_type = to_pyarrow_type(base_dtype)
File "/Users/james/projects/pandas-dev/pandas/pandas/core/arrays/arrow/array.py", line 145, in to_pyarrow_type
pa_dtype = pa.from_numpy_dtype(dtype)
File "pyarrow/types.pxi", line 3330, in pyarrow.lib.from_numpy_dtype
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported numpy type 17
cc @mroeschke @phofl for visibility
Expected Behavior
I would expect convert_dtypes to behave the same, even if the DataFrame is empty.
Installed Versions
INSTALLED VERSIONS
------------------
commit : 7f2aa8f46a4a36937b1be8ec2498c4ff2dd4cf34
python : 3.10.4.final.0
python-bits : 64
OS : Darwin
OS-release : 22.2.0
Version : Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.0.0.dev0+1309.g7f2aa8f46a
numpy : 1.24.1
pytz : 2022.1
dateutil : 2.8.2
setuptools : 59.8.0
pip : 22.0.4
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.2.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : 2022.12.1.dev6
fsspec : 2023.1.0+5.g012816b
gcsfs : None
matplotlib : 3.5.1
numba : None
numexpr : 2.8.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 11.0.0.dev316
pyreadstat : None
pyxlsb : None
s3fs : 2022.10.0
scipy : 1.9.0
snappy :
sqlalchemy : 1.4.35
tables : 3.7.0
tabulate : None
xarray : 2022.3.0
xlrd : None
zstandard : None
tzdata : None
qtpy : None
pyqt5 : None
Comment From: mroeschke
Since the default of infer_objects=True in convert_dtypes
For object-dtyped columns, if infer_objects is True, use the inference rules as during normal Series/DataFrame construction. Then, if possible, convert to StringDtype, BooleanDtype or an appropriate integer or floating extension type, otherwise leave as object.
So since the data is inferred to be empty of df_empty, I think the result should be object. I don't think it's possible to have "prior" knowledge here that it used to be string-object like data.
Comment From: phofl
I agree in general, but this creates problems in other areas (concat comes to mind).
I guess we need something like a null dtype, that is ignored in all operations if we try to find a common dtype?