Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of pandas.

  • [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({"a": [0, 1]})
df2 = pd.DataFrame({"a": [2, 3]})

print(df.dtypes)
# a    int64
# dtype: object
print(df2.dtypes)
# a    int64
# dtype: object

df3 = pd.concat([df, df2], axis=0)
print(df3.dtypes)
# a    int64
# dtype: object

df = df.astype('category')
df2 = df2.astype('category')

print(df.dtypes)
# a    category
# dtype: object
print(df2.dtypes)
# a    category
# dtype: object

df3 = pd.concat([df, df2], axis=0)
print(df3.dtypes)
a    int64
dtype: object

Issue Description

pd.concat doesn't preserve categorical dtype if the dfs have categorical columns, whereas pd.concat preserves int dtype.

Expected Behavior

The result dtype should be categorical, shouldn't it?

Installed Versions

pandas : 1.5.3 numpy : 1.24.1 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 66.1.1 pip : 22.3.1 Cython : None pytest : 7.2.1 hypothesis : None sphinx : 6.1.3 blosc : None feather : 0.4.1 xlsxwriter : None lxml.etree : 4.9.2 html5lib : None pymysql : None psycopg2 : 2.9.3 jinja2 : 3.1.2 IPython : 8.8.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : None brotli : fastparquet : 2022.12.0 fsspec : 2023.1.0 gcsfs : None matplotlib : 3.6.3 numba : None numexpr : 2.8.3 odfpy : None openpyxl : 3.0.10 pandas_gbq : 0.15.0 pyarrow : 10.0.1 pyreadstat : None pyxlsb : None s3fs : 2023.1.0 scipy : 1.10.0 snappy : None sqlalchemy : 1.4.46 tables : 3.7.0 tabulate : None xarray : 2023.1.0 xlrd : 2.0.1 xlwt : None zstandard : None tzdata : None

Comment From: jbrockmendel

The dtypes are both categorical, but not matching categoricals. With different dtypes, you get casting to a common dtype, in this case int64. (A reasonable case could be made for casting to a union_categorical dtype, i think there are issues about this). If you cast to a common dtype first, you retain the categorical-ness.

df['a'] = df['a'].cat.set_categories([0, 1, 2, 3])
df2['a'] = df2['a'].cat.set_categories([0, 1, 2, 3])

df3 = pd.concat([df, df2], axis=0)

>>> df3.dtypes
a    category
dtype: object

Comment From: ddrinka

I just ran into this as well. I expected concatenating two dataframes with different sets of categorical values to result in a new dataframe with a common set of categorical values. Instead the column was converted to a string.

Here's some history: https://github.com/pandas-dev/pandas/issues/25412

Comment From: YarShev

A reasonable case could be made for casting to a union_categorical dtype

I would like this to get fixed in 1.5.4.

Comment From: jbrockmendel

xref #42840

Comment From: jbrockmendel

Looks like #14177 is the API discussion. Mostly seems in favor.

Comment From: phofl

I would like this to get fixed in 1.5.4.

While I'd be in favour of combining the categories, this does not qualify as a regression fix and hence is out of scope for 1.5.4

Comment From: YarShev

This should probably be included in 1.6.0 then.

Comment From: jbrockmendel

There is no 1.6. 2.0rc is expected this week

Comment From: jbrockmendel

Discussed this on the dev call today, agreed to add a keyword to pd.concat to enable this behavior.

Comment From: YarShev

Cool! Will that feature be included in 2.0?

Comment From: jbrockmendel

no, a 2.0rc has already been released. it'll get into 2.1 if someone implements it

Comment From: YarShev

Good, thanks! Hope someone will pick this up to add into 2.1.