• [x ] I have checked that this issue has not already been reported.

  • [ x] I have confirmed this bug exists on the latest version of pandas.

  • [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

Pandas's dtype Int8Dtype is converted to object when melted.

Code Sample, a copy-pastable example

df = pd.DataFrame({"a": pd.Series([1, 2], dtype=pd.Int8Dtype())})
melted = df.melt()
print(melted.dtypes)

Problem description

When melted, the Int8 column is casted to object type.

Expected Output

melted.dtypes should return:

variable    object
value       Int8
dtype: object

Output of ``pd.show_versions()

INSTALLED VERSIONS ------------------ commit : b5958ee1999e9aead1938c0bba2b674378807b3d python : 3.7.0.final.0 python-bits : 64 OS : Linux OS-release : 4.15.0-143-generic Version : #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 1.1.5 numpy : 1.19.4 pytz : 2020.5 dateutil : 2.8.1 pip : 21.0.1 setuptools : 54.1.2 Cython : None pytest : 6.2.2 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.6.2 html5lib : None pymysql : None psycopg2 : None jinja2 : 2.11.2 IPython : 7.19.0 pandas_datareader: None bs4 : 4.9.3 bottleneck : 1.3.2 fsspec : None fastparquet : None gcsfs : None matplotlib : 3.3.3 numexpr : None odfpy : None openpyxl : 3.0.6 pandas_gbq : None pyarrow : None pytables : None pyxlsb : 1.0.8 s3fs : None scipy : 1.5.4 sqlalchemy : 1.3.22 tables : None tabulate : 0.8.7 xarray : None xlrd : 2.0.1 xlwt : None numba : 0.53.1

Comment From: mzeitlin11

Thanks for the clear report @etiennekintzler! Issue persists on master, investigations to fix are welcome! (This looks like an issue with other ExtensionDType like StringDType as well)

Comment From: etiennekintzler

Hi @mzeitlin11

After checking the function melt, it looks like the problem appears when the attribute DataFrame._values is accessed : https://github.com/pandas-dev/pandas/blob/34b0bdc5d315c5ebae582291da381c5f075de372/pandas/core/reshape/melt.py#L147

For example 

df_int8d = pd.DataFrame({"a": [1, 2]}, dtype=pd.Int8Dtype())
print(repr(df_int8d._values.dtype))

returns dtype('O') which doesn't seem right.

As expected pd.DataFrame({"a": [1, 2]})._values.dtype returns dtype('int64')

I don't know to what extent it is comparable with the attribute _values for Series but this issue doesn't appear for this class:

s = pd.Series(data=[1, 2], dtype=pd.Int8Dtype())
print(repr(s._values.dtype))

returns Int8Dtype() which is what we expect.

Issue persists on master, investigations to fix are welcome!

Would be glad to contribute however I am not very familiar with the pandas internals ; would you have any idea of which source file I could look into ? I've just found this https://github.com/pandas-dev/pandas/blob/v1.2.4/pandas/core/arrays/integer.py#L605-L608

Comment From: mzeitlin11

Not sure about the best way to approach fixing this. The problem is that for melt we essentially want to take data and flatten it. We flatten it by converting to a numpy array and usingravel - the conversion to numpy is the problem since ExtensionDType data will become object.

There's also the further issue that for a case like 

df = pd.DataFrame(
    {"a": pd.Series([1, 2], dtype=pd.Int8Dtype()),
     "b": pd.Series([1, 2], dtype=pd.Int16Dtype())}
)

keeping extension types would require invoking some find common type logic (maybe related to #22791)

So for fixing this issue it isn't limited to IntegerDType, so the file linked above probably won't be useful. It would be more about finding a different procedure for flattening the columns in a way which preserves types better (while still being performant). This sounds kind of like a concat, so I'd suggest looking into how something like pd.concat([df, df]) (using the example df from above) manages to maintain the type correctly.

Sorry I can't be of more help here - if you're looking to contribute you'd of course be welcome to look into this further (maybe someone with more experience with this part of the codebase has better suggestions for fixing this) but there are also tons of other issues which may be a more approachable starting point.