Problem description
Pandas float formatters ignore the nullable floating extension types Float64 which was introduced with 1.2.0.
In particular, this is extremely confusing when using DataFrame.convert_dtypes since the default is now with convert_floating=True which converts all float colums to Float64 columns.
That is, the following does not produce the expected float formatting:
import pandas as pd
df = pd.DataFrame([0.123456789, 1.123456789, 2.123456789], columns=["value"])
df = df.convert_dtypes()
print(df.to_string(formatters=["{:.2f}".format]))
gives
value
0 0.123457
1 1.123457
2 2.123457
I am not sure, whether this classifies as a bug or missing documentation (or both) however having not heard of this new nullable floating type Float64 I did not notice this subtle difference at first. Also, I did not expect DataFrame.convert_dtypes() to result in a non-formatable DataFrame. So, if Float64 is not expected to format as float64, the documentation should make this clear.
However, I believe the best would be that Float64 formats just as the regular float dtype.
Expected Output
The formatting should be applied to the Float64 dtype just as when using the float dtype such as
import pandas as pd
df = pd.DataFrame([0.123456789, 1.123456789, 2.123456789], columns=["value"])
print(df.to_string(formatters=["{:.2f}".format]))
which gives
value
0 0.12
1 1.12
2 2.12
Output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit : 3e89b4c4b1580aa890023fc550774e63d499da25
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.9-arch1-1
Version : #1 SMP PREEMPT Tue, 19 Jan 2021 22:06:06 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_DK.UTF-8
LOCALE : en_DK.UTF-8
pandas : 1.2.0
numpy : 1.19.2
pytz : 2020.5
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
Comment From: simonjayhawkins
Thanks @MaxSchambach for the report
the new experimental FloatingArray, introduced in 1.2 https://pandas.pydata.org/docs/whatsnew/v1.2.0.html#experimental-nullable-data-types-for-float-data and still some work to be done #38110 (including Honor the formatting options (eg precision))
Comment From: simonjayhawkins
cc @jorisvandenbossche
Comment From: MaxSchambach
Thanks @MaxSchambach for the report
the new experimental FloatingArray, introduced in 1.2 https://pandas.pydata.org/docs/whatsnew/v1.2.0.html#experimental-nullable-data-types-for-float-data and still some work to be done #38110 (including Honor the formatting options (eg precision))
Thanks for the quick response!
As long as the nullable floating type is still experimental I would suggest DataFrame.convert_dtypes not to default to convert_floating=True. I don't think I am the only one that was surprised by this :)
Comment From: jorisvandenbossche
This is indeed still a missing feature in the new float dtype. Also the nullable integer dtype has the same problem (but it's something you will typically do more often with floats, for sure)
Note that the full convert_dtypes method (and all the data types it enables) is still considered as experimental, and not only the Float data type (it's newer, though, so has more rough edges).
The docstring should maybe put more attention on this experimental status, but it does mention that the output can change if new nullable data types get added to pandas:
In the future, as new dtypes are added that support
pd.NA, the results of this method will change to support those new types.
Comment From: MaxSchambach
Yes that is true. To be fair, I have not used the convert_dtypes method before, however reading the documentation, I did not have the impression that it was still an experimental feature. As far as the documentation goes, I could tell it was introduced in 1.0 with the float behavior changing in 1.2.0. Now that I understand the issue, things are more clear to me. However since I spend at least 2 hours on getting down to the issue (starting with the behavior in DataFrame.to_latex) I would like to spare other people facing the same issue :)
Comment From: jorisvandenbossche
Out of curiosity, can I ask why you were using the convert_dtypes method?
Comment From: MaxSchambach
Well that's a good point!
I was dealing with DataFrames obtained from a MongoDB instance which contained mixed data in some columns, hence those columns were initially object types. After filtering out all non-floats, those columns were still object columns and since I there were a lot of them, I thought convert_dtypes is exactly what I need. And despite the confusion with Float64 it worked as expected.
Comment From: jorisvandenbossche
Yes, due to the name "convert_dtypes", that's a logical thing to think. But, the method is not meant to generally convert object dtypes, but specifically to convert normal dtypes to the nullable dtypes (so if you explicitly want to use/test those experimental data types). For inferring object type (exactly for the use case that you describe), there is the infer_objects method to use for this.
(we had a long discussion about the name when the method was added, I personally find it confusing and not explicit, but unfortunately I lost the debate ;))
Comment From: MaxSchambach
Aha, yes I see!
I agree, the name is confusing, however, when nullable types are going to be non-experimental, the behavior would be as expected I believe. In the meantime, the docs of convert_dtypes should be more elaborating, possibly even referring to infer_objects as well.
And thanks for the hint, I will update my usage of convert_dtypes to infer_objects :)