-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[ ] (optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
s1 = pd.Series([pd.NA, -1], dtype='Int64')
s2 = pd.Series([pd.NA, -1]) # the dtype is now object
Problem description
s1.fillna(np.nan) and s2.fillna(np.nan) return different series. Specifically, the first one keeps the pd.NA cell untouched and does not replace it with np.nan. Replacing with any other value that np.nan does not trigger this bug.
Expected Output
> s2.fillna(np.nan)
0 NaN
1 -1.0
dtype: float64
Output of pd.show_versions()
Comment From: mzeitlin11
Thanks for reporting this @cynddl! While certainly confusing, I believe this is intended behavior. For object type, pd.NA and np.nan are both valid missing values and treated as distinct.
For nullable types like Int64, there is a notion of a single missing value, pd.NA. Generally (though there are probably inconsistencies here), using other missing values than pd.NA with these types will still treat that value as pd.NA. For example using a missing value in construction,
In [3]: pd.Series([np.nan, -1], dtype="Int64")
Out[3]:
0 <NA>
1 -1
dtype: Int64
If you want pandas to treat pd.NA and np.nan differently, your best bet would be object type data (or perhaps nullable float data depending on how #32265 goes)
Comment From: simonjayhawkins
If you want
pandasto treatpd.NAandnp.nandifferently, your best bet would beobjecttype data (or perhaps nullable float data depending on how #32265 goes)
adding no action label for now. needs to considered as part of discussion in #32265. cc @jorisvandenbossche
Comment From: simonjayhawkins
if the outcome of the discussion is that fillna() on a FloatingArray should be changed...
>>> s = pd.Series([pd.NA, -1], dtype='Float64')
>>> s
0 <NA>
1 -1.0
dtype: Float64
>>>
>>> s.fillna(np.nan)
0 <NA>
1 -1.0
dtype: Float64
then for the IntegerArray, which does not support floats, then the expected behavior would need to be revisted, but for now the behaviour of IntegerArray and FloatingArray are consistent.
Comment From: jorisvandenbossche
As long as we allow NaN in FloatArray (which we currently do, but dependent on https://github.com/pandas-dev/pandas/issues/32265), it should be possible to fill your missing values with NaN, so the example above of pd.Series([pd.NA, -1], dtype='Float64').fillna(np.nan) still returning a Series with NA seems wrong. We should change that to have a Series with NaN.
The problem is that this generates some inconsistencies.
In general, when the user uses np.nan as input value, we automatically convert that to NA for nullable dtypes. For example, pd.Series([np.nan, 1], dtype="Float64") converts the NaN into NA, as shown above. But this behaviour for constructors also propagates to other cases. Eg in setitem, such as s[mask] = values, if values is NaN or contains NaNs, those are also treated as NA.
Following that, it makes some sense that also the input argument (the value to fill with) to fillna is treated consistently, and NaN is treated as NA. And that gives the current behaviour, since filling with NA doesn't do anything.
So as long as we automatically convert NaN to NA on input by default, we will probably need a way to specify if you don't want this conversion and you actually want to use NaN (such as the Series[float].fillna(np.nan if you actually want to replace NA with NaN). For methods as fillna, that could maybe be done with a keyword.
We might also want to start raise a UserWarning, to avoid suprises, if NaN is silently treated as NA, especially in cases where it is the input to methods like fillna.
Comment From: phofl
This is basically a duplicate of #39926 and #25288 so closing here