-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] (optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
import pandas as pd
import numpy as np
data = {
"A": [15, np.nan, 5, 4],
"B": [15, 5, np.nan, 4],
}
df = pd.DataFrame(data)
df = df.astype({"A": "Int64", "B": "float64"})
df["C"] = df["A"] / df["B"]
print(df)
print(df.dtypes)
print(df.dropna(subset=["C"]))
This example outputs:
A B C
0 15 15.0 1.0
1 <NA> 5.0 <NA>
2 5 NaN NaN
3 4 4.0 1.0
A Int64
B float64
C Float64
dtype: object
A B C
0 15 15.0 1.0
2 5 NaN NaN
3 4 4.0 1.0
Problem description
Dividing an Int64 column by a float64 column results in a Float64 column that can contain both <NA> and NaN values. However, only <NA> values will be removed by the dropna function. I would expect only <NA> values in the new column and I would also expect the dropna function to remove all NA variants (both <NA> and NaN).
Expected Output
The row with index 2 should be dropped.
Output of pd.show_versions()
Comment From: simonjayhawkins
Thanks @tobiasbrodd for the report. cc @jorisvandenbossche
Comment From: sudhanshu2
Can I take this?
Comment From: sudhanshu2
take
Comment From: sudhanshu2
@simonjayhawkins a potential solution for fixing this bug is _isna_array to treat all arrays of numeric type as same for the purpose of null-types. So essentially check for all possible numeric null-types (such as NaN and <NA>) in all numeric arrays. Do you think this is a viable solution? I assume this will need some documentation update.
Comment From: jbrockmendel
Blocked by #32265
Comment From: jorisvandenbossche
This doesn't necessarily need to be blocked by https://github.com/pandas-dev/pandas/issues/32265, I think. Currently, for constructors or casting, when a non-nullable array is converted to a nullable one, we convert NaN to NA:
>>> np_arr = np.array([1, np.nan], dtype="float64")
>>> pd.array(np_arr, dtype="Float64")
<FloatingArray>
[1.0, <NA>]
Length: 2, dtype: Float64
If we follow that logic, I think also for an operation that involves a nullable and non-nullable array which results in a nullable array, I think it is up to the nullable array to properly cast the non-nullable array in its ops implementation. So for example in:
>>> pd.Series([0, 1], dtype="Float64") / pd.Series(np_arr)
0 0.0
1 NaN
dtype: Float64
I think it is the responsibility of FloatArray.__div__(self, other) to properly cast the other operand, so the above would be consistent with doing this cast manually:
>>> pd.Series([0, 1], dtype="Float64") / pd.Series(np_arr).astype("Float64")
0 0.0
1 <NA>
dtype: Float64
(and if at some point we decide to stop converting NaN to NA in the cast from np.float64 to pd.Float64Dtype, then the behaviour here can also follow that. But on the short term I think we can already make it internally consistent)
Comment From: jbrockmendel
when a non-nullable array is converted to a nullable one, we convert NaN to NA [...] But on the short term I think we can already make it internally consistent
AFAICT making everything internally consistent with the constructor behavior is (close to?) equivalent to deciding on #32265 that pd.NA precludes NaN.
Comment From: jorisvandenbossche
I don't think that is necessarily related. Whether we allow np.nan to be present or not in the nullable columns, we would regard np.nan in input in both cases as missing (by default). So I also think that arithmetic operations should do that (by default).
If you adapt the example slightly to add some 0's:
data = {
"A": [15, np.nan, 5, 4, 0],
"B": [15, 5, np.nan, 4, 0],
}
df = pd.DataFrame(data)
df = df.astype({"A": "Int64", "B": "float64"})
df["C"] = df["A"] / df["B"]
>>> df
A B C
0 15 15.0 1.0
1 <NA> 5.0 <NA>
2 5 NaN NaN
3 4 4.0 1.0
4 0 0.0 NaN
The above is on master, while if we would regard the NaNs in column "B" as missing, but still allow NaNs to be created in operations, the result would be:
>>> df
A B C
0 15 15.0 1.0
1 <NA> 5.0 <NA>
2 5 NaN <NA>
3 4 4.0 1.0
4 0 0.0 NaN
If we would decide to never have NaNs in a nullable columns (and thus convert NaN to NA in the result of operations that can create NaNs), the result would be:
>>> df
A B C
0 15 15.0 1.0
1 <NA> 5.0 <NA>
2 5 NaN <NA>
3 4 4.0 1.0
4 0 0.0 <NA>
But so in both cases (as I have showed it here), the third row of the last column is NA, and not NaN (as on the main branch).
Comment From: jbrockmendel
Whether we allow np.nan to be present or not in the nullable columns, we would regard np.nan in input in both cases as missing (by default)
I think I understand (and re-reading your earlier comment, could have communicated better earlier).
I think also for an operation that involves a nullable and non-nullable array which results in a nullable array, I think it is up to the nullable array to properly cast the non-nullable array in its ops implementation
IIUC the decision that "cast[ing] the non-nullable array" is the "proper" constitutes deciding that in the affected cases, we don't distinguish between np.nan and pd.NA. Though I now see that it isn't equivalent to a decision on #32265, it would only pin down a subset of cases in there.
AFAICT there are two reasonable-ish things that IntegerArray.__truediv__ could do when operating with a ndarray[float64]:
def __truediv__1(self, other):
[omitting handling for non-ndarray cases]
other = pd.array(other)
other_vals = other._data
other_mask = other._mask
res_mask = self._mask | other_mask
res_values = self._data / other_vals
return FloatingArray(res_values, res_mask)
def __truediv__2(self, other):
[omitting handling for non-ndarray cases]
res_values = self._data / other
res_mask = self._mask.copy()
return FloatingArray(res_values, res_mask)
Suppose we decide in #32265 to treat np.nan and pd.NA as semantically distinct. In that case __truediv__2 correctly retains nan entries while __truediv__1 changes them. On the flip side if we decide they are not distinct, then __truediv__1 behavior is more correct.