Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[ ] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
s1 = pd.Series([pd.NaT, pd.NaT, datetime.datetime(1900, 1, 1)])
s2 = pd.Series([pd.NaT, datetime.datetime(1900, 1, 2), datetime.datetime(1900, 1, 3)])
df = pd.DataFrame({'c1':s1, 'c2':s2})
df_utc = pd.DataFrame({'c1':s1.dt.tz_localize("UTC"), 'c2':s2.dt.tz_localize("UTC")})
df.min(axis=1,skipna=False) # expected result with 2 NaT
df.min(axis=1,skipna=True) # expected result with 1 NaT
df_utc.min(axis=1,skipna=False # ISSUE unexpected result with 1 NaT instead of 2
df_utc.min(axis=1,skipna=True) # ISSUE warning + unexpected results: all NaNs
Issue Description
We would expect 2 NaTs in the following call, not just one
In [71]: df_utc.min(axis=1,skipna=False)
Out[71]:
0 NaT
1 1900-01-02 00:00:00+00:00
2 1900-01-01 00:00:00+00:00
dtype: datetime64[ns, UTC]
We would expect a single NaT, but we get a warning and a series of float NaNs
In [72]: df_utc.min(axis=1,skipna=True)
<ipython-input-72-c024a9c167c1>:1: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError. Select only valid columns before calling the reduction.
df_utc.min(axis=1,skipna=True)
Out[72]:
0 NaN
1 NaN
2 NaN
dtype: float64
The simliar calls with df (no timezones) return expected results.
In [73]: df.min(axis=1,skipna=False)
Out[73]:
0 NaT
1 NaT
2 1900-01-01
dtype: datetime64[ns]
In [74]: df.min(axis=1,skipna=True)
Out[74]:
0 NaT
1 1900-01-02
2 1900-01-01
dtype: datetime64[ns]
Expected Behavior
see description
Installed Versions
INSTALLED VERSIONS
------------------
commit : 8dab54d6573f7186ff0c3b6364d5e4dd635ff3e7
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.191-el7.lime.2.x86_64
Version : #1 SMP Thu Jul 14 19:09:52 EDT 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.5.2
numpy : 1.21.3
pytz : 2022.1
dateutil : 2.8.2
setuptools : 65.2.0
pip : 22.1.2
Cython : None
pytest : 4.6.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.0.3
IPython : 7.33.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli :
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.3.4
numba : 0.50.1+3.g7f7b8af82
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 4.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : 1.4.9
tables : None
tabulate : 0.8.9
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
zstandard : None
tzdata : None
Comment From: rhshadrach
Thanks for the report! Confirmed on main - the 2nd example now raises due to the numeric_only changes so tagging as 2.0. #50224 would fix; I'm going to revisit that and see if I can get it working.
In the meantime, the workaround df_utc.T.min(skipna=False) can be used (that's what #50224 is doing too).