For some reason, it's not possible to subtract a numpy.datetime64 object from a Series, even though subtracting a datetime.datetime object is possible. I found this bug when trying to work around issue #8554.
Code Sample, a copy-pastable example if possible
import pandas as pd
import numpy as np
from datetime import datetime
EPOCH = datetime(1970, 1, 1)
EPOCH64 = np.datetime64(EPOCH)
df = pd.DataFrame(pd.date_range('2014-11-19', '2014-11-21', freq='D'))
epoch_ts = df[0] - EPOCH # Works
epoch_tsnp = pd.Series(df[0].values - EPOCH64, name=0) # Works
epoch_ts64 = df[0] - EPOCH64 # TypeError
Expected Output
epoch_ts and epoch_tsnp are both:
0 16393 days
1 16394 days
2 16395 days
Name: 0, dtype: timedelta64[ns]
There is a TypeError raised when epoch_ts64 is calculated. A full stack trace follows:
~/anaconda3/lib/python3.5/site-packages/pandas/core/ops.py in wrapper(left, right, name, na_op)
605
606 time_converted = _TimeOp.maybe_convert_for_time_op(left, right, name,
--> 607 na_op)
608
609 if time_converted is None:
~/anaconda3/lib/python3.5/site-packages/pandas/core/ops.py in maybe_convert_for_time_op(cls, left, right, name, na_op)
565 return None
566
--> 567 return cls(left, right, name, na_op)
568
569
~/anaconda3/lib/python3.5/site-packages/pandas/core/ops.py in __init__(self, left, right, name, na_op)
279
280 lvalues = self._convert_to_array(left, name=name)
--> 281 rvalues = self._convert_to_array(right, name=name, other=lvalues)
282
283 self.name = name
~/anaconda3/lib/python3.5/site-packages/pandas/core/ops.py in _convert_to_array(self, values, name, other)
400 if (supplied_dtype is None and other is not None and
401 (other.dtype in ('timedelta64[ns]', 'datetime64[ns]')) and
--> 402 isnull(values).all()):
403 values = np.empty(values.shape, dtype='timedelta64[ns]')
404 values[:] = iNaT
~/anaconda3/lib/python3.5/site-packages/pandas/core/common.py in isnull(obj)
174 pandas.notnull: boolean inverse of pandas.isnull
175 """
--> 176 return _isnull(obj)
177
178
~/anaconda3/lib/python3.5/site-packages/pandas/core/common.py in _isnull_new(obj)
184 raise NotImplementedError("isnull is not defined for MultiIndex")
185 elif isinstance(obj, (ABCSeries, np.ndarray, pd.Index)):
--> 186 return _isnull_ndarraylike(obj)
187 elif isinstance(obj, ABCGeneric):
188 return obj._constructor(obj._data.isnull(func=isnull))
~/anaconda3/lib/python3.5/site-packages/pandas/core/common.py in _isnull_ndarraylike(obj)
275 result = values.view('i8') == tslib.iNaT
276 else:
--> 277 result = np.isnan(values)
278
279 # box
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 3.19.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
pandas: 0.18.0
nose: 1.3.7
pip: 8.1.1
setuptools: 20.2.2
Cython: 0.23.4
numpy: 1.10.4
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: 4.0.3
sphinx: 1.3.5
patsy: 0.4.0
dateutil: 2.5.1
pytz: 2016.3
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.4.6
matplotlib: 1.5.1
openpyxl: 2.3.2
xlrd: 0.9.4
xlwt: 1.0.0
xlsxwriter: 0.8.4
lxml: 3.5.0
bs4: 4.4.1
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.11
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.39.0
Comment From: jreback
dupe of #7996
Comment From: jreback
pull-request are welcome to fix, its actually quite easy as all it needs is coercion if it not M8[ns]. Right about where the error is now.
In [48]: df[0] - EPOCH64.astype('M8[ns]')
Out[48]:
0 16393 days
1 16394 days
2 16395 days
Name: 0, dtype: timedelta64[ns]