Research
-
[X] I have searched the [pandas] tag on StackOverflow for similar questions.
-
[X] I have asked my usage related question on StackOverflow.
Link to question on StackOverflow
https://stackoverflow.com/questions/39207640/python-pandas-how-do-i-convert-from-datetime64ns-to-datetime
Question about pandas
df[column].astype('datetime64') doesnt convert "datetime64[ns]" to datetime
Comment From: MarcoGorelli
thanks for your issue
going to need a reproducible example I'm afraid - closing for now then, can reopen if you post one
Comment From: ChillarAnand
Here is a simple reproducible example. @MarcoGorelli
Comment From: MarcoGorelli
thanks
screenshots aren't searchable, can you paste code please?
Comment From: MarcoGorelli
along with expected output
Comment From: ChillarAnand
Current behaviour:
In [9]: s = pd.to_datetime(pd.Series(['20201212']))
In [10]: s
Out[10]:
0 2020-12-12
dtype: datetime64[ns]
In [11]: s.astype('datetime64')
Out[11]:
0 2020-12-12
dtype: datetime64[ns]
In [12]: np.dtype('datetime64[ns]') == np.dtype('datetime64')
Out[12]: False
Expected:
In [11]: s.astype('datetime64')
Out[11]:
0 2020-12-12
dtype: datetime64
Comment From: MarcoGorelli
thanks
there isn't a datetime64 dtype, you can have:
- datetime64[s]
- datetime64[ms]
- datetime64[us]
- datetime64[ns]
besides, in pandas 2.0 .astype('datetime64') doesn't even work
Comment From: ChillarAnand
Even this doesn't work.
In [26]: s
Out[26]:
0 1991-12-12
dtype: datetime64[ns]
In [27]: s.astype('datetime64[s]')
Out[27]:
0 1991-12-12
dtype: datetime64[ns]
If datetime64 dtype isn't available, shouldn't it throw an exception like this.
In [1]: s = pd.to_datetime(pd.Series(['19911212']))
In [2]: s
Out[2]:
0 1991-12-12
dtype: datetime64[ns]
In [3]: s.astype('datetime64[s]')
Out[3]:
0 1991-12-12
dtype: datetime64[ns]
In [4]: s.astype('datetime644')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 s.astype('datetime644')
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/generic.py:6240, in NDFrame.astype(self, dtype, copy, errors)
6233 results = [
6234 self.iloc[:, i].astype(dtype, copy=copy)
6235 for i in range(len(self.columns))
6236 ]
6238 else:
6239 # else, only a single dtype is given
-> 6240 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
6241 return self._constructor(new_data).__finalize__(self, method="astype")
6243 # GH 33113: handle empty frame or series
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/managers.py:448, in BaseBlockManager.astype(self, dtype, copy, errors)
447 def astype(self: T, dtype, copy: bool = False, errors: str = "raise") -> T:
--> 448 return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, ignore_failures, **kwargs)
350 applied = b.apply(f, **kwargs)
351 else:
--> 352 applied = getattr(b, f)(**kwargs)
353 except (TypeError, NotImplementedError):
354 if not ignore_failures:
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/blocks.py:526, in Block.astype(self, dtype, copy, errors)
508 """
509 Coerce to the new dtype.
510
(...)
522 Block
523 """
524 values = self.values
--> 526 new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
528 new_values = maybe_coerce_values(new_values)
529 newb = self.make_block(new_values)
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/dtypes/astype.py:279, in astype_array_safe(values, dtype, copy, errors)
273 msg = (
274 f"Expected an instance of {dtype.__name__}, "
275 "but got the class instead. Try instantiating 'dtype'."
276 )
277 raise TypeError(msg)
--> 279 dtype = pandas_dtype(dtype)
280 if isinstance(dtype, PandasDtype):
281 # Ensure we don't end up with a PandasArray
282 dtype = dtype.numpy_dtype
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/dtypes/common.py:1781, in pandas_dtype(dtype)
1778 # try a numpy dtype
1779 # raise a consistent TypeError if failed
1780 try:
-> 1781 npdtype = np.dtype(dtype)
1782 except SyntaxError as err:
1783 # np.dtype uses `eval` which can raise SyntaxError
1784 raise TypeError(f"data type '{dtype}' not understood") from err
TypeError: Invalid datetime metadata string "4"
In [5]: s.astype('stringa')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Input In [5], in <cell line: 1>()
----> 1 s.astype('stringa')
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/generic.py:6240, in NDFrame.astype(self, dtype, copy, errors)
6233 results = [
6234 self.iloc[:, i].astype(dtype, copy=copy)
6235 for i in range(len(self.columns))
6236 ]
6238 else:
6239 # else, only a single dtype is given
-> 6240 new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
6241 return self._constructor(new_data).__finalize__(self, method="astype")
6243 # GH 33113: handle empty frame or series
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/managers.py:448, in BaseBlockManager.astype(self, dtype, copy, errors)
447 def astype(self: T, dtype, copy: bool = False, errors: str = "raise") -> T:
--> 448 return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/managers.py:352, in BaseBlockManager.apply(self, f, align_keys, ignore_failures, **kwargs)
350 applied = b.apply(f, **kwargs)
351 else:
--> 352 applied = getattr(b, f)(**kwargs)
353 except (TypeError, NotImplementedError):
354 if not ignore_failures:
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/internals/blocks.py:526, in Block.astype(self, dtype, copy, errors)
508 """
509 Coerce to the new dtype.
510
(...)
522 Block
523 """
524 values = self.values
--> 526 new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
528 new_values = maybe_coerce_values(new_values)
529 newb = self.make_block(new_values)
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/dtypes/astype.py:279, in astype_array_safe(values, dtype, copy, errors)
273 msg = (
274 f"Expected an instance of {dtype.__name__}, "
275 "but got the class instead. Try instantiating 'dtype'."
276 )
277 raise TypeError(msg)
--> 279 dtype = pandas_dtype(dtype)
280 if isinstance(dtype, PandasDtype):
281 # Ensure we don't end up with a PandasArray
282 dtype = dtype.numpy_dtype
File ~/homebrew/Caskroom/mambaforge/base/lib/python3.10/site-packages/pandas/core/dtypes/common.py:1781, in pandas_dtype(dtype)
1778 # try a numpy dtype
1779 # raise a consistent TypeError if failed
1780 try:
-> 1781 npdtype = np.dtype(dtype)
1782 except SyntaxError as err:
1783 # np.dtype uses `eval` which can raise SyntaxError
1784 raise TypeError(f"data type '{dtype}' not understood") from err
TypeError: data type 'stringa' not understood
Comment From: MarcoGorelli
Even this doesn't work.
yes it does, you're on an old version of pandas