ser = pd.Series([pd.NaT, pd.Timestamp.now()])
>>> ser.astype("string")
0 NaT
1 2020-09-05 16:44:52.259773
dtype: string
>>> pd.Series(ser._values.astype("string"))
0 <NA>
1 2020-09-05 16:44:52.259773
dtype: string
It would be nice to have the Series code (i.e. the Block.astype) dispatch to the array code.
Note also:
vals = ser._values.astype("string")
>>> vals.astype(ser.dtype)
ValueError: Could not convert object to NumPy datetime
>>> pd.Series(vals, dtype=ser.dtype)
ValueError: Could not convert object to NumPy datetime
>>> pd.to_datetime(vals) # <-- works
Comment From: jorisvandenbossche
It would be nice to have the Series code (i.e. the Block.astype) dispatch to the array code.
This already happens in general for extension dtypes:
https://github.com/pandas-dev/pandas/blob/aca77f7b1cc7987a5b757a3f80a278e1f5fc7998/pandas/core/internals/blocks.py#L583-L585
But DatetimeBlock (and probably TimedeltaBlock as well?) is the special case of not being an extension block, so taking a different path.
Now, I suppose that also the different path in theory should handle it correctly:
In [8]: pd.core.dtypes.cast.astype_nansafe(np.array(['NaT'], dtype="datetime64[ns]"), pd.StringDtype())
Out[8]:
<StringArray>
[<NA>]
Length: 1, dtype: string
so not really sure what is going wrong there.
Comment From: jbrockmendel
@jorisvandenbossche could use your help medium-priority as this is becoming a blocker on ArrayManager work.
dta = pd.date_range("2016-01-01", periods=3)._data
dta[0] = pd.NaT
dta2 = dta.tz_localize("US/Pacific")
ser = pd.Series(dta)
ser2 = pd.Series(dta2)
assert (dta.astype(str) == ser.astype(str).values).all()
assert (dta2.astype(str) == ser2.astype(str)._values).all()
In [9]: ser2.astype("string")
Out[9]:
0 <NA>
1 2016-01-02 00:00:00-08:00
2 2016-01-03 00:00:00-08:00
dtype: string
In [10]: pd.Series(dta2.astype("string"))
Out[10]:
0 <NA>
1 2016-01-02 00:00:00-08:00
2 2016-01-03 00:00:00-08:00
dtype: string
In [11]: assert ser2.astype("string").equals(pd.Series(dta2.astype("string")))
In [12]: ser.astype("string")
Out[12]:
0 NaT
1 2016-01-02
2 2016-01-03
dtype: string
In [13]: pd.Series(dta.astype("string"))
Out[13]:
0 <NA>
1 2016-01-02T00:00:00.000000000
2 2016-01-03T00:00:00.000000000
dtype: string
In [12] I think we want the first entry to be pd.NA, not "NaT". On the flip side, we probably want the latter two entries of [13] to match the latter two entries of [12].
i.e. dta.astype("string")[~dta.isna()] == dta.astype(str)[~dta.isna()] seems desirable.
This opens a new can of worms w/r/t consistency:
dta.astype(str)
dta.__array__(dtype=str)
dta.to_numpy(dtype=str)
StringArray.astype("datetime64[ns]")
StringArray._from_sequence(ndarray[dt64])
StringArray._from_sequence(DatetimeArray)
Comment From: jorisvandenbossche
In [12] I think we want the first entry to be pd.NA, not "NaT". On the flip side, we probably want the latter two entries of [13] to match the latter two entries of [12].
Yes, agreed.
First question: why is it different?
So it seems that Series.astype for tz-naive data (case In[12]), goes through Block.astype, which ends up calling Block.to_native_types for this case which on its turn uses DatetimeArray._format_native_types:
https://github.com/pandas-dev/pandas/blob/8dbb593d0c94107ec8b91a4723c40af537807ca4/pandas/core/internals/blocks.py#L2211-L2216
The DatetimeArray.astype for tz-naive data (case In[13]), on the other hand, use StringArray._from_sequence for string dtype (but still _format_native_types to do astype(str), the non-extension string type):
https://github.com/pandas-dev/pandas/blob/8dbb593d0c94107ec8b91a4723c40af537807ca4/pandas/core/arrays/datetimelike.py#L346-L351
I suppose we can fix DatetimeArray.astype to also use _format_native_types for the extension string dtype, specifying na_rep=pd.NA to avoid converting pd.NaT to a string.
In the end, this also relates a lot to https://github.com/pandas-dev/pandas/issues/22384 for more thoroughly clean this up (but I think we can certainly already fix the inconsistency above). I will try to carve out some time to look back at this issue.
Comment From: jorisvandenbossche
@jbrockmendel this seems to be working correctly now (using your initial example):
In [25]: ser = pd.Series([pd.NaT, pd.Timestamp.now()])
...:
...: ser.astype("string")
Out[25]:
0 <NA>
1 2022-12-09 13:00:34.091345
dtype: string
(not sure if it has done intentionally and is tested, or if we should still add tests for it)
Comment From: jbrockmendel
best guess is it still merits a test