Pandas Error in matplotlib with datetime64 with pandas 0.21.0

With pandas 0.21 this code with plotting datetime64 fails:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

fig, sub = plt.subplots()
x = np.array(['2015-07-13T04:32:23+0200', '2015-07-13T04:32:24+0200', '2015-07-13T04:32:25+0200'], dtype="datetime64")
y = np.arange(len(x))
sub.plot(x, y)
sub.set_xlim(xmin=x[0], xmax=x[-1])

TypeError                                 Traceback (most recent call last)
<ipython-input-4-483d578e8eb2> in <module>()
      7 y = np.arange(len(x))
      8 sub.plot(x, y)
----> 9 sub.set_xlim(xmin=x[0], xmax=x[-1])

~\Anaconda3\lib\site-packages\matplotlib\axes\_base.py in set_xlim(self, left, right, emit, auto, **kw)
   2917                  'in singular transformations; automatically expanding.\n'
   2918                  'left=%s, right=%s') % (left, right))
-> 2919         left, right = mtransforms.nonsingular(left, right, increasing=False)
   2920 
   2921         if self.get_xscale() == 'log' and (left <= 0.0 or right <= 0.0):

~\Anaconda3\lib\site-packages\matplotlib\transforms.py in nonsingular(vmin, vmax, expander, tiny, increasing)
   2892     close to zero, it returns -*expander*, *expander*.
   2893     '''
-> 2894     if (not np.isfinite(vmin)) or (not np.isfinite(vmax)):
   2895         return -expander, expander
   2896 

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

While 0.20.3 works fine and produces the result.

I'm not sure if it is a matplotlib or pandas.

Issue in matplotlib: #9771

Comment From: TomAugspurger

Same explanation as in https://github.com/matplotlib/matplotlib/issues/9771.