Pandas pd.to_datetime raises expection when converting date object with unit param

Hello, since pandas 0.18.1, converting to datetime a date defined as object, using the unit parameter, raises a ValueError exception.

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime

df = pd.DataFrame([[datetime.date(2016, 5, 10)]], index=[0], columns=['date'])

pd.to_datetime(df.date, unit='D')

Expected Output

Without the unit parameter, it does not raise the exception

output of `pd.show_versions()`

INSTALLED VERSIONS

commit: None python: 2.7.3.final.0 python-bits: 64 OS: Linux OS-release: 3.5.0-37-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8

pandas: 0.18.1 nose: 1.3.6 pip: None setuptools: 5.1 Cython: None numpy: 1.11.0 scipy: 0.17.0 statsmodels: 0.5.0 xarray: None IPython: 3.1.0 sphinx: 1.2.2 patsy: 0.2.1 dateutil: 2.5.3 pytz: 2016.4 blosc: None bottleneck: None tables: 2.3.1 numexpr: 2.5 matplotlib: 1.4.3 openpyxl: 1.6.1 xlrd: 0.9.3 xlwt: 0.7.5 xlsxwriter: None lxml: None bs4: 4.3.2 html5lib: None httplib2: 0.7.2 apiclient: None sqlalchemy: 1.0.11 pymysql: None psycopg2: None jinja2: 2.7.2 boto: 2.27.0 pandas_datareader: 0.2.1

I've already tested with the master branch.

Comment From: jreback

I don't know why you would think this would work.

from the doc-string.

unit : unit of the arg (D,s,ms,us,ns) denote the unit in epoch
    (e.g. a unix timestamp), which is an integer/float number.