Code Sample, a copy-pastable example if possible
I have a dataframe with two columns start
and end
both have the dtype of datetime64[ns, tzoffset(None, -21600)]
but depending on how I access it the output removes the timezone info for for example
df.ix[0, 'end']
Out[30]: Timestamp('2016-07-17 06:26:00')
but
df['end'][0]
Out[32]: Timestamp('2016-07-17 00:26:00-0600', tz='tzoffset(None, -21600)')
Issue rises when I am doing something like this (length
is pandas.tslib.Timedelta):
df.ix[0, 'end'] = df.ix[0, 'start'] + df.ix[0, 'length']
ValueError Traceback (most recent call last)
<ipython-input-34-f63ebfd12a2e> in <module>()
----> 1 df.ix[0, 'end'] = df.ix[0, 'start'] + df.ix[0, 'length']
/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in __setitem__(self, key, value)
130 key = com._apply_if_callable(key, self.obj)
131 indexer = self._get_setitem_indexer(key)
--> 132 self._setitem_with_indexer(indexer, value)
133
134 def _has_valid_type(self, k, axis):
/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in _setitem_with_indexer(self, indexer, value)
534 # scalar
535 for item in labels:
--> 536 setter(item, value)
537
538 else:
/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in setter(item, v)
459 s._consolidate_inplace()
460 s = s.copy()
--> 461 s._data = s._data.setitem(indexer=pi, value=v)
462 s._maybe_update_cacher(clear=True)
463
/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in setitem(self, **kwargs)
2915
2916 def setitem(self, **kwargs):
-> 2917 return self.apply('setitem', **kwargs)
2918
2919 def putmask(self, **kwargs):
/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in apply(self, f, axes, filter, do_integrity_check, consolidate, raw, **kwargs)
2888
2889 kwargs['mgr'] = self
-> 2890 applied = getattr(b, f)(**kwargs)
2891 result_blocks = _extend_blocks(applied, result_blocks)
2892
/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in setitem(self, indexer, value, mgr)
640
641 # coerce args
--> 642 values, _, value, _ = self._try_coerce_args(self.values, value)
643 arr_value = np.array(value)
644
/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in _try_coerce_args(self, values, other)
2270 # test we can have an equal time zone
2271 if tz is None or str(tz) != str(self.values.tz):
-> 2272 raise ValueError("incompatible or non tz-aware value")
2273 other_mask = isnull(other)
2274 other = other.tz_localize(None).value
ValueError: incompatible or non tz-aware value
However if I do it like this:
df.ix[0, 'end'] = df['start'][0] + df.ix[0, 'length']
It works fine since that type of selection does not lose the timezone info.
output of pd.show_versions()
INSTALLED VERSIONS
commit: None python: 2.7.11.final.0 python-bits: 64 OS: Linux OS-release: 4.5.7-200.fc23.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.utf8
pandas: 0.18.1 nose: 1.3.7 pip: 8.1.2 setuptools: 20.1.1 Cython: 0.23.4 numpy: 1.11.1 scipy: 0.17.1 statsmodels: 0.6.1 xarray: None IPython: 4.1.1 sphinx: 1.2.3 patsy: 0.4.1 dateutil: 2.5.3 pytz: 2016.6.1 blosc: None bottleneck: 0.6.0 tables: None numexpr: 2.4.6 matplotlib: 1.5.1 openpyxl: 1.8.6 xlrd: None xlwt: None xlsxwriter: None lxml: 3.4.4 bs4: None html5lib: None httplib2: None apiclient: None sqlalchemy: 1.0.12 pymysql: None psycopg2: 2.6.1 (dt dec pq3 ext lo64) jinja2: 2.8 boto: 2.40.0 pandas_datareader: None
Comment From: jreback
pls show a complete copy-pastable example
Comment From: ekipmanager
This fails
from dateutil.tz import tzoffset
df = pd.DataFrame({'A': pd.Timestamp('20130102', tz=tzoffset(None, -21600)),
'B': pd.Timestamp('20130102', tz=tzoffset(None, -21600)),
'C': np.array([3] * 4, dtype='int32'), })
df.ix[0, 'A'] = df.ix[0, 'B'] + pd.tslib.Timedelta(1e9)
But not this:
df.ix[0, 'A'] = df['B'][0] + pd.tslib.Timedelta(1e9)
Comment From: jreback
dupe of #12938 thanks for the report
Comment From: jreback
in general we don't recommend using .ix
as is not semantically as correct as .loc
; further we NEVER recommend using chained indexing df['B'][0]
, but it is marked as a bug in any event. pull requests to fix are welcome!