Code Sample, a copy-pastable example if possible
import pytz
import pandas as pd
from datetime import datetime
px=[100.0, 101.0, 102.0, 103.0]
idx = pd.date_range('2017-01-01', periods=4, tz="UTC")
df = pd.DataFrame({'px': px}, index=idx)
cutoff = datetime(2017, 1, 3, 8, 0, 0)
cutoff = pytz.UTC.localize(cutoff)
df.loc[df.index < cutoff, ['px']] += 88.8
print(df)
Problem description
Running the code above, you will find that all affected rows in df becomes NaN:
px
2017-01-01 00:00:00+00:00 NaN
2017-01-02 00:00:00+00:00 NaN
2017-01-03 00:00:00+00:00 NaN
2017-01-04 00:00:00+00:00 103.0
What’s probably equally strange is, the problem goes away with either one of the following two modifications: - Make idx timezone-unaware i.e. remove the tz=”UTC” in the definition of idx, and the NaNs will go away - Make px a list of integers instead of floats: px=[100, 101, 102, 103], and the NaNs will go away!
Expected Output
The expected output is df with all rows before cutoff time incremented by given value (88.8)
px
2017-01-01 00:00:00+00:00 188.8
2017-01-02 00:00:00+00:00 189.8
2017-01-03 00:00:00+00:00 190.8
2017-01-04 00:00:00+00:00 103.0
Output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.35-pv-ts2
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.20.3
pytest: 3.0.3
pip: 8.1.2
setuptools: 27.2.0
Cython: 0.24.1
numpy: 1.11.2
scipy: 0.18.1
xarray: None
IPython: 5.1.0
sphinx: 1.4.8
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: 1.1.0
tables: 3.2.3.1
numexpr: 2.6.1
feather: None
matplotlib: 1.5.3
openpyxl: 2.4.0
xlrd: 1.0.0
xlwt: None
xlsxwriter: 0.9.3
lxml: None
bs4: 4.5.1
html5lib: None
sqlalchemy: 1.1.0
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
pandas_gbq: None
pandas_datareader: None
Comment From: david8373
Temporary work-around as suggested by @jreback :
mask = df.index < cutoff
df.loc[mask] = df.loc[mask, ['px']] + 88.8
Comment From: watercrossing
I can't reproduce this on master - it seems to have been fixed in the mean time.
Comment From: watercrossing
Yes, it was fixed here: 142b5b61a0433c256511649a993b0fe1b6f64524.
Comment From: jreback
is this different enough to warrant a validation test? if so can u put up a PR otherwise i can close
Comment From: watercrossing
I think the tests in 142b5b6 cover this case, so I think it can be closed.
Comment From: jreback
thanks @watercrossing