The major_axis of a panel is often a DateTimeIndex, and often this DateTimeIndex has a locality set by .tz_localize('UTC') for instance. The same can be said of a DataFrame and its index. If the DataFrame's index and the Panel's major_axis are the same, the DataFrame's values can be inserted into the panel at a specified minor_axis 'label'.

However, if both the DataFrame's and Panel's DateTimeIndex are localized, then this results in NaN values.

First, I show this works when the DateTimeIndex is not localized.

>>> import pandas as pd
>>> panel = pd.Panel(items=['a','b'], major_axis=pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']))
>>> df = pd.DataFrame(data = [[7,8],[8,7]], index = pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']), columns=['a','b'])
>>> df
                     a  b
2017-02-09 16:00:00  7  8
2017-02-09 17:00:00  8  7
>>> panel.loc[:,:,'df_insert'] = df
>>> panel.loc[:,:,'df_insert']
                     a  b
2017-02-09 16:00:00  7  8
2017-02-09 17:00:00  8  7

Now we see the only thing I change is to localize the DateTimeIndex of both to 'UTC'. Now it does not work.

>>> panel = pd.Panel(items=['a','b'], major_axis=pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']).tz_localize('UTC'))
>>> df = pd.DataFrame(data = [[7,8],[8,7]], index = pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']).tz_localize('UTC'), columns=['a','b'])
>>> panel.loc[:,:,'df_insert'] = df
>>> panel.loc[:,:,'df_insert']
                            a   b
2017-02-09 16:00:00+00:00 NaN NaN
2017-02-09 17:00:00+00:00 NaN NaN

Problem description

The current behavior is a problem because it means panels with a localized DateTimeIndex major_axis cannot be set, and worse, this may happen unexpectedly upon localizing in the middle of code. Also, I'm not sure if it is a more general problem with dataframes, series, etc....hmmm...Dataframe <- Series works. So probably just panels are affected (?).

Expected Output

The expected output should be identical to the non-localized case.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.12.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None pandas: 0.19.2 nose: 1.3.7 pip: 9.0.1 setuptools: 33.1.0.post20170122 Cython: 0.25.2 numpy: 1.10.4 scipy: 0.17.1 statsmodels: 0.8.0 xarray: None IPython: 5.2.2 sphinx: 1.5.2 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: 3.3.0 numexpr: 2.6.1 matplotlib: 2.0.0 openpyxl: 2.4.1 xlrd: 1.0.0 xlwt: 1.2.0 xlsxwriter: 0.9.6 lxml: 3.7.2 bs4: 4.5.3 html5lib: 0.999 httplib2: None apiclient: None sqlalchemy: 1.1.5 pymysql: None psycopg2: None jinja2: 2.8 boto: 2.45.0 pandas_datareader: 0.2.1

Comment From: jreback

yeah it looks buggy. however, Panel support is extremely limited, in that its going to be deprecated soon. So fixes would have to be community driven.

Comment From: joseortiz3

I would be very sad if Panels become deprecated, because multi-indexed data frames currently are a pain to use and have many bugs and unexpected behaviors. In my opinion, there needs to be some class that natively handles 3+ dimensional data just as dataframes handle two-dimensional data. In MultiIndexed dataframes, the multi-index creates needless entanglement of two axes which really should not be associated with each other, each being completely independent.

Comment From: jreback

@joseortiz3 well, Panels have WAY more bugs than anything else, and are simply not going to be supported. You can certainly use http://xarray.pydata.org/en/stable/ to have first class nd support. But for quite a lot of reasons, Panels will be deprecated (certainly doesn't mean you cannot use them, but the official support will drop).

Comment From: jreback

@joseortiz3 of course if you (or the community) are willing to push fixes for Panels, they would be gladly accepted!

Comment From: jreback

closing as Panel deprecated