The major_axis of a panel is often a DateTimeIndex, and often this DateTimeIndex has a locality set by .tz_localize('UTC') for instance. The same can be said of a DataFrame and its index. If the DataFrame's index and the Panel's major_axis are the same, the DataFrame's values can be inserted into the panel at a specified minor_axis 'label'.
However, if both the DataFrame's and Panel's DateTimeIndex are localized, then this results in NaN values.
First, I show this works when the DateTimeIndex is not localized.
>>> import pandas as pd
>>> panel = pd.Panel(items=['a','b'], major_axis=pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']))
>>> df = pd.DataFrame(data = [[7,8],[8,7]], index = pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']), columns=['a','b'])
>>> df
a b
2017-02-09 16:00:00 7 8
2017-02-09 17:00:00 8 7
>>> panel.loc[:,:,'df_insert'] = df
>>> panel.loc[:,:,'df_insert']
a b
2017-02-09 16:00:00 7 8
2017-02-09 17:00:00 8 7
Now we see the only thing I change is to localize the DateTimeIndex of both to 'UTC'. Now it does not work.
>>> panel = pd.Panel(items=['a','b'], major_axis=pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']).tz_localize('UTC'))
>>> df = pd.DataFrame(data = [[7,8],[8,7]], index = pd.DatetimeIndex(['2017-02-09 16:00:00','2017-02-09 17:00:00']).tz_localize('UTC'), columns=['a','b'])
>>> panel.loc[:,:,'df_insert'] = df
>>> panel.loc[:,:,'df_insert']
a b
2017-02-09 16:00:00+00:00 NaN NaN
2017-02-09 17:00:00+00:00 NaN NaN
Problem description
The current behavior is a problem because it means panels with a localized DateTimeIndex major_axis cannot be set, and worse, this may happen unexpectedly upon localizing in the middle of code. Also, I'm not sure if it is a more general problem with dataframes, series, etc....hmmm...Dataframe <- Series works. So probably just panels are affected (?).
Expected Output
The expected output should be identical to the non-localized case.
Output of pd.show_versions()
Comment From: jreback
yeah it looks buggy. however, Panel
support is extremely limited, in that its going to be deprecated soon. So fixes would have to be community driven.
Comment From: joseortiz3
I would be very sad if Panels become deprecated, because multi-indexed data frames currently are a pain to use and have many bugs and unexpected behaviors. In my opinion, there needs to be some class that natively handles 3+ dimensional data just as dataframes handle two-dimensional data. In MultiIndexed dataframes, the multi-index creates needless entanglement of two axes which really should not be associated with each other, each being completely independent.
Comment From: jreback
@joseortiz3 well, Panels have WAY more bugs than anything else, and are simply not going to be supported. You can certainly use http://xarray.pydata.org/en/stable/ to have first class nd support. But for quite a lot of reasons, Panels will be deprecated (certainly doesn't mean you cannot use them, but the official support will drop).
Comment From: jreback
@joseortiz3 of course if you (or the community) are willing to push fixes for Panels, they would be gladly accepted!
Comment From: jreback
closing as Panel deprecated