Pandas BUG: Create Panel from DataFrame of tz-aware datetimes fails

In pandas 0.18, if you create a Panel where at least one of the items consists of all tz-aware timestamps, then trying to access any elements of that item raises an error.

import pandas, numpy

data = numpy.array(list(pandas.date_range('2015', periods=24, tz='utc'))).reshape(2,3,4)
panel = pandas.Panel(data)

panel[0] # this fails
panel.loc[0,0,0] # fails with the same error
panel.to_frame() # same error

The error raised is:

Traceback (most recent call last):
  File "/Users/ajenkins/Library/Preferences/PyCharm50/scratches/test.py", line 6, in <module>
    panel[0] # this fails
  File "/Users/ajenkins/dev/pandas/pandas/core/panel.py", line 274, in __getitem__
    return super(Panel, self).__getitem__(key)
  File "/Users/ajenkins/dev/pandas/pandas/core/generic.py", line 1344, in __getitem__
    return self._get_item_cache(item)
  File "/Users/ajenkins/dev/pandas/pandas/core/generic.py", line 1352, in _get_item_cache
    res = self._box_item_values(item, values)
  File "/Users/ajenkins/dev/pandas/pandas/core/panel.py", line 567, in _box_item_values
    return self._constructor_sliced(values, **d)
  File "/Users/ajenkins/dev/pandas/pandas/core/frame.py", line 255, in __init__
    copy=copy)
  File "/Users/ajenkins/dev/pandas/pandas/core/frame.py", line 432, in _init_ndarray
    return create_block_manager_from_blocks([values], [columns, index])
  File "/Users/ajenkins/dev/pandas/pandas/core/internals.py", line 3986, in create_block_manager_from_blocks
    mgr = BlockManager(blocks, axes)
  File "/Users/ajenkins/dev/pandas/pandas/core/internals.py", line 2591, in __init__
    (block.ndim, self.ndim))
AssertionError: Number of Block dimensions (1) must equal number of axes (2)

The error seems to be caused by a change in pandas.core.common._possibly_infer_to_datetimelike to make it use a tz-aware DatetimeIndex where possible. In the case where the values parameter is an ndarray of tz-aware timestamps, it returns a tz-aware DatetimeIndex, which loses the original shape of the input. This causes an error later.

Perhaps the fix should be to make this function only perform the conversion to DatetimeIndex when the input is 1D.

output of pd.show_versions()

INSTALLED VERSIONS

commit: 856c3bde62fa7b753a25cfbacc544d1fa415a676 python: 2.7.9.final.0 python-bits: 64 OS: Darwin OS-release: 15.4.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8

pandas: 0.18.0+150.g856c3bd nose: 1.3.7 pip: 8.1.1 setuptools: 20.9.0 Cython: 0.24 numpy: 1.11.0 scipy: 0.15.1 statsmodels: None xarray: None IPython: 3.1.0 sphinx: None patsy: None dateutil: 2.5.3 pytz: 2016.4 blosc: None bottleneck: None tables: None numexpr: None matplotlib: 1.4.2 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None httplib2: None apiclient: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None boto: None pandas_datareader: None

Comment From: ajenkins-cargometrics

This patch seems to fix the problem.

diff --git a/pandas/core/common.py b/pandas/core/common.py
index 14c95e0..232078d 100644
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -1153,11 +1153,12 @@ def _possibly_infer_to_datetimelike(value, convert_dates=False):
                 # we might have a sequence of the same-datetimes with tz's
                 # if so coerce to a DatetimeIndex; if they are not the same,
                 # then these stay as object dtype
-                try:
-                    from pandas import to_datetime
-                    return to_datetime(v)
-                except:
-                    pass
+                if len(shape) == 1:
+                    try:
+                        from pandas import to_datetime
+                        return to_datetime(v)
+                    except:
+                        pass

             except:
                 pass

Comment From: jreback

closing as Panels are deprecated