This involves incorrect pickling of Pandas Panel values. From what I've seen, it occurs when new values of a different data type are assigned to a Pandas Panel axis. Everything functions well until the panel is pickled. When the pickle is read the old values appear. In the example below floats are reassigned as objects, but when the pickle is loaded back up, it contains the old float values. Also noted: - does not happen with DataFrames - does not happen if the new values are of the same data type as the old ones (in that case id() of the Series is the same) - happens whenever data type is changed (in the snippet below float to str but happens in reverse too).
import pandas as pd
import numpy as np
import pickle
wp = pd.Panel(np.random.randn(2,3,2),
items=['Item1', 'Item2'],
major_axis=[1,2,3],
minor_axis=['float', 'string'])
print wp['Item1']
wp['Item1']['string'] = 'blue'
print wp['Item1']
with open('test_pickle', 'wb') as f:
pickle.dump(wp, f)
wp2 = pickle.load( open('test_pickle', 'rb') )
print wp2['Item1']
UPDATE: It's not a pickle problem. For example, a transpose operation on the Panel also erases the new values and brings back the old ones.
wp=wp.transpose(2,0,1)
wp=wp.transpose(1,2,0)
print wp['Item1']
Comment From: jreback
You are assigning to a copy. See the docs here as this is NOT a recommended syntax. Its possible to show the SettingWithCopyWarning
, so for that reason I'll mark this as an issue. You should always use`.loc`` for an assignment.
Docs are here
In [15]: wp = pd.Panel(np.random.randn(2,3,2),
items=['Item1', 'Item2'],
major_axis=[1,2,3],
minor_axis=['float', 'string'])
In [16]: wp
Out[16]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 2 (minor_axis)
Items axis: Item1 to Item2
Major_axis axis: 1 to 3
Minor_axis axis: float to string
In [17]: wp._data
Out[17]:
BlockManager
Items: Index([u'Item1', u'Item2'], dtype='object')
Axis 1: Int64Index([1, 2, 3], dtype='int64')
Axis 2: Index([u'float', u'string'], dtype='object')
FloatBlock: slice(0, 2, 1), 2 x 3 x 2, dtype: float64
In [18]: wp['Item1']['string'] = 'foo'
In [19]: wp._data
Out[19]:
BlockManager
Items: Index([u'Item1', u'Item2'], dtype='object')
Axis 1: Int64Index([1, 2, 3], dtype='int64')
Axis 2: Index([u'float', u'string'], dtype='object')
FloatBlock: slice(0, 2, 1), 2 x 3 x 2, dtype: float64
Comment From: pjanowski
Thanks, very clear! I agree that a SettingWithCopy warning on Panel would be helpful.
Comment From: jreback
closing as Panels are deprecated