How do you write custom objects to an HDF file store? Is there any guidelines for writing custom series to an HDF store? If so where? Thank you

Andrew

Comment From: jreback

what is a custom series? what are you trying to do?

Comment From: achapkowski

I have a custom series that holds a dictionary in each row, sort of like the postgres hstore field type. When I call my dataframe object with the custom series to_hdf() as follows:

df = pd.DataFrame([[{'a':1}, 1, 'a']], columns=["A", "B", "C"])
df.to_hdf(r"c:\temp\test.h5", key='mydf') # error

I get the following error:

builtins.TypeError: cannot properly create the storer for: [_TYPE_MAP] [group->/asdf (Group) '',value->,format->fixed,append->False,kwargs->{'encoding': None}]

Comment From: jreback

that's quite non-performant in general. DataFrames are designed to handle scalar values, not generic collections.

you can pickle it if you want. but HDF5 is based around fix-length string storage. It doesn't handle generic nested objects like that at all.

Comment From: TomAugspurger

There are a few stackoverflow questions about storing dictionaries in HDF5. Pandas uses pytables under the hood, so you're welcome to use that. But since nested data aren't handled that well in pandas, serializing them isn't something we'd add to the library.

Comment From: achapkowski

@TomAugspurger Thank you for the link @jreback So it looks like it's a pytable thing, not pandas. good to know.