Pandas ENH: use carray as a base rather than SparseArray

http://carray.pytables.org/docs/manual/

obviously these also offer persistence as well but this is a another dependence

Comment From: heroxbd

The URL changed to,

http://www.pytables.org/usersguide/libref/homogenous_storage.html

I really like to have HDF5 read/write in CArray to/from Pandas.

For example:

ptdump tt-014662.h5
/ (RootGroup) ''
...
/t4 (CArray(12671,), shuffle, zlib(4)) ''

As of 0.17.1 read_hdf of Pandas cannot read CArray "/t4" in the file.

But PyTables (3.2.2) can read it with:

import tables as tb
x = tb.open_file("tt-014662.h5")
x.get_node("/t4")

/t4 (CArray(12671,), shuffle, zlib(4)) ''
  atom := Float32Atom(shape=(), dflt=0.0)
  maindim := 0
  flavor := 'numpy'
  byteorder := 'little'
  chunkshape := (1584,)

Comment From: jreback

this issue is about a different issue actually. more of a mem-mapped back thing.

only PyTables Table's are readable directly in pandas. You can just read other structures using exactly the code you show above. What is the problem?

Comment From: heroxbd

Thanks for your explanation and sorry for hijacking this issue.

I just expected Pandas to read CArray off-the-shelf. Maybe it is a good idea to document this limitation of read_hdf5 in http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5.

Comment From: jreback

if u want to submit a pull request to the effect that pandas does not in general read PyTables / h5py formats (except for specific external compat) then that would be ok

Comment From: heroxbd

On my list, thanks.