Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
def test_pandas_parquet_writer_bug():
import pandas as pd
from fs import open_fs
df = pd.DataFrame(data={"A": [0, 1], "B": [1, 0]})
with open_fs('osfs://./').open('.test.parquet', 'wb') as f:
print(f.name)
df.to_parquet(f)
Issue Description
Running the example above with Pandas 1.5.0 results in the exception below, but works fine in 1.4.3.
File ".venv\lib\site-packages\pandas\io\parquet.py", line 202, in write
self.api.parquet.write_table(
File ".venv\lib\site-packages\pyarrow\parquet.py", line 1970, in write_table
with ParquetWriter(
File ".venv\lib\site-packages\pyarrow\parquet.py", line 655, in __init__
self.writer = _parquet.ParquetWriter(
File "pyarrow\_parquet.pyx", line 1387, in pyarrow._parquet.ParquetWriter.__cinit__
File "pyarrow\io.pxi", line 1579, in pyarrow.lib.get_writer
TypeError: Unable to read from object of type: <class 'bytes'>
This seems to be because of this new block added to io.parquet.py:
if (
isinstance(path_or_handle, io.BufferedWriter)
and hasattr(path_or_handle, "name")
and isinstance(path_or_handle.name, (str, bytes))
):
path_or_handle = path_or_handle.name
Which assumes that path_or_handle.name will always be of type str, but is actually bytes in this example.
A solution that works for me is to simply append this below the block:
if isinstance(path_or_handle, bytes):
path_or_handle = path_or_handle.decode("utf-8")
Alternatively, just passing thr BufferedWriter through to the subsequent self.api.parquet.write_to_dataset call works as well, but I'm not sure of the implications of reverting to this old behaviour (which I why I haven't submitted a pull request, but happy to do so if someone can confirm).
Cheers, Sam.
Expected Behavior
No exception is thrown
Installed Versions
Comment From: phofl
commit a15ca9755f8e43f9208771f72794e7dd01af92ff
Author: Chandrasekaran Anirudh Bhardwaj <anirudhsekar96@gmail.com>
Date: Sat Jan 22 16:21:22 2022 -0800
closes #44914 by changing the path object to string if it is of io.BufferWriter (#45480)
Comment From: phofl
cc @Anirudhsekar96
Comment From: twoertwein
Do we really want to allow paths in bytes? I think users will then also expect to use bytes paths for all other IO functions (including some which accept the bytes content - not a bytes path)
Comment From: phofl
No strong opinion, could deprecate, but silent behavior change is not great
Comment From: SamRWest
A BufferedWriter with a name property of bytes seems to be valid - eg: open(b'./test.parquet', 'rb') works fine.
And if PyFileSystem2 handles are of this format, and the previous pandas version supported them, it'd be good to continue supporting this.
Comment From: Anirudhsekar96
@SamRWest Thanks for checking! Since the docs mentioned str like path hence I did not check for bytes like object with name attribute in my commit.
The docs indicate that FS like object is also an expected input hence a PyFileSystem2 handle as input would also be expected behavior, as @phofl mentioned if past versions supported a PyFileSystem2 handle then we should not allow silent behavior change and allow for bytes like object with name attribute as the path.
Comment From: MarcoGorelli
Hey @SamRWest
Thanks for having reported this - there's now a release candidate for pandas 2.0.0, which you can install with
mamba install -c conda-forge/label/pandas_rc pandas==2.0.0rc0
or
python -m pip install --upgrade --pre pandas==2.0.0rc0
If you try it out and report bugs, then we can fix them before the final 2.0.0 release