Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of pandas.

  • [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from io import StringIO
data = 'A\n' + '\n'.join(['CAT'] * 1)  # Number of CAT lines needs to be at least 1, works with 10,000 too
df = pd.read_csv(StringIO(data), engine='pyarrow', na_values='CAT')
assert df['A'].isna()[0]  # Doesn't raise AssertionError, as expected
# Give the cat some food:
data = 'A\n' + '\n'.join(['CAT'] * 1) + '\n' + 'food'
df = pd.read_csv(StringIO(data), engine='pyarrow', na_values='CAT')
assert df['A'].isna()[0]  # Raises AssertionError unexpectedly

Issue Description

When using read_csv with engine='pyarrow', na_values is not respected unless the whole column has NA values.

This does not reproduce without the engine='pyarrow' argument.

Expected Behavior

na_values should always be respected.

Installed Versions

INSTALLED VERSIONS

commit : 37ea63d540fd27274cad6585082c91b1283f963d python : 3.11.1.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19043 machine : AMD64 processor : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : Hebrew_Israel.1255 pandas : 2.0.1 numpy : 1.24.3 pytz : 2023.3 dateutil : 2.8.2 setuptools : 65.7.0 pip : 22.3.1 Cython : None pytest : 7.3.1 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.12.0 pandas_datareader: None bs4 : 4.12.2 bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : 11.0.0 pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : 2023.3 qtpy : 2.3.1 pyqt5 : None

Comment From: phofl

Hi, thanks for your report. This is fixed on main, but might need a test, not sure. cc @lithomas1

Comment From: natmokval

I would like to work on this.

Comment From: eli-b

No arrow label? It doesn't reproduce without using the pyarrow backend.

Even if it's fixed on main, I guess it's good that the issue would be documented for versions 2.0.0 and 2.0.1.

Comment From: phofl

We don't document bugs

Comment From: anshuman0123

Hey , I want to work on this issue

Comment From: lithomas1

I think this has been fixed by me in #52087, and it should be tested, so I'm going to close this. (The issue was that the strings_can_be_null flags wasn't set, so NA values in string columns weren't picked up).

@natmokval @anshuman0123 If you're still interested in working on the Arrow engine for read_csv, you can try fixing some of the tests marked as skip_pyarrow or xfail_pyarrow in the codebase.

(Just a warning, though: Some may be easier than others, and others are unfixable due to missing features in Pyarrow).

You can also try filtering by the "Arrow" and "IO CSV" labels to find more similar issues.

Comment From: eli-b

I didn't mean to imply any extra work is needed from anybody. I'm happy that this is fixed in main. I meant it is now documented in this issue. Sorry.

We don't document bugs

I see bugs documented in the whatnew files.