Research

  • [X] I have searched the [pandas] tag on StackOverflow for similar questions.

  • [X] I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/43254699/pandas-to-dict-returns-none-mixed-with-nan

Question about pandas

pddata as below: intc strc floatc 0 1 a 1 1 \<NA> b 2 2 3 c 3

listdata = pddata.to_dict('records')

then listdata with value pd.NA, which is not convenient to check in the code, [{'intc': 1, 'strc': 'a', 'floatc': 1}, {'intc': , 'strc': 'b', 'floatc': 2}, {'intc': 3, 'strc': 'c', 'floatc': 3}] usually need loop to change all the pd.NA to None. Whether pddata.to_dict support this data transfer and with better performence compare to python list loop method when process large of data, for example, a list of 10million dict element. If not support, maybe in the futher enhence to_dict method to support it?

Comment From: topper-123

Can you give a copy-pasteable code example that demonstrates the problem, please, so I can take a look at this.

Comment From: flyly0755

dftest = pd.DataFrame({
    'intc': [1, pd.NA, 3],
    'strc': ['a', 'b', 'c'],
    'floatc': [1.0, 2.0, 3.0],
    'intc2': [1, pd.NA, pd.NA],
})

with pandas data as above, i want to transform to list with dict element listdata = pddata.to_dict('records') but after that, i need to loop listdata for changing every pd.NA to general None This is the situation with pandas 1.5.

But with pandas 2.0, the result is different, default change pd.NA to None [{'intc': 1, 'strc': 'a', 'floatc': 1.0, 'intc2': 1}, {'intc': None, 'strc': 'b', 'floatc': 2.0, 'intc2': None}, {'intc': 3, 'strc': 'c', 'floatc': 3.0, 'intc2': None}]

Comment From: topper-123

Do you think the old or new behaviour is better? I got trouble understanding what your question, sorry.

Comment From: flyly0755

The new behaviour is what i want, thx! so changing pandas version from 1.5 to 2.0 can solve this problem.