Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
pd.options.mode.copy_on_write = True
# pd.options.mode.copy_on_write = "warn"
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
Issue Description
The result is the following error output:
Traceback (most recent call last): File "/home/rolf/jupyter/venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py", line 1429, in setitem values[indexer] = casted ~~~~~~^^^^^^^^^ ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/rolf/code/python/bug.py", line 7, in
Expected Behavior
Should not give an error. When uncommenting the copy_on_write="warn" line the code runs with no error message as expected (I'd expect a warning in this case, but thats a separate issue)
Installed Versions
Comment From: sooooooing
take
Comment From: jorisvandenbossche
@kameamea thanks a lot for testing the CoW mode and for the report! I can confirm that I see the same for 2.2 (it works fine without enabling CoW, but gives that error when enabled), and it also appears on the main development branch for 3.0. So something we should definitely fix.
Comment From: jorisvandenbossche
FWIW I think is related to https://github.com/pandas-dev/pandas/pull/51435#discussion_r1245201566, i.e. the fact that we pass integer indices instead of a slice(None) (:) to the underlying method, which ends up using this for indexing the numpy array, which fails in this case.
Comment From: bobcorthals
FYI. Remarkably, the reported error is raised only on the first call. Demonstration of this behavior (pandas 2.2.3):
import pandas as pd
# 2.2.3
pd.options.mode.copy_on_write = True
dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]})
df=dftest[["B","C"]]
try:
print('try: I failed\n')
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
except:
df.iloc[[1,3],:] = [[2, 2],[2 ,2]]
print('except: I worked\n')
print(df)
Prints:
try: I failed
except: I worked
B C
0 2 3
1 2 2
2 2 1
3 2 2
Comment From: jorisvandenbossche
@bobcorthals I think the reason for that is because the initial dataframes starts under the hood with the three columns stored in a single "block" (a single 2d numpy array). Then in the first assignment (which fails) it splits the block, and then the second time you try the assignment, because the block is already split at that point, it takes a different code path that does not have the bug.
Comment From: chilin0525
Hi @jorisvandenbossche and @bobcorthals, I retested on the main branch and got the same error message output when executing with and without the pd.options.mode.copy_on_write = True setting. It seems like we should pass the slice(None) parameter for BlockManager?
https://github.com/pandas-dev/pandas/blob/6bcd30397d67c3887288c7a82c2c235ce8bc3c7f/pandas/core/internals/managers.py#L575
The following is some testing result in main branch codebase:
* Without pd.options.mode.copy_on_write = True
```python
import pandas as pd
# pd.options.mode.copy_on_write = True dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]}) df = dftest[["B","C"]]
df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
print(dftest)
print(df)
* output:
Traceback (most recent call last):
File "/home/pandas/pandas/core/internals/blocks.py", line 1135, in setitem
values[indexer] = casted
ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pandas/test.py", line 9, in <module>
df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
File "/home/pandas/pandas/core/indexing.py", line 920, in __setitem__
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/pandas/pandas/core/indexing.py", line 1940, in _setitem_with_indexer
self._setitem_single_block(indexer, value, name)
File "/home/pandas/pandas/core/indexing.py", line 2204, in _setitem_single_block
self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
File "/home/pandas/pandas/core/internals/managers.py", line 578, in setitem
self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
File "/home/pandas/pandas/core/internals/blocks.py", line 1138, in setitem
raise ValueError(
ValueError: setting an array element with a sequence.
```
- With
pd.options.mode.copy_on_write = True```python import pandas as pd
pd.options.mode.copy_on_write = True dftest = pd.DataFrame({"A":[1,4,1,5], "B":[2,5,2,6], "C":[3,6,1,7]}) df = dftest[["B","C"]]
df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
print(dftest)
print(df)
* output:
Traceback (most recent call last):
File "/home/pandas/pandas/core/internals/blocks.py", line 1135, in setitem
values[indexer] = casted
ValueError: shape mismatch: value array of shape (2,2) could not be broadcast to indexing result of shape (2,)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pandas/test.py", line 9, in <module>
df.iloc[[1,3], :] = [[2, 2],[2 ,2]]
File "/home/pandas/pandas/core/indexing.py", line 920, in __setitem__
iloc._setitem_with_indexer(indexer, value, self.name)
File "/home/pandas/pandas/core/indexing.py", line 1940, in _setitem_with_indexer
self._setitem_single_block(indexer, value, name)
File "/home/pandas/pandas/core/indexing.py", line 2204, in _setitem_single_block
self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
File "/home/pandas/pandas/core/internals/managers.py", line 578, in setitem
self.blocks[0].setitem((indexer[0], np.arange(len(blk_loc))), value)
File "/home/pandas/pandas/core/internals/blocks.py", line 1138, in setitem
raise ValueError(
ValueError: setting an array element with a sequence.
```
Comment From: chilin0525
take