Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Edit [rhshadrach]: The code below does not reproduce the issue.
# No idea how to exactly reproduce it, but it occurs sometimes. Logic is this:
import pandas as pd
bool_df = pd.DataFrame([
{"first": True, "second": False, "third": True},
{"first": True, "second": True, "third": True},
{"first": True, "second": False, "third": True},
{"first": True, "second": True, "third": True},
{"first": True, "second": True, "third": True},
{"first": True, "second": False, "third": True},
{"first": True, "second": True, "third": True},
{"first": False, "second": False, "third": True},
{"first": True, "second": True, "third": True},
{"first": True, "second": False, "third": True},
])
bool_df = bool_df[bool_df["third"]][["first", "second"]]
# In some cases, this line prints the length of the DataFrame (10)
print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10
# This line prints the expected output (1)
print(len(bool_df[(bool_df["first"] == False) & (bool_df["second"] == False)])) # Prints 1
# Using De Morgan's law also returned with the expected output
print(len(bool_df[~((bool_df["first"]) | (bool_df["second"]))])) # Prints: 1
Issue Description
We don't know when and why this occurs. We werre looking for any rational explanation for hours. Anyone else experienced similar? How could this be possible? (Environment: MacBook Pro 2023, Sequoia 15.3)
Expected Behavior
print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Print 1
Installed Versions
Comment From: snitish
I'm unable to reproduce this behavior both on 2.2.3 as well as dev. @adamszelestey-se do you have any stats on how often this line prints 10?
print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10
Comment From: rhshadrach
- How are you running this? A Python script, a notebook like Jupyter, IPython? If a notebook/IPython, are you restarting the kernel between runs?
- Can you add
print(bool_df.dtypes)on the line prior to the firstprint. What is the output of this when the unexpected result occurs?
Comment From: adamszelestey-se
I'm unable to reproduce this behavior both on 2.2.3 as well as dev. @adamszelestey-se do you have any stats on how often this line prints 10?
print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10
Yeah unfortunately it is really unpredicable for us too, but it occured pretty often, I would say 1 out of 2 cases. The code is not exactly the one I provided, but the logic shouldn't differ.
- How are you running this? A Python script, a notebook like Jupyter, IPython? If a notebook/IPython, are you restarting the kernel between runs?
- Can you add
print(bool_df.dtypes)on the line prior to the first
We were running it as a regular python module in a poetry environment.
I wanted to create the ticket if someone else also experience something like this, they could add more context, I can understand that you can not do anything with this until we figure out how to reproduce it.
Comment From: rhshadrach
The code is not exactly the one I provided, but the logic shouldn't differ.
Am I correct in saying that you can never reproduce the bug with the code example that's in the OP?
Comment From: adamszelestey-se
The code is not exactly the one I provided, but the logic shouldn't differ.
Am I correct in saying that you can never reproduce the bug with the code example that's in the OP?
Unfortunately, you are correct meant to showcase that as the pseudo code if someone comes across the same issue
This is the best I can give you, but can't reproduce the error at the moment: `
from concurrent.futures import as_completed, ThreadPoolExecutor
import pandas as pd
import time
import random
random.seed(42)
def get_result(_id: str):
result = {}
sleep = random.random() * 5
time.sleep(sleep)
result["id"] = _id
return result
def get_single_metric(result, expected):
sleep = random.random() * 5
time.sleep(sleep)
result["internal_success"] = random.random() > 0.5
result["perfect_extraction"] = random.random() > 0.5
return result
def create_global_metrc(metric_df):
print(len(metric_df)) # 50
print(metric_df[["internal_success", "perfect_extraction"]].dtypes) # bool, bool
confusion_matrix = {
"Correct": {
"TP": len(metric_df[metric_df["internal_success"] & metric_df["perfect_extraction"]]),
"FN": len(metric_df[~metric_df["internal_success"] & metric_df["perfect_extraction"]]),
},
"Incorrect": {
"FN": len(metric_df[metric_df["internal_success"] & ~metric_df["perfect_extraction"]]),
"TN": len(
metric_df[~metric_df["internal_success"] & ~metric_df["perfect_extraction"]]
),
},
}
print(confusion_matrix) # {'Correct': {'TP': 11, 'FN': 10}, 'Incorrect': {'FN': 17, 'TN': 12}}, but it printed {'Correct': {'TP': 11, 'FN': 10}, 'Incorrect': {'FN': 17, 'TN': 50}}, where 50 is incorrect
ids = [f"invoice_{i}" for i in range(1, 51)]
expected = [True] * 50
metrics = []
results = []
with ThreadPoolExecutor(max_workers=25) as extraction_executor, ThreadPoolExecutor(
max_workers=25
) as metrics_executor: # Create two separate thread pools - one for extractions and one for metrics
extraction_futures = {
extraction_executor.submit(get_result, id): id for id in ids # First submit all extraction tasks
}
metrics_futures = []
for future in as_completed(extraction_futures): # Then submit metrics tasks as extractions complete
result = future.result()
invoice_id = extraction_futures[future]
results.append(result)
metrics_futures.append(
metrics_executor.submit(
get_single_metric, # fill metric fields of result
result,
expected
)
)
for future in as_completed(metrics_futures): # Collect all metrics results
metrics.append(future.result())
metric_df = pd.DataFrame.from_dict(metrics)
results_df = pd.DataFrame.from_dict(results)
global_metric = create_global_metrc(metric_df)
`
Comment From: rhshadrach
Unfortunately, you are correct meant to showcase that as the pseudo code if someone comes across the same issue
Thanks for clarifying. In the future, please make this more clear when you open an issue as maintainers spend time reproducing bugs that are reported.
I would suggest to try to capture the output of print(bool_df.dtypes) on your real data if possible, it may shed some light on what's going on.
I understand that it can be difficult to find a reproducer, but unfortunately we cannot do anything until one is produced. So closing for now. If you are able to come up with a reproducible example, post it here and we'll be glad to reopen!