Pandas version checks

  • [x] I have checked that this issue has not already been reported.

  • [x] I have confirmed this bug exists on the latest version of pandas.

  • [x] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

Edit [rhshadrach]: The code below does not reproduce the issue.

# No idea how to exactly reproduce it, but it occurs sometimes. Logic is this:
import pandas as pd
bool_df = pd.DataFrame([
    {"first": True, "second": False, "third": True},
    {"first": True, "second": True, "third": True},
    {"first": True, "second": False, "third": True},
    {"first": True, "second": True, "third": True},
    {"first": True, "second": True, "third": True},
    {"first": True, "second": False, "third": True},
    {"first": True, "second": True, "third": True},
    {"first": False, "second": False, "third": True},
    {"first": True, "second": True, "third": True},
    {"first": True, "second": False, "third": True},
])
bool_df = bool_df[bool_df["third"]][["first", "second"]]
# In some cases, this line prints the length of the DataFrame (10)
print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10

# This line prints the expected output (1)
print(len(bool_df[(bool_df["first"] == False) & (bool_df["second"] == False)])) # Prints 1

# Using De Morgan's law also returned with the expected output
print(len(bool_df[~((bool_df["first"]) | (bool_df["second"]))])) # Prints: 1

Issue Description

We don't know when and why this occurs. We werre looking for any rational explanation for hours. Anyone else experienced similar? How could this be possible? (Environment: MacBook Pro 2023, Sequoia 15.3)

Expected Behavior

print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Print 1

Installed Versions

INSTALLED VERSIONS ------------------ commit : 0691c5cf90477d3503834d983f69350f250a6ff7 python : 3.13.1 python-bits : 64 OS : Darwin OS-release : 24.3.0 Version : Darwin Kernel Version 24.3.0: Thu Jan 2 20:24:23 PST 2025; root:xnu-11215.81.4~3/RELEASE_ARM64_T6031 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8 pandas : 2.2.3 numpy : 2.2.3 pytz : 2025.1 dateutil : 2.9.0.post0 pip : 24.3.1 Cython : None sphinx : None IPython : 9.0.1 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 blosc : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None html5lib : None hypothesis : None gcsfs : None jinja2 : 3.1.5 lxml.etree : 5.3.1 matplotlib : 3.10.1 numba : None numexpr : None odfpy : None openpyxl : 3.1.5 pandas_gbq : None psycopg2 : None pymysql : None pyarrow : 19.0.1 pyreadstat : None pytest : 8.3.5 python-calamine : None pyxlsb : None s3fs : None scipy : 1.15.2 sqlalchemy : 2.0.38 tables : None tabulate : 0.9.0 xarray : None xlrd : None xlsxwriter : None zstandard : 0.23.0 tzdata : 2025.1 qtpy : None pyqt5 : None

Comment From: snitish

I'm unable to reproduce this behavior both on 2.2.3 as well as dev. @adamszelestey-se do you have any stats on how often this line prints 10?

print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10

Comment From: rhshadrach

  • How are you running this? A Python script, a notebook like Jupyter, IPython? If a notebook/IPython, are you restarting the kernel between runs?
  • Can you add print(bool_df.dtypes) on the line prior to the first print. What is the output of this when the unexpected result occurs?

Comment From: adamszelestey-se

I'm unable to reproduce this behavior both on 2.2.3 as well as dev. @adamszelestey-se do you have any stats on how often this line prints 10?

print(len(bool_df[(~bool_df["first"]) & (~bool_df["second"])])) # Sometimes prints 10

Yeah unfortunately it is really unpredicable for us too, but it occured pretty often, I would say 1 out of 2 cases. The code is not exactly the one I provided, but the logic shouldn't differ.

  • How are you running this? A Python script, a notebook like Jupyter, IPython? If a notebook/IPython, are you restarting the kernel between runs?
  • Can you add print(bool_df.dtypes) on the line prior to the first print. What is the output of this when the unexpected result occurs?

We were running it as a regular python module in a poetry environment.

I wanted to create the ticket if someone else also experience something like this, they could add more context, I can understand that you can not do anything with this until we figure out how to reproduce it.

Comment From: rhshadrach

The code is not exactly the one I provided, but the logic shouldn't differ.

Am I correct in saying that you can never reproduce the bug with the code example that's in the OP?

Comment From: adamszelestey-se

The code is not exactly the one I provided, but the logic shouldn't differ.

Am I correct in saying that you can never reproduce the bug with the code example that's in the OP?

Unfortunately, you are correct meant to showcase that as the pseudo code if someone comes across the same issue

This is the best I can give you, but can't reproduce the error at the moment: `

from concurrent.futures import as_completed, ThreadPoolExecutor
import pandas as pd
import time
import random

random.seed(42)

def get_result(_id: str):
    result = {}
    sleep = random.random() * 5
    time.sleep(sleep)
    result["id"] = _id
    return result


def get_single_metric(result, expected):
    sleep = random.random() * 5
    time.sleep(sleep)
    result["internal_success"] = random.random() > 0.5
    result["perfect_extraction"] = random.random() > 0.5
    return result


def create_global_metrc(metric_df):
    print(len(metric_df)) # 50
    print(metric_df[["internal_success", "perfect_extraction"]].dtypes)  # bool, bool

    confusion_matrix = {
        "Correct": {
            "TP": len(metric_df[metric_df["internal_success"] & metric_df["perfect_extraction"]]),
            "FN": len(metric_df[~metric_df["internal_success"] & metric_df["perfect_extraction"]]),
        },
        "Incorrect": {
            "FN": len(metric_df[metric_df["internal_success"] & ~metric_df["perfect_extraction"]]),
            "TN": len(
                metric_df[~metric_df["internal_success"] & ~metric_df["perfect_extraction"]]
            ),
        },
    }
    print(confusion_matrix)  # {'Correct': {'TP': 11, 'FN': 10}, 'Incorrect': {'FN': 17, 'TN': 12}}, but it printed {'Correct': {'TP': 11, 'FN': 10}, 'Incorrect': {'FN': 17, 'TN': 50}}, where 50 is incorrect

ids = [f"invoice_{i}" for i in range(1, 51)]
expected = [True] * 50

metrics = []
results = []


with ThreadPoolExecutor(max_workers=25) as extraction_executor, ThreadPoolExecutor(
    max_workers=25
) as metrics_executor: # Create two separate thread pools - one for extractions and one for metrics

    extraction_futures = {
        extraction_executor.submit(get_result, id): id for id in ids     # First submit all extraction tasks
    }


    metrics_futures = []
    for future in as_completed(extraction_futures):    # Then submit metrics tasks as extractions complete
        result = future.result()
        invoice_id = extraction_futures[future]
        results.append(result)

        metrics_futures.append(
            metrics_executor.submit(
                get_single_metric,  # fill metric fields of result
                result,
                expected
            )
        )


    for future in as_completed(metrics_futures):     # Collect all metrics results
        metrics.append(future.result())

metric_df = pd.DataFrame.from_dict(metrics)
results_df = pd.DataFrame.from_dict(results)
global_metric = create_global_metrc(metric_df)

`

Comment From: rhshadrach

Unfortunately, you are correct meant to showcase that as the pseudo code if someone comes across the same issue

Thanks for clarifying. In the future, please make this more clear when you open an issue as maintainers spend time reproducing bugs that are reported.

I would suggest to try to capture the output of print(bool_df.dtypes) on your real data if possible, it may shed some light on what's going on.

I understand that it can be difficult to find a reproducer, but unfortunately we cannot do anything until one is produced. So closing for now. If you are able to come up with a reproducible example, post it here and we'll be glad to reopen!