Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame({"string_values": ["1", "1.0", "1.1"]})
df.string_values.str.isnumeric()
Issue Description
The series method .isnumeric() only works on integer strings. If a string number is decimal, it will return False. When running the example below, the following is returned:
This is the docs description for the method:
Expected Behavior
Running the method on decimal strings should return True.
Installed Versions
Comment From: akj2018
Hi @sf-dcp,
This is not a bug in pd.Series.isnumeric() but expected behavior. As per documentation for str.isnumeric(), . (decimal point) is not a numeric character, so strings with decimal points like "1.1" fail the isnumeric() check.
str.isnumeric() method checks if:
- All characters in the string are numeric characters.
- There is at least one character in the string.
Numeric characters are those with the Unicode property values: - Numeric_Type=Digit (e.g., "1", "2", "3") - Numeric_Type=Decimal (e.g., standard decimal digits like "1", "2") - Numeric_Type=Numeric (e.g., fractions like "⅕", Roman numerals, etc.)
Therefore, . (decimal character) does not have the Unicode numeric property; it's categorized as punctuation.
Solution 01 - Using regex
import re
def is_numeric(string):
pattern = r'^-?\d+(\.\d+)?$'
return bool(re.match(pattern, string))
Note:
- correctly identifies valid numeric strings like "1", "-2.5" and "0.0".
- rejects invalid strings like "1.0.0", "abc", or empty strings.
- Does not works for scientific notations, "1e10" returns False
Solution 02 - Using float()
def is_numeric(string):
try:
float(string)
return True
except ValueError:
return False
Note: - correctly identifies valid numeric strings like "1", "-2.5", "1e10" (scientific notation), and "0.0" - rejects invalid strings like "1.0.0", "abc", or empty strings.
Performance Consideration
- Regex: Slower due to the regex engine's overhead of parsing and matching patterns.
- Float-based: Faster, as it directly leverages Python’s built-in C-optimized parsing.
# Test dataset (mix of numeric and non-numeric strings)
data = ["1", "123.45", "-987.65", "0", "abc", "123.45.67"] * 1_000_000
- Regex time: 5.59 seconds
- Float time: 2.23 seconds
Comment From: sf-dcp
Hi @akj2018, thanks for the prompt reply and suggestions. I think the method name is a bit misleading then where I would expect a string number (with a dot or a negative integer string) to be returned True. But I also understand the logic behind it. At a minimum, I would recommend updating the method docs with a note or warning calling out the method's behavior when it comes to decimal numbers and/or negative numbers. Again, thanks for clarifying.
Comment From: rhshadrach
Thanks for the report! This mirrors Python behavior:
print("1.2".isnumeric())
# False
The first line of the docstring states:
Check whether all characters in each string are numeric.
Are you saying there might be some confusion as to whether . is a numeric character?
Comment From: sf-dcp
@rhshadrach, yep, I may be biased here but I can interpret . being numeric like in a decimal number :) I believe an explicit message about numeric values other than integers (negative and decimal string values) would be beneficial for at least a portion of pandas users.
Comment From: rhshadrach
I'd be okay with this addition. PRs are welcome!
Comment From: akj2018
take
Comment From: JyotiP24
Hi, I'd like to take on this issue and contribute to improving the documentation to explicitly clarify the behavior of pd.Series.str.isnumeric() regarding decimal and negative numbers. I'll ensure the updated docs provide clear examples and explanations for users. Please let me know if there are any additional points you'd like me to cover.
Comment From: akj2018
Hi @JyotiP24 , I am currently working on the issue and will be making a PR tomorrow. Thanks
Comment From: prakhar-bip
what if we try this code ?
def is_numeric(string):
if (string.isnumeric()):
return True;
else:
try:
float(string)
return True;
except ValueError:
return False
Comment From: arnav1209
I noticed that pd.Series.isnumeric() currently returns False for '3.14' due to the behavior of Python’s str.isnumeric(). Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?
Comment From: rhshadrach
Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?
No, pandas' isnumeic should match Python's isnumeric.
Comment From: pranavgudagi
Is this issue still open i want to contribute to this issue .Please assign me this issue .
Comment From: akj2018
Hi @pranavgudagi , I am currently working on the issue. Thanks
Comment From: WalkingDevFlag
@forbid_nonstring_types(['bytes'])
def isnumeric(self, extended=False):
if extended:
regex = r'^[+-]?(\d+([.]\d*)?([eE][+-]?\d+)?|[.]\d+([eE][+-]?\d+)?)$'
return self._data.str.match(regex, na=False)
return self._data._str_map(
fname="isnumeric", dtype=np.dtype("bool"), na_value=False
)
would this be a good solution to handling decimal strings?
Comment From: Varun270
@akj2018 Are You still working on this?
Comment From: Anurag-Varma
@arnav1209
I noticed that pd.Series.isnumeric() currently returns False for '3.14' due to the behavior of Python’s str.isnumeric(). Would the intended fix be to have isnumeric() return True for strings that represent valid decimal numbers? Also, should we extend this to handle signs (like -3.14) or scientific notation (like 1e3), or keep it simple?
As mentioned, we should not change the code, this bug is a documentation fixing issue
No, pandas'
isnumeicshould match Python'sisnumeric.
Comment From: Anurag-Varma
Hey @akj2018
It's been 2 weeks, can you confirm if you are still working on it or if I can take the issue ?
Comment From: akj2018
Hi @Anurag-Varma,
Yes, I am still working on it. Due to health related concerns, I was not able to complete it. I will make a PR today without further delay.
In future, please me mindful to wait for a while before taking an action.
Thanks
Comment From: Anurag-Varma
Hi @Anurag-Varma,
Yes, I am still working on it. Due to health related concerns, I was not able to complete it. I will make a PR today without further delay.
In future, please me mindful to wait for a while before taking an action.
Thanks
It's been 3 weeks since your last update on PR and 2 weeks ago you have mentioned that you are still working. I didn't find any updates on your remote repository in your account related to this issue so I just took it. But if you are still working then you can continue and send a PR.
Comment From: akj2018
@Anurag-Varma Yes, because I haven't updated the remote with my local changes.
Kindly refer the contribution guide in such situations, which mentions to ask the assigne along with period of inactivity.
https://pandas.pydata.org/docs/development/contributing.html#finding-an-issue-to-contribute-to
Rest assured that I will making a PR today so there is no further delay.