df = pd.DataFrame(
{
"Timestamp": [
1565083561,
1565083561 + 86400,
1565083561 + 86500,
1565083561 + 86400 * 2,
1565083561 + 86400 * 3,
1565083561 + 86500 * 3,
1565083561 + 86400 * 4,
],
"Food": ["apple", "apple", "banana", "banana", "orange", "orange", "pear"],
}
).drop([3])
df["Datetime"] = to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")
gb = df.groupby(pd.Grouper(freq="1D", key="Datetime"))
print(gb.value_counts())
raises ValueError: operands could not be broadcast together with shapes (5,) (6,) (5,)
Comment From: AfolabiOlaoluwa
I don't know what your environment looks like, but this is what I did to investigate the problem.
It appears you are passing in an object for value_counts(). You CAN'T supply in a Class Object to value_counts() function
df["Datetime"] = pd.to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")
# note that I had to pass in the pd constructor to "to_datetime"
# print(df["Datetime"]) # Note that printing this returned properly
gb = df.groupby(pd.Grouper(freq="1D", key="Datetime"))
# Note that the response was <pandas.core.groupby.generic.DataFrameGroupBy object at 0x3e58080>
# print(df.groupby(pd.Grouper(freq="1D", key="Datetime")))
If you look at the code below, it returns a TimeGrouper Class
print(pd.Grouper(freq="1D", key="Datetime"))
>>> TimeGrouper(key='Datetime', freq=<Day>, axis=0, sort=True, closed='left', label='left', how='mean', convention='e', origin='start_day')
If you use a dot notation on the response, you can group by any item from the response. e.g. pd.Grouper(freq="1D", key="Datetime").key, which picks the value for the "key" which is "Datetime".
When you call .value_counts() on that, you will have an output. See the code below and the output:
veee = df.groupby(pd.Grouper(freq="1D", key="Datetime").key)
print(veee) # This outputs an object class which you are passing to value_counts()
print(veee.value_counts())
Your output will be:
# print(veee)
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x4308910>
# print(veee.value_counts())
Datetime Timestamp Food
2019-08-06 09:26:01 1565083561 apple 1
2019-08-07 09:26:01 1565169961 apple 1
2019-08-07 09:27:41 1565170061 banana 1
2019-08-09 09:26:01 1565342761 orange 1
2019-08-09 09:31:01 1565343061 orange 1
2019-08-10 09:26:01 1565429161 pear 1
dtype: int64
I hope this leads you to fixing the problem.
You may look more in detail about .groupby function here.
Also you can look at the value_counts() function here
Comment From: AfolabiOlaoluwa
Also, note that if you leave your code untouched, you can .count() and .size().
Comment From: rhshadrach
Thanks @AfolabiOlaoluwa - however the code in the top comment (without any modification) should not raise an error.
Comment From: AfolabiOlaoluwa
I agree it shouldn't fail!
It looks like TimeGrouper is only imported when freq is not None and then calls super() function on it to access its components.
If we look at "new" function under grouper.py, when freq is absent (None) in the Grouper class, it works fine, and your solutions work fine without freq, but as a Grouper class.
import pandas as pd
import numpy as np
df = pd.DataFrame(
{
"Timestamp": [
1565083561,
1565083561 + 86400,
1565083561 + 86500,
1565083561 + 86400 * 2,
1565083561 + 86400 * 3,
1565083561 + 86500 * 3,
1565083561 + 86400 * 4,
],
"Food": ["apple", "apple", "banana", "banana", "orange", "orange", "pear"],
}
).drop([3])
df["Datetime"] = pd.to_datetime(df["Timestamp"].apply(lambda t: str(t)), unit="s")
gb = df.groupby(pd.Grouper(key="Datetime"))
print(gb.value_counts())
output:
Datetime Timestamp Food
2019-08-06 09:26:01 1565083561 apple 1
2019-08-07 09:26:01 1565169961 apple 1
2019-08-07 09:27:41 1565170061 banana 1
2019-08-09 09:26:01 1565342761 orange 1
2019-08-09 09:31:01 1565343061 orange 1
2019-08-10 09:26:01 1565429161 pear 1
dtype: int64
Its output is not different from the solution with df.groupby(pd.Grouper(freq="1D", key="Datetime").key) but as a Grouper instead of TimeGrouper
So, I am worried about why it looks like freq is empty in the TimeGrouper as freq=<Day>.
TimeGrouper(key='Datetime', freq=<Day>, axis=0, sort=True, closed='left', label='left', how='mean', convention='e', origin='start_day')