Gorm [BUG] strings.(*Builder).WriteByte memory leak in .FirstOrCreate with PrepareStmts: true setting

GORM Playground Link

Got no time for it, but I'll think about trying it if I have some free time.

Description

Hello there, and first of all, I'm sorry to bother you 🙇‍♂️.

Golang version: 1.20 GORM versions:

gorm.io/driver/mysql v1.4.4
gorm.io/gorm v1.24.2
gorm.io/datatypes v1.0.7

Database spec: AWS Aurora RDS 5.7.mysql_aurora.2.11.4

After we put in production the service in trouble and the service gained some decent amount of traffic, we experienced a memory leak that we didn't figure out from where it was coming from, and it was driving our k8s pods to crash by OOMKilled.

Screenshot 2024-03-07 at 14 03 09

We used pprof profiling to get a clue, and surprisingly, all the arrows where pointing at gorm.

Graph: Screenshot 2024-03-07 at 13 41 39

Flame Graph: Screenshot 2024-03-07 at 14 00 12 Screenshot 2024-03-07 at 13 43 10

Looks like it is related to .FirsOrCreate calls that internally use strings.(*Builder).WriteByte when PrepareStmts gorm setting flag is on.

    conn, err := gorm.Open(
        mysql.Open(vars.InstancesMySQLDSN),
        &gorm.Config{
            PrepareStmt: true,
            Logger: logger.Default.LogMode(logger.Silent),
        })
    if err != nil {
        return nil, err
    }

    db, err := conn.DB()
    if err != nil {
        return nil, err
    }

    db.SetMaxIdleConns(45)
    db.SetMaxOpenConns(90)
    db.SetConnMaxLifetime(time.Hour)
    db.SetConnMaxIdleTime(time.Minute)

m.db.WithContext(ctx).Table(m.tableName).FirstOrCreate(
    &mysqlSnap,
    "campaign_id = ? AND user_id = ? AND descriptor_id = ?",
    snap.CampaignID.String(), snap.UserID.String(), snap.DescriptorID.String(),
)

note: the 3 fields are string type (VARCHAR).

Although we use same settings in other services using FirstOrCreate call that are not having the leak issue, but must add, those are not as data intensive as the one in trouble (~8 Req/s during prime time) ...

Once removed the PrepareStmts flag and deployed, memory leak gone.

Screenshot 2024-03-07 at 13 47 48

Tried to reproduce in our staging environment without success, tho (~21 Req/s). But maybe I didn't test it right, since I only stressed one of the tables of the 3 endpoints that use different table each one. 🤔

I Hope there is enough information to start scratching your head. Let me know if you need missing data I can provide. I'll keep you posted if get something new about it.

Comment From: github-actions[bot]

The issue has been automatically marked as stale as it missing playground pull request link, which is important to help others understand your issue effectively and make sure the issue hasn't been fixed on latest master, checkout https://github.com/go-gorm/playground for details. it will be closed in 30 days if no further activity occurs. if you are asking question, please use the Question template, most likely your question already answered https://github.com/go-gorm/gorm/issues or described in the document https://gorm.io ✨ Search Before Asking ✨

Comment From: github-actions[bot]

Comment From: philhuan

Whether the FirstOrCreate method call has an assigned a another value? @qfornaguera

Comment From: philhuan

Can you see the number of go coroutines?

Comment From: github-actions[bot]