Describe the feature

Allow protobuf messages to be stored in the database via GORM without implementing the Scanner/Valuer interface.

This can be done by invoking the proto package's Marshal and Unmarshal functions passing the Go type (msg below) that are generated from a .proto file.

msg := &timestamppb.Timestamp{}
bytes := proto.Marshal(msg)
proto.Unmarshal(bytes, msg)

Motivation

GORM can use the Scanner/Valuer interfaces to marshal/unmarshal Go types, including generated protobuf messages.

This requires access to the generated protobuf message so that the corresponding Scan and Value methods can be placed alongside the generated Go type; see Timestamp example below.

However, certain well-known protobuf types are part of the google.golang.org/protobuf library and thus it isn't practical to augment the types with Scan/Value methods. One example of such a well-known type is the timestamppb.Timestamp type.

// Scan unmarshals value of type []byte from GORM into ts.
func (ts *Timestamp) Scan(value interface{}) error {
    return proto.Unmarshal(value.([]byte), ts)
}

// Value marshals ts into []byte value to be stored using GORM.
func (ts *Timestamp) Value() (driver.Value, error) {
    return proto.Marshal(ts)
}

How can this be implemented?

A protobuf generated message always implements the proto.Message interface, so it's possible to do this:

func Marshal(value interface{}) ([]byte, error) {
    if msg, ok := value.(proto.Message); ok {
        return proto.Marshal(msg)
    }
    return nil, errors.New("not a proto msg")
}

func Unmarshal(bytes []byte, m proto.Message) error {
    return proto.Unmarshal(bytes, m)
}

It may be desirable to only invoke the above marshal/unmarshal functions for fields that are tagged with e.g., gorm:proto.

Related Issues

Not sure, but #4438 seems to be related.

I would be happy to contribute a PR for this feature, given some directions where it would be appropriate to add this.

Comment From: github-actions[bot]

This issue has been automatically marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days

Comment From: nnutter

I just ran into this issue. Obviously, one can decouple the DB model from the protobuf generated model but it doesn't seem this should be required solely due to using GORM.

Comment From: nnutter

A related issue, repeated fields with custom data types, does not seem solvable due to not being able to define receiver methods on slices.

Comment From: oxisto

Since I also ran into this issue. After several different approaches, it seems that what works is to register an additional GORM serializer (https://gorm.io/docs/serializer.html) that converts timestamppb.Timestamp into a time.Time, which the sql package can handle.

Then annotate the respective field in the proto file with GORM tags, e.g., using github.com/srikrsna/protoc-gen-gotag

Comment From: oxisto

Since I also ran into this issue. After several different approaches, it seems that what works is to register an additional GORM serializer (https://gorm.io/docs/serializer.html) that converts timestamppb.Timestamp into a time.Time, which the sql package can handle.

Then annotate the respective field in the proto file with GORM tags, e.g., using github.com/srikrsna/protoc-gen-gotag

If anyone is interested, we implemented this in our project and it might be re-usable for your needs @nnutter https://pkg.go.dev/clouditor.io/clouditor@v1.4.16/persistence/gorm#TimestampSerializer

Comment From: zchenyu

Another issue is that oneofs turn into interfaces: https://developers.google.com/protocol-buffers/docs/reference/go-generated#oneof