Support writing serialized protobuf objects

[nurs-lr]

Hi @mostafa!
How hard is it to add support writing to kafka serialized protobuf objects?

export default function () {

  // build Protobuf object
  const protobufObject = new protobufs.ProtobufObject();
  protobufObject.setIdentifier(identifier);
  protobufObject.setEvent(eventSchema);

  // Write to kafka
  const error = produce(producer, signalEnvelope.serializeBinary()); // xk6 support only []map[string]string
  check(error, {
    'is sent': (err) => err == undefined,
  });

The Error is:

ERRO[0001] could not convert 10,12,10,10,97,117,116,111,109,97,116,105,111,110,58,175,2,10,51,10,35,103,101,110,101,114,97,108,95,105,110,102,111,114,109,97,116,105,111,110,46,108,111,103,95,115,111,117,114,99,101,46,110,97,109,101,18,12,85,110,105,100,101,110,116,105,102,105,101,100,10,70,10,30,103,101,110,101,114,97,108,95,105,110,102,111,114,109,97,116,105,111,110,46,109,101,115,115,97,103,101,95,105,100,18,36,98,102,97,48,51,101,57,53,45,49,97,102,100,45,52,51,52,53,45,57,97,102,55,45,51,102,48,102,99,49,51,53,97,102,50,101,10,54,10,31,103,101,110,101,114,97,108,95,105,110,102,111,114,109,97,116,105,111,110,46,114,97,119,95,109,101,115,115,97,103,101,18,19,78,101,109,111,45,87,114,105,116,101,45,84,111,45,75,97,102,107,97,10,75,10,41,103,101,110,101,114,97,108,95,105,110,102,111,114,109,97,116,105,111,110,46,115,116,97,110,100,97,114,100,95,109,101,115,115,97,103,101,95,116,105,109,101,18,30,50,48,50,49,45,48,54,45,51,48,84,49,52,58,53,57,58,52,55,46,54,56,53,49,52,50,50,48,48,90,10,43,10,29,103,101,110,101,114,97,108,95,105,110,102,111,114,109,97,116,105,111,110,46,116,101,110,97,110,116,95,105,100,18,10,97,117,116,111,109,97,116,105,111,110 to []map[string]string
        at go.k6.io/k6/js/common.Bind.func1 (native)

[mostafa]

Hey @nrsalt,

It shouldn't be that hard. There are two functions to help produce serialized and consume and deserialize Avro message:

1. ToAvro used the producer in key and value to serialize messages from JSON object to Avro and then send them to Kafka.
2. FromAvro used in consumer in key and value to deserialize messages from Avro to a JSON object after reading it from Kafka.

It should probably be refactored with a Go interface that gets implemented by many de/serialization libraries/functions, so as to allow binary, JSON, Avro and possibly Protobuf.

You can write your own de/serializer with protobuf and send a PR. Just keep in mind that some functions are used by Avro to make it work with Schema Registry like addMagicByteAndSchemaIdPrefix and removeMagicByteAndSchemaIdPrefix.

[nurs-lr]

Thank you @mostafa!

Unfortunately, I'm not familiar with Golang at all. Is there a chance to request this kind of enhancement?

Just FYI, producer configuration that I'm looking for:

    Properties props = new Properties();
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

[AbdelrhmanHamouda]

Having protobuf serdes would be really appreciated. Unfortunately, i don't know "go" at all but it would be a really nice feature to push this plugin completeness further.

[mostafa]

@AbdelrhmanHamouda
I am all for it, but if I can find some time to do it, I definitely will.

[thmshmm]

Hey, I have added some code which can be used to serialize Protobuf data.
When using JS Protobuf objects the following code Array.from(data.serializeBinary()) can be used to create the message value with the ByteArray serializer.

Note: The serializeBinary function generates uint8array which I think goja transforms to a map[string]interface{} type and not to a byte array.