Apache Kafka is often used together with Confluent Schema Registry, as the schema registry allows you to store and retrieve your Avro, JSON Schema and Protobuf schemas for Kafka message (de)serialization. By storing a versioned history of schemas for topic values, with configurable enforced compatibility, you ensure producers and consumers can continue to exchange compact serialized messages even as schemas evolve.

As more and more teams and companies adopt Apache Kafka, you can find yourself wanting to share data via replication of one or more topics from one cluster to another. While replication of an entire cluster with all of it’s topics as a means of failover can be achieved with tools such as Mirror Maker and Confluent Replicator, for replication of a single topic there are fewer examples. Even more so when the source topic is serialized with Avro, with the schema stored in Confluent Schema Registry.

Here we present a minimal consumer that replicates a single Avro serialized Kafka topic from one cluster to another, while ensuring (only) the necessary Avro schema is registered in the target cluster Schema Registry.

The December 2019 release of Apache Kafka 2.4.0 saw usability improvements in TopologyTestDriver, with the addition of new TestInputTopic and TestOutputTopic classes. These offer typesafe methods to easily produce and consume messages for your Kafka Streams tests. In this post we’ll explore these new classes in the context of Avro messages, which requires a small trick to get working.