Kafka Topics from config

Akka
1

I manage producer kafka settings in application.conf. Is there a way to add topics to the settings?

This topic is based on a conversation between an anonymous user of Akka and our “Ask AI about Akka” bot in the documentation.

2

Yes, you can add topics to your settings in the application.conf file. However, the topic is not usually specified in the producer settings but rather when you send a message. In Akka, you typically specify the topic when creating a message to send via the producer.

Here is an example of how you might do this in Scala:

val producerSettings =
  ProducerSettings(config, new StringSerializer, new StringSerializer)
    .withBootstrapServers(bootstrapServers)

val done: Future[Done] =
  Source(1 to 100)
    .map(number => new ProducerRecord[String, String]("topic-name", number.toString))
    .runWith(Producer.plainSink(producerSettings))

In this example, “topic-name” is the name of the Kafka topic to which the messages are being sent.

Please note that the topic is not a part of the producer settings but is specified when sending a message. The producer settings are used to configure how the producer interacts with the Kafka cluster, such as specifying the bootstrap servers and serializers for the keys and values.

For more details, you can refer to the Alpakka Kafka Producer documentation.

3

I would like the topic names to come from configurations.

4

If you want to specify the topic names in your configuration file, you can add them to your application.conf file and then reference them in your code. Here’s an example of how you might do this:

In your application.conf:

akka {
  kafka {
    producer {
      kafka-clients {
        // your existing settings...
      }
      topics = ["topic1", "topic2", "topic3"]
    }
  }
}

Then in your code, you can retrieve the list of topics:

val config = system.settings.config.getConfig("akka.kafka.producer")
val topics: List[String] = config.getStringList("topics").asScala.toList

Now you have a list of topics that you can use when sending messages. Remember that you need to choose the appropriate topic when sending a message based on your application’s logic.

Please note that this is a general approach and might need to be adjusted based on your specific use case and environment.

5

Note that application specific config should not generally be defined under the akka namespace but a namespace for your own application, so:

akka {
  ...
}

my-service {
  my-subsystem-a {
    topics = ["topic1", "topic2", "topic3"]
  }
}