Generic Kafka¶
Cyderes has developed a generic Kafka collector to support the ingestion of data from log sources that support Kafka.
Requirements¶
Kafka Setup¶
Below are some guides from common cloud providers on how to leverage Kafka within their environments
There is also the ability to have your own environment outside of a managed Kafka through a cloud provider, more information on that can be found here in the Kafka configuration documentation.
Authentication¶
Authentication can be handled with a few methods that Cyderes' generic Kafka collector supports
- Plain auth
- scram256
- scram512
The collector also supports the use of TLS certs for connecting to a Kafka topic.
Kafka Components¶
The following components are required to be set up in your environment for ingestion to work via Cyderes' generic Kafka ingestion method. Below is the Apache Kafka documentation for each component, if you are deploying with a cloud provider please be sure to follow their docs as component names may be different.
Cyderes will need your Kafka Broker(s) to send from your producer(s) to our consumer, the generic Kafka integration. Information on setting up Kafka Brokers can be found on the Kafka Doc Site.
As well Cyderes will need your subscription topic, documentation on subscription topics can also be found on the Kafka doc site.
Finally, we will need your Consumer group ID.
Kafka Architecture¶
Below is a diagram showing an example Kafka set up to allow Cyderes to pull logs via our Generic Kafka Integration
Gather Information¶
Please provide the following information to Cyderes in order to complete the integration:
- Kafka Broker(s)
- Subscription Topic(s)
- Username and Password
- Consumer Group ID(s) or Partition number(s) (optional, by default we will read at partition 0 unless specified otherwise)
- TLS Cert (Optional)