Glossary¶
What is Chronicle¶
The Chronicle platform allows security teams to cost effectively store and analyze all of their security data in one place to investigate and detect threats at Google speed and scale, all based in Google Cloud. Chronicle disrupts the traditional models constrained by computational sizing, volume limits, and inability to normalize data from disparate platforms.
What are Integrations¶
An integration is a mechanism for getting data from a log source into a Cyderes-supported security platform. Cyderes currently ingests data into Google Chronicle and Azure Sentinel as outputs for Cyderes integrations.
Cyderes Integration Features¶
Flexing Destinations¶
The Cyderes pipeline is able to support sending logs to one or many supported destinations. For example, if logs need to be in both Chronicle and a longer term cold storage option, Cyderes can support that use case. For more information around long term cold storage options, please consult a Cyderes customer success manager.
Reliability¶
Cyderes leverages cursors to track state of ingestion whenever possible. This ensures delivery of logs only once and allows state to be tracked in case of failure scenarios so that no logs are missed and that destinations are not spammed with duplicate data.
Prepare to Send Data¶
Depending on the type of data source and its capabilities, a data source's logs can be ingested in different ways such as via a local syslog forwarder, GCP or S3 buckets, flat files, packet captures, Splunk queries, and even a managed ingestion service. These options allow for incredible flexibility in design choices when planning how to get data to a telemetry ingestion destination such as Chronicle. Please review a data sources' corresponding integration guide for preferred and optional paths for gathering data.
Events versus Context/Metadata¶
There are two basic types of data that Cyderes collects.
Event¶
This includes all events such as logs, telemetry, metrics, or alerts. This data will typically have a timestamp representing the time of occurrence or measurement. These are the most common integrations as context/metadata typically does not hold high value without data to enrich.
Context/Metadata¶
Context data is typically not time sensitive like events are and can be used to enrich ingested data. Examples include: IoCs, user metadata, and asset metadata. Support for enrichment is dependent on the security platform. For this example, Chronicle currently supports user context and aliasing for the following data sources: Azure AD, Okta, Duo, Google Workspace Users, and Workday.
Data Types and Parsing¶
Data type is a concept leveraged within Chronicle and relates to parsing. The Data type for a data source should be determined by data structure and content rather than product type or format. This allows for parsers to be short and not complex.
Integration Types¶
TCP/TLS/UDP¶
Syslog is the most common form of this integration and is accomplished via Cyderes CYCLOPS. Network layer 4 (TCP/TLS/UDP) is leveraged to push data. In the current state, this expects telemetry to be new line delimited but does not actually require data to follow syslog format.
Caveats¶
- Cyderes leverages a syslog forwarder that is meant to be more modern. Instead of having to vertically scale it to support voluminous sources, it is meant to scale out horizontally. That means that ideally products that send syslog data will get the best performance if it supports sending data over multiple TCP connections.
- UDP offers no guarantee of reliability and should not be used for critical data sources. Due to no formal connection actually established, data loss is impossible to detect and data corruption cannot be corrected.
API¶
This type of integration calls an API to retrieve data. This can include a REST API that must be polled (ex: Microsoft Graph API), a message bus (ex: Apache Kafka, AWS SQS/SNS, GCP Pubsub, or Microsoft Azure Eventhub), or cloud storage (ex: AWS S3, GCS, or Azure Blob Storage). These typically come in two different forms: stream or poll. Pollers are the more common form and require Cyderes to periodically reach out to the API and grab data within a time range. Streamers are less common and stream data to the integration once connected and are similar to syslog but instead rely on network layer 7.
Caveats¶
- Not all APIs will have the same expected reliability due to the way they were created. Cyderes will attempt to work with an API as best as possible, but there could potentially be limitations on what may be possible to fix. In this case, a partnership between the client and data source vendor or between Cyderes and data source vendor may be leveraged in an attempt to request updates to an API.
Webhook¶
Similar to API integrations, this leverages the HTTP protocol and REST to send data. However, instead of the integration pulling from a source, the product is expected to push data to the integration. This is similar to syslog but at a higher network layer (layer 7). Typically, JSON is expected but other formats could be supported as needed.
Data Agnostic Integrations¶
Many Cyderes integrations are data type agnostic meaning they support any type of data (with exceptions). For example, the Cyderes forwarder that is used for TCP/UDP integrations can support any data that comes through it, not just syslog. The requirement is that it must leverage new line characters to delimit the end of a single event (this being an exception noted above). AWS S3 or GCS are other prime examples but are typically more complex than TCP/UDP connections because both file structures are involved as well as MIME types, and in some cases, logs that may not be new line delimited requiring more involvement from the Cyderes side.
In addition to new line delimited, another item that requires additional effort is binary data such as those used by Google protocol buffers due to needing to be read correctly and sent to downstream security platforms in a way the platform can understand.
On-Premise vs Cloud¶
Cyderes engineered CYCLOPS with the future in mind and can support select integrations running on CYCLOPS virtual machines or running in the Cyderes cloud environment. Integrations that do not require dependencies external to the on-prem network are prime for deployment on CYCLOPS, the most common being TCP/TLS/UDP integration (ex: Cyderes forwarder or the Chronicle Forwarder).