CYCLOPS¶
The Cyderes CNAP Logging & Operations Server (CYCLOPS) is a virtual appliance built to manage various containerized applications on a Cyderes-managed Kubernetes cluster that enables data forwarding to security analytics platforms like Cyderes CNAP, GCP's Chronicle, and Azure Sentinel. Customers are provided a VM appliance from Cyderes to deploy into their environment. Once online, the node enrolls into the Cyderes-managed configuration management system, loads necessary dependencies, and applications are deployed onto the CYCLOPS Kubernetes cluster.
CYCLOPS can perform as a single node cluster or can be linked with additional nodes to form a High Availability (HA) cluster. This cluster runs in the customer's environment, typically on a virtualization platform (VMware, Hyper-V, KVM, etc) or in a cloud computing environment such as AWS, GCP, or Azure. Kubernetes runs on top of this CYCLOPS. Kubernetes is a container orchestration platform that allows for simplified container deployments, zero downtime configuration updates, load balancing, high availability, and autoscaling.
From CYCLOPS, Cyderes will deploy containerized applications including our data forwarder technologies, logging/metrics collection for CYCLOPS, CYCLOPS management agents, and some Kubernetes components.
Scope and Sizing¶
The CYCLOPS size is derived from a combination of 'Events per Second' and data types that will be configured. Data types are defined as differentiated sources of information. For example, EDR, DNS, and DHCP are all separate data types.
Cyderes requires CYCLOPS be deployed with at least 4 CPU, 16 GB of RAM, and 100 GB of disk space. This sizing allows for CYCLOPS to be instantly capable to accept new data types or features to be added. For additional resources past the minimum requirements, refer to the table below to calculate the additional resources that may better fit your scenario.
CYCLOPS Hardware Requirements¶
| Tier | Log Entries per Hour | Ops/s | CPU | RAM | Disk Space |
|---|---|---|---|---|---|
| Minimum Requirement | < 1 million | < 100 | 4 | 16 GB | 100 GB |
| Advanced Tier | > 20 million | > 1k | 8 | 32 GB | 100 GB |
Note: The actual hardware requirement per node will depend on the actual ingestion quantity. The advanced tier is for reference only.
Cyderes recommends following VMware documentation when choosing network interfaces: https://kb.vmware.com/s/article/1001805
Deployment¶
Cyderes can provide an OVA or similarly packaged virtual appliance or an AWS/GCP/Azure image. The package contains a base Linux operating system with enough necessary dependencies to bootstrap the system and establish initial contact with the Cyderes. The package can be deployed as often as needed to build additional nodes.
- If utilizing an AWS AMI image, please provide Cyderes with AWS account numbers and regions to share the AMI to.
- If utilizing a GCP Compute Image, please provide Cyderes the Admin account address or service account email address that will be given the Compute Image User role in the Cyderes project and can be used to create a copy of the Cyderes CYCLOPS image within your project.
- If utilizing an Azure Shared Image, please provide Cyderes the appropriate Tenant ID.
IMPORTANT: Please provide Cyderes with the hostname(s) set for the forwarder(s). Providing a unique name that specifically identifies each forwarder will help our team get them provisioned as quickly as possible and help troubleshoot any issues in the future. Please ensure there are no spaces, underscores, or special characters other than a dash in the unique hostname(s).
Why deploy a forwarder?¶
Cyderes can handle log ingestion directly in our cloud-hosted forwarders, requiring no on-premise deployments of log collectors whatsoever. However, some customers appreciate having a centralized CYCLOPS log forwarder within a given on-premise environment. Additionally, some customers have legacy systems which cannot communicate over TLS, and having an on-premise forwarder allows the customer to limit the unencrypted traffic to the local network since the CYCLOPS log forwarder handles the encrypted traffic upstream back to Cyderes.
Connectivity Requirements¶
Notes:
- A static/reserved IP address is required for proper appliance functionality.
- If there is TLS/SSL interception enabled, please also bypass for the domains listed below.
- If required, a custom DNS server may be used in place of 8.8.8.8 and 8.8.4.4.
- *.cyderes.io port 123 must be used for NTP.
- Broad opening of sub-domains to .cyderes.cloud and .cyderes.io is encouraged due to changing infrastructure.
If a health check port is required for a load balancer to function properly, please set the health check probe port to 10250, which is a metrics port for the Kubernetes cluster.
| Destination | Port | Direction | Use |
|---|---|---|---|
| gcr.io | TCP/443 | External Outbound | Docker Image Management |
| *.googleapis.com | TCP/443 | External Outbound | Telemetry (High Bandwidth) |
| googleapis.l.google.com | TCP/443 | External Outbound | Authentication |
| accounts.google.com | TCP/443 | External Outbound | Authentication |
| *.cyderes.cloud | TCP/443 | External Outbound | CYCLOPS Management |
| *.cyderes.io | TCP/123,443 UDP/123 | External Outbound | NTP CYCLOPS Metrics |
| sm-ext-01.cyderes.cloud | TCP/4505,4506 | External Outbound | CYCLOPS Management |
| 8.8.8.8 | TCP/53, UDP/53 | External Outbound | DNS |
| 8.8.4.4 | TCP/53, UDP/53 | External Outbound | DNS |
| Data sources | TCP/30000-32767, UDP/30000-32767 | Local Inbound | |
| Product Specific Connections | |||
| cyderes.siemplifycloud.com | TCP/443 | External Outbound | Siemplify |
| malachiteingestion-pa.googleapis.com | TCP/443 | External Outbound | Chronicle Ingestion |
Note: If the CYCLOPS is setup in HA (High Availability) cluster where multiple nodes are deployed, following is required to be open between those nodes for cluster communications
| Port | Type | Description |
|---|---|---|
| 443 | TCP | https |
| 6443 | TCP | kube proxy |
| 9099 | TCP | Canal/Flannel livenessProbe/readinessProbe |
| 9100 | TCP | Prometheus metrics |
| 10254 | TCP | Ingress controller livenessProbe/readinessProbe |
| 10250 | TCP | kubelet API |
| 8472 | UDP | Canal/Flannel VXLAN overlay networking |
| 4789 | UDP | Flannel VXLAN overlay networking on Windows cluster |
Architecture Diagram¶
Generally, for each data type that is sent to CYCLOPS, a new TCP and UDP listener service will be configured to accept the data. The listener service ports will begin at 30000 and be incremented to accommodate additional data types as needed. For example, if there are two data types being sent over syslog, there would be two listener services on 30000 TCP/UDP and 30001 TCP/UDP.
Cyderes recommends utilizing TCP for sending syslog traffic whenever possible as this can help guarantee reliable log delivery.
CYCLOPS will function best when deployed in a load balanced environment to ensure maximum availability. The load balancer should distribute traffic on any TCP or UDP port to allow for new data types listeners on CYCLOPS to be added at any time.
High Availability¶
CYCLOPS supports two layers of high availability and load balancing.
-
Node/VM Layer - Multiple CYCLOPS log forwarder instances can be run as a cluster within a customer-managed load balancer targeting each CYCLOPS forwarder instance. This also allows the customer to scale CYCLOPS out horizontally as well as vertically to meet their needs.
-
Container Layer - By default, each CYCLOPS log forwarder leverages Kubernetes to load-balance multiple instances of the log forwarder service within multiple containers. This enables Cyderes to perform ZDD (zero downtime deployments) updates and meet our High Availability objectives, even on single node CYCLOPS cluster deployments.
Buffering, Resiliency, and Continuity of Log Flow¶
First, Cyderes recommends, whenever possible, customers to configure their systems to queue logs locally as a first layer of resiliency should that system become unable to contact the CYCLOPS log forwarder (e.g. a firewall misconfiguration or routing failure within the customer's environment).
Second, CYCLOPS forwarders leverage in-memory queuing of all incoming log data for times when they cannot communicate upstream to Cyderes, which is sufficient in most cases due to the reliability and high availability of Google Chronicle and the Google Cloud Platform (GCP). See Scope and Sizing above to ensure the deployment allocates sufficient RAM.
Security Considerations¶
Cyderes employs industry standard distributed configuration management and diagnostic solutions which instruct our CYCLOPS forwarders deployed within customer environments to establish outbound-only connections to Cyderes (e.g. Salt/Ansible). In the rare situation where an engineer must manually log into a forwarder, that engineer must successfully authenticate via Cyderes' managed MFA SSO before gaining access to the forwarder’s terminal shell within the customer environment via the outbound connection to Cyderes, at which point the engineer may perform connectivity diagnostics and Operating System level commands or actions to ensure the healthy status of the log ingestion and forwarding. The CYCLOPS forwarders do not allow inbound connectivity (e.g. SSH) for management or interactive control.
For environments with high security or regulatory concerns, Cyderes recommends deploying the forwarder in a DMZ (i.e. a firewall in between the CYCLOPS forwarder and the internal systems in question) as depicted in the architecture diagram above, allowing the internal systems to connect through the firewall to the CYCLOPS forwarders on the specific ports associated with the relevant data types, but no connectivity from the forwarder to the internal network; only allow internet-bound egress traffic to the resources listed above. This should remove the forwarder from the regulatory or high security compliance scope while maintaining a high level of availability for log collection.
For protection of data in transit, CYCLOPS can accept connections with a minimum of TLS 1.2 (TLS 1.0, 1.1, and all SSL versions are not enabled by default). For customers with legacy systems that cannot send logs over encrypted channels, CYCLOPS forwarders can also be configured to listen on specific ports for plaintext syslog (see the Why deploy a forwarder? section above). All outgoing communication from the CYCLOPS forwarders to Cyderes are encrypted over HTTPS using TLS 1.3.
NOTE: Installing third-party tools or add-ons (e.g. VMware Tools) directly on the host machine is not supported by CYCLOPS.
VMware Configuration Reference:
https://kb.vmware.com/s/article/1004099
CYCLOPS V2 (Pre-Release)¶
Prevents OS Drift and OS EOL¶
CYCLOPSv2 uses a unique approach to OS configuration and maintenance. This allows us to build a compact container image that includes all necessary updates typically received during patching and dedicated maintenance cycles. This image is then applied to clusters using an automated upgrade controller. The result is a global image that targets the latest patches for Ubuntu and the latest Kubernetes components without the need for manual package management services. This process is significantly faster and less error-prone.
Revamped Monthly Maintenance Process¶
The automated upgrade controller works directly with the new OS configuration. This process allows us to distribute a plan to apply the latest global image to all CYCLOPSv2 clusters seamlessly. The new OS configuration includes fail-safes that ensure a working configuration is always available on CYCLOPSv2 instances. If a patch fails, the system will revert to the previous working configuration. This plan will automatically cycle clusters to remain available in high availability setups, though single node setups will briefly be unavailable during cycling. The result is a significantly faster process, coordinated more easily, and an OS that can simply be restarted if critical issues arise. Lastly, if customers need to move CYCLOPSv2 instances, an automated task can trigger it to revert to an unregistered cluster immediately.
Lean OS Focused on Security¶
The new OS configuration is focused on security, and our design decisions have led to several advantages. We reduce the OS packages to the bare minimum required for Kubernetes Operations and general OS behavior. Fewer packages mean fewer attack vectors and less concern from customers. Once CYCLOPSv2 instances are registered, security measures are implemented to restrict external connectivity to management services and prevent unauthorized access. This reduction in OS packages and maintenance tools significantly reduces the CYCLOPSv2 resource footprint, resulting in faster boot times and more resources available to the forwarders.
Automation for Every Step of the Way¶
CYCLOPSv1 required multiple manual steps during the onboarding process. CYCLOPSv2 introduces enhanced automation to streamline the registration process, reducing the need for manual intervention. We plan to extend this automation so new global images are built and tested in all supported environments, ensuring a smooth and efficient customer experience.
Public Images with Minimal Cyderes Identifiers¶
CYCLOPSv2 images have minimal references to Cyderes resources. Our registration process uses unique identifiers for verification. Once the configuration is received, the CYCLOPSv2 instance applies it and restricts external connectivity to ensure security. This new design significantly reduces any potential for malicious reverse-engineering, and our web application can easily be enhanced or updated alongside global images to prevent any concerning behavior.
Designed for the Future¶
The new OS configuration ensures the latest versions of all necessary components, providing peace of mind. Previous versions required dedicated maintenance outside of scheduled periods. CYCLOPSv2 integrates these actions within the dedicated monthly maintenance cycle, ensuring a seamless experience. Additionally, the automation is designed to scale effortlessly to as many instances as needed. The goal was to create an image that works in every environment we support, automate all testing, and target any location we need. Many of these systems are already in place, reducing the burden for self-service initiatives. We are committed to continuously improving and expanding our platform to meet evolving customer needs.