Cloud Storage¶
Cyderes has the ability to pull logs from GCP Cloud Storage to ingest into Chronicle
Log Ingestion
When using our GCP Cloud Storage integration, please note that only logs generated after the integration setup will be automatically ingested. For older logs existing prior to the setup, you will need to manually trigger notifications for ingestion. This means that logs generated post-integration will be seamlessly ingested, while any pre-existing older logs require manual notification activation for ingestion.
Cyderes Object Storage Best Practices¶
Creating a GCP Cloud Storage Bucket¶
- Create a new Cloud Storage bucket for the Cloud Storage bucket logs to be stored in. A pre-existing Cloud Storage bucket may be used. GCP Guide.
Access Configuration¶
IAM Service Account for bucket access
Depending on if you provide an IAM Service Account for bucket access or not will determine which steps to follow, please carefully note which steps are being followed depending which option is chosen.
with customer provided service account for bucket access¶
-
Provide the following information to Cyderes for implementation:
- GCP Cloud Storage Bucket Name
- GCP Cloud Storage Bucket File Path
- One Log Type for each GCP Cloud Storage Bucket Path
- Cloud Storage Project Service Account
- IAM Service Account for bucket access
Note: The cloud storage project service account that is required should have the following format:
service-1234567890@gs-project-accounts.iam.gserviceaccount.com
-
Cyderes will use the information to provision a PubSub topic and a service account. Cyderes will then provide the service account email and the PubSub topic name to be used in the next steps.
-
Set up events to go to the Cyderes-owned PubSub topic:
- Use this guide to understand the command to run to enable notifications: Command Guide
- The
OBJECT_FINALIZE
event type is needed for Cyderes to process logs from the storage bucket. If this is only managed by automation, theOBJECT_METADATA_UPDATE
may be added to enable ease of reprocessing should the need arise. Docs to reference. - Example command from above: run
gsutil notification create -s -t projects/cyderes-prod/topics/(TOPIC PROVIDED BY Cyderes) -f json -e OBJECT_FINALIZE gs://(YOUR BUCKET NAME)
without customer provided service account for bucket access¶
-
Provide the following information to Cyderes for implementation:
- GCP Cloud Storage Bucket Name
- GCP Cloud Storage Bucket File Path
- One Log Type for each GCP Cloud Storage Bucket Path
- Cloud Storage Project Service Account
Note: The cloud storage project service account Cyderes requires should have the following format:
service-1234567890@gs-project-accounts.iam.gserviceaccount.com
-
Cyderes will use the information to provision a PubSub topic and a service account. Cyderes will then provide the service account email and the PubSub topic name to be used in the next steps.
- Grant permissions to retrieve objects from the Cloud Storage bucket to the Cyderes provided service account by following this GCP Guide.
The required permissions are
Storage Legacy Bucket Reader (roles/storage.legacyBucketReader)
andStorage Object Viewer (roles/storage.objectViewer)
. -
Set up events to go to the Cyderes-owned PubSub topic:
- Use this guide to understand the command to run to enable notifications: Command Guide
- The
OBJECT_FINALIZE
event type is needed for Cyderes to process logs from the storage bucket. If this is only managed by automation, theOBJECT_METADATA_UPDATE
may be added to enable ease of reprocessing should the need arise. Docs to reference. - Example command from above: run
gsutil notification create -s -t projects/cyderes-prod/topics/(TOPIC PROVIDED BY Cyderes) -f json -e OBJECT_FINALIZE gs://(YOUR BUCKET NAME)