AWS S3¶

Cyderes supports ingesting logs from an AWS S3 bucket.

Setup¶

To ingest AWS S3 objects, Cyderes subscribes to a customer owned SNS topic and either uses an IAM Role or an IAM User to retrieve objects from the bucket. Please follow the instructions below to set up the bucket, SNS topic, and IAM Role or IAM User.

Log Ingestion

When using our AWS S3 cloud storage integration, please note that only logs generated after the integration setup will be automatically ingested. For older logs existing prior to the setup, you will need to manually trigger notifications for ingestion. This means that logs generated post-integration will be seamlessly ingested, while any pre-existing older logs require manual notification activation for ingestion.

Please refer to our object storage best practices documentation before following the steps below.

Bucket and SNS¶

Please follow this guide for creating S3 buckets. Creating AWS S3 Bucket Guide.
Once the bucket has been created, create an SNS topic for bucket event notifications. Please follow this guide for creating SNS Topics. Creating AWS SNS Topic Guide.

SNS Encryption

If you choose to enable encryption at rest in your SNS topic, please see this troubleshooting article to ensure the proper KMS key is set up and that the AWS S3 principle has permissions to use that KMS key
Set up notifications from the bucket to go to the newly created SNS topic. Enabling Event Notifications Guide
Allow Cyderes to subscribe to the SNS topic and publish messages using the following IAM policy as a template.

  {
    "Statement": [
      {
        "Sid": "Example SNS subscribe policy",
        "Effect": "Allow",
        "Principal": {
          "AWS": "237482752974"
        },
        "Action": [
          "sns:Subscribe"
        ],
        "Resource": "arn:aws:sns:us-east-1:444455556666:YourTopicArn"
      },
      {
        "Sid": "Example SNS publish policy",
        "Effect": "Allow",
        "Principal": {
          "AWS": "237482752974",
          "Service": "s3.amazonaws.com"
        },
        "Action": [
          "sns:Publish"
        ],
        "Resource": "SNS-topic-ARN",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:s3:*:*:`bucket-name`"
          },
          "StringEquals": {
            "aws:SourceAccount": "`bucket-owner-account-id`"
          }
        }
      }
    ]
  }

More information on configuring the bucket for the SNS Topic can be found in AWS' documentation.

SNS Subscription

Please do NOT create an SNS subscription to the SNS topic on Cyderes' behalf. Cyderes will create and set up the SNS subscription to receive notifications in a Cyderes owned AWS account.

IAM Role or User Setup¶

Cyderes supports two methods of AWS authentication, an AWS IAM Role, or an AWS IAM User. The preferred access method for Cyderes is the AWS IAM Role, but an IAM user is supported if necessary.

To provide Cyderes access to read from the S3 bucket, follow the instructions below and set up only one of the following.

IAM Role¶

In the console navigate to IAM
Click Roles
Click Create Role
Select Another AWS Account in the top bar
For Account ID, fill in 237482752974
Click Require external ID. Populate this box with a value. This value will need to be provided to Cyderes to connect to the account.
Click Next
Click Create Policy to create a new policy to attach to this role
Change the editor to JSON

In the JSON editor, copy and paste in these values:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketAcccessCYDERES",
      "Effect": "Allow",
      "Action": [
          "s3:GetObject",
          "s3:ListBucket"
      ],
      "Resource": [
          "arn:aws:s3:::BUCKETNAME",
          "arn:aws:s3:::BUCKETNAME/*"
      ]
    },
    {
      "Sid": "KMSAcccessCYDERES",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": [
          "arn:aws:kms:*:*:key/*"
      ]
    }
  ]
}

Edit the JSON IAM policy making sure to replace the BUCKETNAME with the name of the S3 bucket that Cyderes will be accessing
KMSAccess is only needed if KMS encryption is being used
Name the policy and click save
After creating the policy, return to the create role window. Refresh the policies. Find and select the newly created policy and click Next.
Fill out any preferred tags on the role and then click Next
Fill in the AWS IAM role name and then click on Create Role
Once the role is created click into it and find the role ARN
Send the following to Cyderes when completed:
- IAM Role ARN
- External ID
- S3 Bucket ARN
- S3 Bucket Region
- S3 Path Prefix to Data
- Folder structure of data itself (example: folder path includes the date the data is created)
- SNS Topic ARN for Bucket Event Notifications

IAM User¶

Create a user for Cyderes to use to access the S3 Bucket. This guide can be followed User Guide.
When creating the user, the access type needed is Programmatic Access

When setting permissions, create a new policy. The policy generator can be used to grant the user access to the S3 bucket. Alternatively, this json policy can be used:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BucketAcccessCYDERES",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKETNAME",
        "arn:aws:s3:::BUCKETNAME/*"
      ]
    },
    {
      "Sid": "KMSAcccessCYDERES",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": [
          "arn:aws:kms:*:*:key/*"
      ]
    }
  ]
}

After creating the user, return to the users window. Refresh the users. Find and select the newly created user.
Click on the Security Credentials tab and create a new access key by clicking on Create access key.
Choose Third-party service and then click Next.
Make sure to note down the Access key and Secret Access Key, as these will need to be sent to Cyderes.
Send the following to Cyderes when completed:
- IAM User Access Key ID
- IAM User Secret Access Key
- S3 Bucket ARN
- S3 Bucket Region
- S3 Path Prefix to Data
- Folder structure of data itself (example: folder path includes the date the data is created)
- SNS Topic ARN for Bucket Event Notifications

Additional Considerations¶

Sometimes, the above steps aren't all that's required for Cyderes to retrieve objects from an AWS S3 bucket. Below are some additional considerations you will want to understand in case your AWS environment meets any of these criteria.

KMS key resides in a different AWS account¶

If a KMS key is used to encrypt objects in the S3 bucket and the KMS key exists in a different AWS account than the S3 bucket, there is an extra configuration step required. Reference: AWS Documentation

On the KMS key, add the following policy statements replacing the AWS account ids with the account id where the S3 bucket lives. This will allow the AWS account where the S3 bucket exists to access the KMS key in the other AWS account so the objects can be downloaded. Without this policy in place, it will not be possible for Cyderes to access encrypted S3 objects.

{
    "Sid": "Allow use of the key",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::ACCOUNT-ID:root"
        ]
    },
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
    ],
    "Resource": "*"
},
{
    "Sid": "Allow attachment of persistent resources",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::ACCOUNT-ID:root"
        ]
    },
    "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
    ],
    "Resource": "*",
    "Condition": {
        "Bool": {
            "kms:GrantIsForAWSResource": "true"
        }
    }
}