Skip to content

AWS S3

Cyderes has the ability to pull logs from AWS S3.

Creating a AWS S3 Bucket

  1. Please follow this guide for creating S3 buckets. Creating AWS S3 Bucket Guide.

Access Configuration

To provide access for Cyderes, follow the instructions below.

Cyderes supports two methods of AWS authentication. The preferred method is the creation of an IAM role though an IAM user is also supported if required.

IAM Role

  1. In the console navigate to IAM
  2. Click Roles
  3. Click Create Role
  4. Select Another AWS Account in the top bar
  5. For Account ID, fill in 237482752974
  6. Click Require external ID. Populate this box with a value. This value will need to be provided to Cyderes to connect to the account.
  7. Click Next
  8. Click Create Policy to create a new policy to attach to this role
  9. Change the editor to JSON
  10. In the JSON editor, copy and paste in these values:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "S3BucketAcccessCYDERES",
          "Effect": "Allow",
          "Action": [
              "s3:GetObject",
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::BUCKETNAME",
              "arn:aws:s3:::BUCKETNAME/*"
          ]
        },
        {
          "Sid": "KMSAcccessCYDERES",
          "Effect": "Allow",
          "Action": "kms:Decrypt",
          "Resource": [
              "arn:aws:kms:*:*:key/*"
          ]
        }
      ]
    }
    
  11. Edit the JSON IAM policy making sure to replace the BUCKETNAME with the name of the S3 bucket that Cyderes will be accessing

  12. KMSAccess is only needed if KMS encryption is being used
  13. Name the policy and click save
  14. After creating the policy, return to the create role window. Refresh the policies. Find and select the newly created policy and click Next.
  15. Fill out any preferred tags on the role and then click Next
  16. Fill in the AWS IAM role name and then click on Create Role
  17. Once the role is created click into it and find the role ARN
  18. Send the following to Cyderes when completed:

    • IAM Role ARN
    • External ID
    • S3 Bucket Name
    • S3 Bucket Region
    • S3 Path Prefix to Data
    • Folder structure of data itself (example: folder path includes the date the data is created)
  19. Cyderes recommends that object creation events to SQS is set up. This especially helps with high volume log sources. Cyderes needs to set up the SQS queue first before the S3 Bucket is configured to for SQS. Once Cyderes has set up the SQS queue, follow the directions below.

IAM User

  1. Create a user for Cyderes to use to access the S3 Bucket. This guide can be followed User Guide.
  2. When creating the user, the access type needed is Programmatic Access
  3. When setting permissions, create a new policy. The policy generator can be used to grant the user access to the S3 bucket. Alternatively, this json policy can be used:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "S3BucketAcccessCYDERES",
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::Account-ID:user/CYDERES"
          },
          "Action": [
            "s3:ListBucket",
            "s3:GetObject"
          ],
          "Resource": [
            "arn:aws:s3:::BUCKETNAME"
          ]
        },
        {
          "Sid": "KMSAcccessCYDERES",
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::Account-ID:user/CYDERES"
          },
          "Action": "kms:Decrypt",
          "Resource": [
              "arn:aws:kms:*:*:key/*"
          ]
        }
      ]
    }
    
  4. Send the following to Cyderes when completed:

    • IAM User Access Key ID
    • IAM User Secret Access Key
    • S3 Bucket Name
    • S3 Bucket Region
    • S3 Path Prefix to Data
    • Folder structure of data itself (example: folder path includes the date the data is created)
  5. Cyderes recommends that object creation events to SQS is set up. This especially helps with high volume log sources. Cyderes needs to set up the SQS queue first before the S3 Bucket is configured to for SQS. Once Cyderes has set up the SQS queue, follow the directions below.

KMS Keys

If a KMS key is used to encrypt objects being written into the S3 bucket and the KMS exists outside of the account where the S3 bucket lives, there is an extra configuration step. Reference: AWS Documentation

On the KMS key, add the following policy statements replacing the AWS account ids with the account id where the S3 bucket lives. This will allow the AWS account where the S3 bucket exists to access the KMS key so that the objects can be downloaded. Without this policy in place, it will not be possible for Cyderes to access encrypted S3 objects.

{
    "Sid": "Allow use of the key",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::ACCOUNT-ID:root"
        ]
    },
    "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
    ],
    "Resource": "*"
},
{
    "Sid": "Allow attachment of persistent resources",
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "arn:aws:iam::ACCOUNT-ID:root"
        ]
    },
    "Action": [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
    ],
    "Resource": "*",
    "Condition": {
        "Bool": {
            "kms:GrantIsForAWSResource": "true"
        }
    }
}

SQS Queue/SNS Topic Configuration

Cyderes supports sending notifications directly to a Cyderes owned SQS queue or receiving notifications from a customer owned SNS topic. This will need to be configured for each s3 bucket that Cyderes is expected to pull logs from.

  1. Follow the AWS Enabling Event Notifications Guide to complete the configuration.
  2. In the Event Types section , select all s3:Object.Created events.
  3. In the Destination Section:

    • SQS Queue:
      • To send to a Cyderes SQS queue, select SQS Queue as the destination type, then enter the SQS ARN provided by Cyderes.
    • SNS Topic:
      • If using an SNS topic, select SNS topic as the destination type, then enter the ARN for the SNS topic that Cyderes will subscribe to.
      • Cyderes needs permissions to subscribe to the SNS topic before notifications can be received. To do this, the following policy should be applied to the SNS topic. Be sure to replace the example ARN in the Resource field with the SNS Topic ARN.
        {
          "Statement": [{
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
              "AWS": "237482752974"
            },
            "Action": ["sns:Subscribe"],
            "Resource": "arn:aws:sns:us-east-1:444455556666:TopicArn"
          }]
        }