Amazon S3 Connector

Connect your S3 buckets to AI pipelines for seamless data integration.

Setup Instructions

1. Navigate to Data Integrations

Open your flow and go to the Data Integrations tab.

2. Select S3 Integration

Click Add Connector, then search for Amazon S3 and select it.

3. Configure Basic Settings

Fill in the following fields:

• Connector Name: Give your S3 connector a descriptive name
• Bucket Name: Enter the name of your S3 bucket (e.g., my-bucket-name)
• Region: Select the AWS region where your bucket is located (e.g., eu-central-1)
• Folder (Optional): Specify a destination folder in the file manager
  • If left empty, data will be stored in the root directory

4. Create AWS User and Access Keys

To connect securely, you need to create an IAM user with appropriate permissions:

1. Go to the AWS Console and navigate to IAM (Identity and Access Management)
2. Click on Users in the left sidebar
3. Click Create User
4. Enter a username (e.g., aicuflow-s3-connector)
5. Click Next to proceed to permissions

5. Set Up Permissions

Instead of adding the user to a group, we'll create an inline policy:

1. Skip the group assignment and click Next
2. After creating the user, click on the username to open the user details
3. Go to the Permissions tab
4. Click Add Permissions > Create Inline Policy
5. Switch to the JSON tab
6. Paste the following policy (replace YOUR-BUCKET-NAME with your actual bucket name):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
    }
  ]
}

7. Click Review Policy
8. Give the policy a name (e.g., S3ReadAccess)
9. Click Create Policy

6. Generate Access Keys

1. In the user details page, go to the Security Credentials tab
2. Scroll down to Access Keys
3. Click Create Access Key
4. Select Application running outside AWS as the use case
5. Click Next
6. Add a description tag (optional)
7. Click Create Access Key
8. Important: Copy both the Access Key ID and Secret Access Key
   • The secret key will only be shown once
   • Store it securely

7. Complete the Connection

1. Return to the connector setup in your flow
2. Paste the Access Key ID in the corresponding field
3. Paste the Secret Access Key in the corresponding field
4. Configure sync settings:
   • File Pattern (Optional): Specify patterns to filter files (e.g., *.csv, data/*.json)
   • Recursive: Enable to include files in subdirectories
5. Click Create Connection

8. Monitor Sync Status

1. Navigate to Data Synchronization to see the import progress
2. Once complete, go to File Manager
3. Your S3 data will appear in the specified folder
4. You can now use this data in your AI pipelines and flows

Best Practices:

• Use specific file patterns to avoid importing unnecessary files
• Keep your AWS credentials secure and never share them
• Regularly rotate access keys for enhanced security
• Monitor your S3 usage to avoid unexpected costs