Import Data from Amazon S3

The data import feature lets you efficiently import large volumes of data directly from your AWS S3 bucket. This integration enhances your marketing efforts by enabling personalized and targeted campaigns with up-to-date data. It’s beneficial for businesses storing large datasets in S3 and looking to leverage them within Insider for segmentation, personalization, and automation.

In this guide, you will learn the steps required to set up the integration and import data from Amazon S3 to Insider One, ensuring a seamless flow of information to enhance your marketing initiatives.

S3 Import functionality supports .jsonl, .csv files, and attributes, event, purchase, and user delete data.

Essentials before Integration

To make the most of the Amazon S3 integration, you'll need these essential elements:

• Create a new SQS queue

• S3 Bucket Designation: Identify the S3 bucket where your data is stored. This is crucial for directing Insider One to the correct data source for import.

• Authorization for Insider One: Grant Insider One permission to read data from your S3 bucket. This ensures that Insider Onecan access and process your data securely.

  • SQS queue for new file notifications: Set up an SQS queue to receive notifications of new files. This allows for automated data import as new data becomes available.

  • IAM user: Create a dedicated IAM user with the necessary permissions. This user will facilitate secure access and management of your AWS resources.

These elements are essential to ensure a seamless and secure integration process, enabling efficient data import and utilization within Insider One.

Authorization for Insider One to Access your S3 Bucket

You can refer to the steps below to grant Insider One permission to access your S3 bucket for importing user data. Insider One supports the IAM User method, a dedicated IAM user with the necessary permissions for authorization.

For more details on policy management, you can refer to the official AWS documentation:

IAM Access Policies

Managing IAM Group Policies

Definitions

IAM (Identity and Access Management): AWS service that securely controls access to AWS resources.

IAM User: An AWS identity with specific permissions that can interact with AWS resources.

SQS (Simple Queue Service): A fully managed message queuing service that enables decoupling and scaling of microservices, distributed systems, and serverless applications.

ARN (Amazon Resource Name): A unique identifier for AWS resources.

Trust Policy: A JSON policy that defines which AWS accounts or services are allowed to assume an IAM Role.

Policy: It is a JSON that defines which resources any AWS entity can access.

Before following the steps, ensure you're logged into your AWS account and authorized to create a new SQS and S3 bucket. You can log in here.

Step 1: Create S3 Source Integration

1. Navigate to Components > Integrations > External Integrations on the Insider One’s InOne panel.  

2. Select the Amazon S3 platform and create an S3 Source Integration.

   

3. Name your integration, select Source as the integration type, and click the Save button.

   

Step 2: Grant Access via IAM User

Step 2.1: Create a Source S3 Bucket

If you haven't already created a source bucket, follow these steps:

1. Choose the appropriate AWS Region.

2. Use the search bar to navigate to S3 or click here for the service.

3. Click Create bucket.

4. Enter a Bucket name (e.g., insider-user-data-bucket).

5. In Block Public Access settings for this bucket, ensure all options remain enabled to prevent unauthorized access.

6. Click Create bucket.

7. Please see this part for data requirements and file types.

Step 2.2: Create an SQS

1. Use the search bar to navigate to SQS or click here to go to the service.

2. Click the Create queue button.

3. Enter a queue name in the Name field.

4. In the Access policy section, click the Advanced button.

5. Change the required fields and paste the JSON below into the JSON editor.

   {
     "Version": "2012-10-17",
     "Id": "__default_policy_ID",
     "Statement": [
       {
         "Sid": "__owner_statement",
         "Effect": "Allow",
         "Principal": {
           "Service": "s3.amazonaws.com"
         },
         "Action": "sqs:*",
         "Resource": "arn:aws:sqs:<YOUR_REGION>:<YOUR_ACCOUNT_ID>:<YOUR_SQS_QUEUE_NAME>",
         "Condition": {
           "StringEquals": {
             "aws:SourceAccount": "<YOUR_ACCOUNT_ID>"
           },
           "ArnLike": {
             "aws:SourceArn": "arn:aws:s3:::<YOUR_S3_BUCKET_NAME>"
           }
         }
       }
     ]
   }

6. Leave the rest of the settings by default.

7. Click the Create queue button at the bottom of the page.

8. Click on the queue you have already created with this link. Copy the URL from the page that opens to register it on the Insider One’s InOne panel later.

   

Step 2.3: Assign SQS to S3

After completing steps 1 and 2, perform the following steps:

1. Enter the S3 Bucket you created earlier.

2. Click the Properties button in the tabs.

3. Find the Event notifications section and click the Create event notification button.

4. Fill the Event Name field as you want.

5. In the Event types step, select the All object create events checkbox.

6. In the Destination step, first click the SQS queue button and then select the SQS you have created before from the SQS queue dropdown.

7. Click the Save changes button.

   

Step 2.4: Create an IAM User

1. Sign in to the AWS Management Console and navigate to the IAM service.

2. Go to Users and click Create user.

3. Enter a User Name (e.g., InsiderS3Access) and click the Next button.

4. Click the Next button.

Step 2.5: Attach the Required Permissions

1. Select Attach existing policies directly.

2. Choose JSON in the Policy editor section.

3. Apply the following policy to the JSON editor and click the Next button:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "sqs:ReceiveMessage",
                   "sqs:DeleteMessage",
                   "sqs:GetQueueUrl",
                   "sqs:GetQueueAttributes"
               ],
               "Resource": [
                   "arn:aws:sqs:YOUR_REGION:YOUR_ACCOUNT_ID:YOUR_SQS_QUEUE_NAME"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket",
                   "s3:GetObjectAttributes"
               ],
               "Resource": [
                   "arn:aws:s3:::YOUR_BUCKET_NAME",
                   "arn:aws:s3:::YOUR_BUCKET_NAME/*"
               ]
           }
       ]
   }

4. (Replace YOUR_BUCKET_NAME, YOUR_REGION, YOUR_ACCOUNT_ID, and YOUR_SQS_QUEUE_NAME with your actual values.)

5. Enter a name in the Policy name field.

6. Click the Create policy button.

7. Return to the user creation page and click the Refresh icon to the left of the Create policy button.

8. Find out your created policy via the search bar.

9. Click the policy checkbox and click the Next button.

10. Ensure that the policy adheres to the best practices outlined in the IAM Access Policies guide.

11. After attaching the policy, click the Create user button at the final step.

Step 2.6: Share Access Credentials

1. After creating the user, navigate to Security credentials.

2. Click the Create access key button to generate a new access key.

3. Select the Third-party service use case and click the Next button.

4. Click the Create access key button.

5. Copy the Access Key ID and Secret Access Key.

6. Paste SQS URL, Access Key ID, and Secret Access Key into the Insider One’s InOne panel.

   

Step 3: Add your S3 Bucket Details

1. Add the desired AWS Bucket Name and AWS Folder Path Name on the Insider One’s InOne panel.

   

2. When you save it, if Insider One identifies any possible errors for authentication or folder access, you will see a message at the top of the page with the error message. You can check the Error Responses List for the possible errors by AWS.

3. If you see no errors on the Authentication page, click Save and Continue.

Step 4: Upload Mapping File (Optional)

1. Upload your mapping file.

   

2. Once you upload your mapping file, all match fields will be filled automatically if the data matches; otherwise, you can easily select the headers from your file to be mapped to Insider One attributes and event parameters.

   - File should contain the headers that will be synced to Insider One from your S3 bucket.

   - Insider One will only read the first row of the file.

   To learn how to fill in the required fields, you can refer to the video below:

Step 5: Match your Data with your S3 Bucket Files

1. In the Events Collection field, select the appropriate Event Collection method:

1. Collect from File Column: This method allows you to map a column to an event name in Insider One.

   i. Select the column where your event name is mapped. If uploaded, you can select from the dropdown; otherwise, you can enter it directly in the input field. Select column values will act as event names when ingesting data to Insider One.

   ii. Select the timestamp and timestamp format to ensure the event is synced. Each event should have its timestamp.

   Timestamp in your file can have a date in any of the following formats:

   - Unix Timestamp

   - ISO 8601

   - RFC2822

   - RFC3339

   b. Collect from File Name: This method maps the entire file to a single event name in Insider One.

   i. Enter the file prefix, so Insider One can detect the file in your S3 bucket (via SQS) and map it as a single event in Insider One.

   Example: File Name: purchase_event_1756814468 Prefix: purchase_event

   

   Each file must have a unique name. If file names are not unique, Insider One will reject duplicates.

   File names and prefixes are case-sensitive. Make sure the prefix matches the file name exactly.

   ii. Select the timestamp and timestamp format for the event to be synced. Each event should have its own timestamp.

   Timestamp in your file can have a date in any of the following formats

   1. Unix Timestamp

   2. ISO 8601

   3. RFC2822

   4. RFC3339

   c. Collect No Events: When this option is selected, Insider One will not collect any event data from your S3 Bucket.

2. In the Identifier Mapping field, you can map your identifier in your S3 bucket to one of Insider One's identifiers.

-You have to map at least one identifier.

-Identifiers selected during identity resolution can’t be selected as attributes; ensure you select them here if you want to sync any identifier.

3. In the Attribute and Event Parameters Mapping,  you must map your attribute and event parameters either manually or using the data uploaded as a file. Here, you can easily map any of the columns in your CSV or JSON file to an Insider One attribute or event parameter to be inserted. If you have uploaded a file, all the columns will be listed here; if not, you can manually insert your column names to be mapped to Insider One fields.

   

After following these steps, you can securely grant Insider One access to your S3 bucket via an IAM User, allowing Insider One to pull data directly from your specified S3 bucket and path. If you have any questions, please contact the Insider One team for support and assistance.

File Format

Files can be in  .csv, .jsonl , csv.gz or jsonl.gz formats.

• Each file must contain at least one identifier.

• Each file must include at least one event or attribute; otherwise, the data will not be processed.

• Each file must be comma-delimited.

Default Attribute, Event & Parameter Data Mapping table

For all default events and attributes, refer to Events & Attributes.

Example File Formats

• JSON example file

  S3 Sample

  3.56 KB

• CSV example file

  S3 Sample Data

  1.30 KB

Limitations

• Only  .csv, .jsonl, csv.gz or jsonl.gz file formats are supported; other file formats are not supported.

• 5 GB file size limit: If a file exceeds 5 GB, the entire file will be skipped.

• Only one bucket per integration: Each integration can only be linked to a single S3 bucket.

• Maximum of 5 paths per bucket: You can specify up to five paths per integration.

• All JSON lines must be valid: Every line in a JSON file must be properly formatted; otherwise, the file will be skipped.

• An identifier must be included in the file. Otherwise, the file will not be processed.

• Any custom attribute or event parameter must be created on Insider One’s Attributes and Events page first; otherwise, the data will not be ingested.

• File names must be unique. If a file with the same name has already been processed, it will be skipped (e.g., a repeated purchase_timestamp.csv file will not be ingested again).

• The maximum file count is up to 25 files per day that can be ingested.

• The maximum total file size is the combined size of all files per day and must not exceed 20 GB.

FAQ

Q: What happens if there are existing messages in the queue before launching the integration?
A: If the integration is paused, messages in the SQS queue will still be collected. Once resumed, files will be processed, but delays might cause:

• Campaign delays – Real-time updates will be delayed, reducing engagement.

• Missed opportunities – New leads and user actions won’t be processed.

• Outdated data – Segmentation and personalization might become less effective.

• Inaccurate metrics – Reports might not reflect real-time user behavior.

To minimize impact, reactivate the integration promptly to ensure all accumulated data is processed.

Q: Can I use the same S3 bucket for multiple integrations?
A: No, each integration can only be linked to a single S3 bucket.

Q: Does the integration support delta updates or only full data loads?
A: The integration supports full data loads but does not automatically handle incremental or delta updates. Each update must be provided in a new file and pushed to the SQS queue.

Q: Can I use a file format other than JSON or CSV?
A: No, only JSON and CSV formats are supported. Any other format will be ignored.

Q: Does the integration support GZIP?
A: Yes, it supports JSON and CSV gzip formats. Make sure that your file type is csv.gz and jsonl.gz.

Q: How does Insider One handle file parsing errors?
A: If the file contains errors (e.g., invalid JSON or improperly structured CSV), the entire file will be skipped.

Q: How long do I need to keep my files in the S3 bucket?
A: Although Insider One will process your files in near real-time, we highly recommend keeping your files in the bucket for at least 12 hours to ensure Insider One can completely read and sync the data.

Q: What happens if Insider One cannot authenticate with my S3 bucket?
A: If authentication fails, an error message will appear on the authentication page detailing the issue (e.g., incorrect IAM permissions, invalid credentials, or inaccessible paths). You can refer to the AWS article for possible errors.

Q: Do I need an UPDATED_AT column like in data warehouse integrations?
A: No, this integration does not require or support the UPDATED_AT column. New files are processed based on S3 event notifications rather than an update timestamp.

Q: Is it mandatory to include an identifier in the file?
A: Yes, an identifier must be included. However, if the file contains only an identifier without an event or attribute, the data will not be processed.

Q: What happens if a custom attribute or event parameter is not created in Insider One?
A: If a custom attribute or event parameter does not exist in Insider One, the data will not be ingested. You must create the attribute or event parameter on Insider One’s Attributes and Events page before sending data.

Q: Does the integration support user deletion events?
A: No, user deletion events are not currently supported.

Q: How does Insider One handle data ingestion errors?
A: When importing data from Amazon S3, the system applies specific error-handling rules to ensure valid data is still processed whenever possible:

• Attribute errors: If an attribute is sent in the wrong data type and triggers an error, only that attribute is rejected. The rest of the data in the same row is successfully ingested.

• Event parameter errors: If an event parameter has the wrong data type and causes an error, the entire event is rejected. None of the event’s data will be imported.

• Date-time values: When attributes or event parameters include date-time inputs, only RFC 3339-formatted values are accepted (for example, 2025-11-11T14:30:00Z).

Any other date-time format will be rejected.

Q: Does Insider One support multiple buckets to be connected?
A: Yes, you can connect up to 5 different buckets to sync your data to Insider One. The limit can be adjusted according to your use case.