This guide summarizes the limits, sync frequency options, and performance considerations for the data ingestion methods supported by Insider One.
Use this page to compare ingestion methods and plan your data volume and sync schedule. For setup requirements, data mapping rules, and error behavior specific to a connector, refer to the related integration guide.
This guide lists the standard limits available for each ingestion method. If a throughput, concurrency, or processing-time figure is not specified, contact Insider One for detailed sizing based on your data volume, source configuration, and sync requirements.
How to interpret sync frequency
Sync frequency determines how often Insider One can start retrieving new or changed data from a source.
It does not represent a guaranteed processing or completion time. The time required to complete a sync can vary depending on:
The number of records
The size and complexity of each record
The volume of new or changed data
Source-system capacity
Concurrent workloads
Data validation results
Automatic journey-trigger throttling
When ingested data exceeds the thresholds in the following table, Insider One still ingests the data in full. However, it suppresses the Architect journey starters and the data streams that run per row for that data. No user is dropped and no row is skipped. The sync completes as a success and produces no error or notification.
The following table lists the thresholds for each ingestion method.
Ingestion method | Unit counted | Threshold | Applied over |
|---|---|---|---|
Amazon S3 | Parsed rows, per account | 400,000 | Rolling 15-minute window |
Snowflake, BigQuery, and Databricks | Rows in one incremental sync | 400,000 | Each incremental sync |
Amazon S3: Insider One counts rows per account, not per integration, and counts rows rather than distinct users. Because the window rolls, several small files uploaded within the same 15 minutes can reach the threshold together.
Data warehouse integrations: The threshold applies to incremental syncs only. The first full sync never triggers journeys, regardless of its size. This threshold is separate from the 1-billion-row limit below. Reaching the 1-billion-row limit does not fail the sync or archive the integration.
If you need journeys to trigger above these thresholds, contact the Insider One team.
Snowflake
Snowflake imports run on a schedule. The first sync performs a full-table import, while subsequent syncs import new or changed data.
Limitations
Maximum rows per sync: 1 billion
Maximum row size: 350 KB
Available sync frequency: Every 15 minutes up to once daily
Journeys and data streams that run per row are suppressed when the data volume exceeds the journey-trigger threshold. For details, refer to Automatic journey-trigger throttling.
Maximum distinct event names per sync: 100. A sync above this limit is not processed, and the integration moves to Action Required. For details, see TOO_MANY_UNIQUE_EVENTS.
Before launching, check that the initial full sync and each incremental sync will not exceed 1 billion rows. If needed, use a smaller source table. More frequent syncs can reduce incremental volumes, but not the initial load. Contact Insider One before launch if any sync could exceed the limit. Syncs above the limit are not processed, and the integration is archived.
Performance considerations
The size of the Snowflake warehouse affects throughput. Scale the warehouse appropriately for large initial imports or high data volumes.
When the same warehouse supports multiple integrations, make sure it has enough capacity for the combined workload. Schedule syncs at different times where possible to prevent workloads from overlapping.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions and connector-specific limitations, see Import Data from Snowflake to Insider One.
Google BigQuery
Google BigQuery imports run on a schedule. The first sync performs a full-table import, while subsequent syncs use the configured last-updated column to retrieve new or changed rows.
Limitations
Maximum rows per sync: 1 billion
Maximum row size: 350 KB
Available sync frequencies:
Every 15 minutes
Every 30 minutes
Every 45 minutes
Hourly
Daily
Journeys and data streams that run per row are suppressed when the data volume exceeds the journey-trigger threshold. For details, refer to Automatic journey-trigger throttling.
Maximum distinct event names per sync: 100. A sync above this limit is not processed and the integration moves to Action Required. For details, see TOO_MANY_UNIQUE_EVENTS.
Before launching, check that the initial full sync and each incremental sync will not exceed 1 billion rows. If needed, use a filtered view or smaller source table. More frequent syncs can reduce incremental volumes, but not the initial load. Contact Insider One before launch if any sync could exceed the limit. Syncs above the limit are not processed, and the integration is archived.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions and connector-specific limitations, see Send Google BigQuery Data to Insider One.
Databricks
Databricks imports use Delta tables with Change Data Feed enabled. The first sync imports the table's existing data, while subsequent syncs retrieve change records captured by Change Data Feed.
Limitations
Maximum records per sync: 1 billion
Maximum row size: 350 KB
Available sync frequencies:
Every 15 minutes
Every 30 minutes
Every 45 minutes
Hourly
Daily
Journeys and data streams that run per row are suppressed when the data volume exceeds the journey-trigger threshold. For details, refer to Automatic journey-trigger throttling.
Maximum distinct event names per sync: 100. A sync above this limit is not processed and the integration moves to Action Required. For details, see TOO_MANY_UNIQUE_EVENTS.
The 1-billion-record limit is calculated differently depending on the sync type:
Initial full sync: The limit applies to the table's total row count.
Incremental sync: The limit applies to all Change Data Feed records created between two syncs.
For incremental syncs, each Change Data Feed record counts toward the 1-billion-record limit, including repeated updates to the same row. Sync more often to reduce records accumulated between runs. For the first sync, use a smaller, pre-filtered source table if needed; views are not supported. Contact Insider One before launch if either sync type could approach the limit. Syncs exceeding it are not processed, and the integration is archived.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions, Change Data Feed requirements, and connector-specific limitations, see Import Data from Databricks to Insider One.
Amazon S3
Amazon S3 ingestion is event-driven. New files are detected through Amazon S3 event notifications delivered through Amazon SQS.
Amazon S3 ingestion does not use a scheduled sync interval.
Limitations
Maximum file size: 5 GB per file
Maximum files ingested: 25 files per day
Maximum total file volume: 20 GB per day
Maximum row size: 5 MB
Files larger than 5 GB are skipped. The combined size of all files processed in one day must not exceed 20 GB. Split the file into parts of 5 GB or less and upload the parts again. Keep within 25 files and 20 GB per day.
Rows larger than 5 MB are not upserted. Reduce the row to under 5 MB, usually by removing or shortening very long text values or large arrays, then upload the affected rows again in a new file. The rest of the file is still processed.
Journeys and data streams that run per row are suppressed when the data volume exceeds the journey-trigger threshold. For details, refer to Automatic journey-trigger throttling.
Maximum distinct event names per file: 100, checked on the first 1,000 rows of each file. A file above this limit is not processed, and the integration moves to Action Required. For details, see TOO_MANY_UNIQUE_EVENTS.
Concurrency and processing latency
No standard concurrent file-processing limit or completion-time range is publicly specified. Processing starts after Insider One receives the corresponding event notification.
Contact Insider One for detailed sizing based on your expected file volume, file size, and file arrival pattern.
For setup instructions, supported file formats, and connector-specific limitations, see Import Data from Amazon S3.
FTP and SFTP
FTP and SFTP imports read CSV or TSV files from your server on a schedule.
Limitations
Maximum file size: 2 GB per file. Keep files under 500 MB for best performance.
Directory scanning is not recursive.
API-based ingestion
API-based ingestion methods have endpoint-specific request, payload, batch, and rate limits.
For example, the Upsert User Data API has the following limits:
Maximum request size: 5 MB
Maximum records per request: 1,000 users
Shared rate limit: 25,000 requests per minute across the Upsert User Data API and Delete User Attribute API
Maximum record throughput with full batching: 25 million user records per minute
These limits apply specifically to the Upsert User Data API and Delete User Attribute API. Other endpoints can have different limits.
A 200 response from the Upsert User Data API means that Insider One accepted the request. It does not mean that every profile was written, so check the failed users in the response. Retry 429 and 5xx responses with backoff.
To reconcile counts with your source system, refer to Data Warehouse and Storage Import Monitoring and Error Guide.
See API Rate Limits for the current published limits for each API.
Plan your ingestion
Before launching an integration:
Estimate the maximum number of rows, change records, or files expected in a single run.
Confirm that each row, request, and file remains within its published size limit.
Select a sync frequency that allows enough time for the previous workload to complete before the next scheduled sync starts.
Consider the capacity of the source system, especially when multiple integrations use the same source compute resources.
Contact Insider One if the expected workload approaches a published limit or requires detailed throughput, concurrency, or completion-time planning.
If journeys or data streams must trigger on this data, keep the volume below the journey-trigger threshold.
Do not assume that an ingestion method has unlimited capacity when a figure is not listed. Contact Insider One for detailed sizing when your use case requires throughput, concurrency, or processing-time planning.