This guide summarizes the limits, sync frequency options, and performance considerations for the data ingestion methods supported by Insider One.
Use this page to compare ingestion methods and plan your data volume and sync schedule. For setup requirements, data mapping rules, and error behavior specific to a connector, refer to the related integration guide.
This guide lists the standard limits available for each ingestion method. If a throughput, concurrency, or processing-time figure is not specified, contact Insider One for detailed sizing based on your data volume, source configuration, and sync requirements.
How to interpret sync frequency
Sync frequency determines how often Insider One can start retrieving new or changed data from a source.
It does not represent a guaranteed processing or completion time. The time required to complete a sync can vary depending on:
The number of records
The size and complexity of each record
The volume of new or changed data
Source-system capacity
Concurrent workloads
Data validation results
Snowflake
Snowflake imports run on a schedule. The first sync performs a full-table import, while subsequent syncs import new or changed data.
Limitations
Maximum rows per sync: 1 billion
Maximum row size: 350 KB
Available sync frequency: Every 15 minutes up to once daily
The 1-billion-row limit applies to both the initial full sync and subsequent incremental syncs. If a sync exceeds this limit, the sync is not processed and the integration is moved to Archived.
Performance considerations
The size of the Snowflake warehouse affects throughput. Scale the warehouse appropriately for large initial imports or high data volumes.
When the same warehouse supports multiple integrations, make sure it has enough capacity for the combined workload. Schedule syncs at different times where possible to prevent workloads from overlapping.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions and connector-specific limitations, see Import Data from Snowflake to Insider One.
Google BigQuery
Google BigQuery imports run on a schedule. The first sync performs a full-table import, while subsequent syncs use the configured last-updated column to retrieve new or changed rows.
Limitations
Maximum rows per sync: 1 billion
Maximum row size: 350 KB
Available sync frequencies:
Every 15 minutes
Every 30 minutes
Every 45 minutes
Hourly
Daily
The 1-billion-row limit applies to both the initial full sync and subsequent incremental syncs. If a sync exceeds this limit, the sync is not processed and the integration is moved to Archived.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions and connector-specific limitations, see Send Google BigQuery Data to Insider One.
Databricks
Databricks imports use Delta tables with Change Data Feed enabled. The first sync imports the table's existing data, while subsequent syncs retrieve change records captured by Change Data Feed.
Limitations
Maximum records per sync: 1 billion
Maximum row size: 350 KB
Available sync frequencies:
Hourly
Daily
The 1-billion-record limit is calculated differently depending on the sync type:
Initial full sync: The limit applies to the table's total row count.
Incremental sync: The limit applies to all Change Data Feed records created between two syncs.
Change Data Feed records each change separately. If the same row is updated multiple times between two syncs, each update counts as a separate record.
If a sync exceeds the applicable limit, the sync is not processed and the integration is moved to Archived.
Concurrency and processing latency
No standard concurrency limit or completion-time range is publicly specified. Contact Insider One for detailed sizing.
For setup instructions, Change Data Feed requirements, and connector-specific limitations, see Import Data from Databricks to Insider One.
Amazon S3
Amazon S3 ingestion is event-driven. New files are detected through Amazon S3 event notifications delivered through Amazon SQS.
Amazon S3 ingestion does not use a scheduled sync interval.
Limitations
Maximum file size: 5 GB per file
Maximum files ingested: 25 files per day
Maximum total file volume: 20 GB per day
Maximum row size: 5 MB
Files larger than 5 GB are skipped. The combined size of all files processed in one day must not exceed 20 GB.
Rows larger than 5 MB are not upserted.
Concurrency and processing latency
No standard concurrent file-processing limit or completion-time range is publicly specified. Processing starts after Insider One receives the corresponding event notification.
Contact Insider One for detailed sizing based on your expected file volume, file size, and file arrival pattern.
For setup instructions, supported file formats, and connector-specific limitations, see Import Data from Amazon S3.
API-based ingestion
API-based ingestion methods have endpoint-specific request, payload, batch, and rate limits.
For example, the Upsert User Data API has the following limits:
Maximum request size: 5 MB
Maximum records per request: 1,000 users
Shared rate limit: 25,000 requests per minute across the Upsert User Data API and Delete User Attribute API
Maximum record throughput with full batching: 25 million user records per minute
These limits apply specifically to the Upsert User Data API and Delete User Attribute API. Other endpoints can have different limits.
See API Rate Limits for the current published limits for each API.
Plan your ingestion
Before launching an integration:
Estimate the maximum number of rows, change records, or files expected in a single run.
Confirm that each row, request, and file remains within its published size limit.
Select a sync frequency that allows enough time for the previous workload to complete before the next scheduled sync starts.
Consider the capacity of the source system, especially when multiple integrations use the same source compute resources.
Contact Insider One if the expected workload approaches a published limit or requires detailed throughput, concurrency, or completion-time planning.
Do not assume that an ingestion method has unlimited capacity when a figure is not listed. Contact Insider One for detailed sizing when your use case requires throughput, concurrency, or processing-time planning.