This document describes the various configuration parameters for the config.yaml file. You can fine-tune them to suit your application's needs.
These configuration parameters are also applicable for the
config.toml file, in case you have an older RudderStack deployment.To best understand some of the terms covered in this guide, we recommend going through the RudderStack Architecture first.
Global Parameters
| Parameter name | Type | Description | Default value |
|---|---|---|---|
maxProcess | Int | Number of parallel threads used in server. Should be set to number of cores. | 12 |
gwDBRetention | String | The time for which the events are stored in the Gateway database after they have been processed. Examples: 1h, 30m, 35s, etc. | 0h |
routerDBRetention | String | The time for which the events are stored in the Router database after they have been processed. Examples: 1h, 30m, 45s, etc. | 0h |
enableProcessor | Boolean | This variable enables or disables the Processor module. It will be set to false when running in Degraded mode. | true |
enableRouter | Boolean | This variable enables or disables the Router module. Will be set to false when running in Degraded mode. | true |
enableStats | Boolean | This variable enables or disables stats. | true |
For more information on the Processor, Gateway, and Router modules of the RudderStack backend, refer to the RudderStack Architecture.
[Gateway]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
webPort | Int | The port on which the RudderStack server runs. | 8080 |
maxUserWebRequestWorkerProcess | Int | RudderStack spawns this specified number of processes to consume the events at a user level. | 64 |
maxDBWriterProcess | Int | As requests come in to the Gateway and are batched, RudderStack runs the maxDBWriterProcess writers to send these batches to the database and the Configuration Backend. | 256 |
CustomVal | String | For the creation of a job in the backend PostgreSQL database (jobsDB), the value of this variable will be assigned to the CustomVal column. | GW |
maxBatchSize | Int | The batch size used in the Gateway. The requests are batched up to this size before writing to the database. | 32 |
batchTimeout | String | In case the request batches do not meet the maxBatchSize, the batches are are sent to the database and the Configuration Backend in this interval. | 20ms |
maxReqSizeInKB | Int | An error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses this value. | 4000 |
enableDedup | Boolean | Enables or disables deduplication of events. RudderStack uses message_id to de-dup. The duplicate events are dropped at the Gateway. | false |
dedupWindow | String | Events with the same message_id within this timeframe are considered duplicate and are dropped. | 3600s |
enableRateLimit | Boolean | Rate limits the number of requests accepted by the Gateway. This is used for running the RudderStack-hosted service. | false |
[SourceDebugger]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
disableEventUploads | Boolean | Enables or disables the event schema upload. | false |
maxBatchSize | Int | The maximum size of the live events batch sent to the Configuration Backend. | 32 |
maxESQueueSize | Int | The maximum size of the live events queue in the memory. | 1024 |
maxRetry | Int | The maximum number of attempts RudderStack makes to upload the request batches, in case of any errors. | 3 |
batchTimeout | String | In case the request batches do not meet the maxBatchSize, the request batches are uploaded to Configuration Backend in this time interval. | 2s |
retrySleep | String | In case of errors while uploading the request batches, RudderStack waits for this time interval before retrying. This is done until the maxRetry limit is reached. | 100ms |
[JobsDB]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
jobDoneMigrateThres | Float64 | If (deletedJobsCount / totalJobsCount) > jobDoneMigrateThres, RudderStack migrates the table of the jobs that have been processed. | 0.8 |
jobStatusMigrateThres | Float64 | If ( RudderStack migrates the table of jobs that have been processed. | 5 |
maxDSSize | Int | The maximum size of a table. If the maximum size is reached, RudderStack migrates all the jobs in that table. | 100000 |
maxMigrateOnce | Int | The maximum number of tables that can be migrated together. | 10 |
maxTableSizeInMB | Int | The maximum size of the tables in MB. The tables are migrated if they cross this limit. | 300 |
migrateDSLoopSleepDuration | String | The time RudderStack waits before migrating the datasets. | 30s |
maxMigrateDSProbe | Int64 | The maximum number of tables probed to find if they can be migrated. | 10 |
addNewDSLoopSleepDuration | String | The time RudderStack waits before adding a dataset. | 5s |
backupCheckSleepDuration | String | RudderStack waits for this time interval before checking if a dataset needs backing up. | 5s |
enableBackup | Boolean | Enables or disables the backup. This is set to false in the Degraded mode. | true |
[Router]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
jobQueryBatchSize | Int | The size of a jobs batch to get from the database. This includes the retry list, processing list, unprocessed list, or executed list. | 10000 |
updateStatusBatchSize | Int | The minimum size needed to update the status of a batch of jobs. | 1000 |
readSleep | String | The time RudderStack waits before fetching the next jobs batch from the database, in case the length of unprocessed and retry list is 0. | 1000ms |
noOfWorkers | Int | RudderStack starts this number of workers to send events to the destinations. | 64 |
noOfJobsPerChannel | Int | The number of jobs a channel in each worker can contain. | 1000 |
maxSleep | String | The time to wait when the response status code is not 200. This is to give RudderStack some time before the next retry. | 60s |
minSleep | String | The time to sleep when the response status code is 200. | 0s |
maxStatusUpdateWait | String | The time to sleep before ending one round of stats collection. | 5s |
useTestSink | Boolean | Runs internal tests if set to true. | false |
maxFailedCountForJob | Int | The maximum number of times a job can fail before it is marked as aborted. | 8 |
guaranteeUserEventOrder | Boolean | RudderStack maintains the order of user events if set to true. | true |
retryTimeWindow | String | The minimum retry window in case of 5XX, 429 errors. | 180m |
minRetryBackoff | String | The minimum time before the next retry in case of 5XX, 429 errors. | 10s |
maxRetryBackoff | String | The maximum allowed time between the errors in case of 5XX, 429 errors. | 300s |
[BatchRouter]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
mainLoopSleep | String | The timeout while running the main loop. | 2s |
noOfWorkers | Int | The number of workers to batch jobs before deletion. | 8 |
jobQueryBatchSize | Int | The number of events picked up from the batch router's database (Jobs DB) in each query. | 100000 |
uploadFreq | String | The frequency with which the batch router dumps the events to the storage destinations. | 30s |
maxFailedCountForJob | Int | The maximum number of times a job can fail before marking it as aborted. | 128 |
[Warehouse]
You can ignore these settings if you don't have any Warehouse Destinations configured in RudderStack.
| Parameter name | Type | Description | Default value |
|---|---|---|---|
stagingFilesTable | String | Table name of the staging files. | wh_staging_files |
loadFilesTable | String | Table name of load files. | wh_load_files |
uploadsTable | String | Table name of uploads. | wh_uploads |
schemasTable | String | Table name of schemas. | wh_schemas |
uploadFreq | String | The frequency of the upload in seconds. | 1800s |
noOfWorkers | Int | Number of concurrent writes to the warehouse. | 8 |
mainLoopSleep | String | The time RudderStack waits between multiple warehouse writes. | 5s |
stagingFilesBatchSize | Int | The batch size of the staging files. | 960 |
[Processor]
| Variable name | Type | Description | Default value |
|---|---|---|---|
loopSleep | String | In case the length of the user jobs process queue is 0 or the unprocessed and retry list is empty, RudderStack sleeps for this specified time. | 10ms |
maxLoopSleep | String | Maximum loop sleep time for the Processor. | 5000ms |
dbReadBatchSize | Int | The total number of events to get as a batch from the database. | 10000 |
transformBatchSize | Int | Batch size of the events added to the request queue before sending them for transformation. | 100 |
userTransformBatchSize | Int | Batch size of the events added to request queue before sending them to the custom transformation server. NOTE: This is used only when a user transformation function is connected to a destination. | 200 |
sessionThresholdEvents | Int | The minimum number of events needed to be process further. | 20 |
sessionThreshold | String | The minimum time needed before a new session is created. | 10s |
maxChanSize | Int | The maximum channel size for the request and response queue in the transformer. | 2048 |
processSessions | Boolean | If set to true, the status of the job in the database is updated. | false |
numTransformWorker | Int | Specifies the number of Go transform workers. | 8 |
maxRetry | Int | The maximum number of times a transformer retries hitting the API in case of an error. | 30 |
retrySleep | String | The sleep time in case of a transformer error while hitting the API, before retrying. | 100ms |
[BackendConfig]
| Variable name | Type | Description | Default value |
|---|---|---|---|
pollInterval | String | The frequency of updating data from the Configuration Backend. | 5s |
configFromFile | Boolean | When set to true, RudderStack reads the backend workspace configuration from a JSON file instead of fetching it from the API. | false |
configJSONPath | String | The path of the JSON file which contains the backend workspace configuration. | /etc/rudderstack /workspaceConfig.json |
[RateLimit]
| Variable name | Type | Description | Default value |
|---|---|---|---|
eventLimit | Int64 | The maximum number of events to be allowed in a time interval. | 1000 |
rateLimitWindow | String | The rolling time interval used to limit the allowed number of events. | 60m |
noOfBucketsInWindow | Int32 | The number of buckets rateLimitWindow is broken down into. | 12 |
[Diagnostics]
| Parameter name | Type | Description | Default value |
|---|---|---|---|
enableDiagnosis | Boolean | RudderStack sends the server diagnostics report to the user. Disabling this will disable sending all diagnostics information. | true |
gatewayTimePeriod | String | The time interval to send the Gateway requests report | 60s |
routerTimePeriod | String | The time interval to send the Router requests report. | 60s |
batchRouterTimePeriod | String | The time interval to send the Batch router requests report. | 10m |
enableServerStartMetric | Boolean | Sends the server start event. | true |
enableConfigIdentifyMetric | Boolean | Sends the workspace config received event. | true |
enableServerStartedMetric | Boolean | Sends the successful server start event. | true |
enableConfigProcessedMetric | Boolean | Sends the workspace config details. | true |
enableGatewayMetric | Boolean | Sends the Gateway request metrics. | true |
enableRouterMetric | Boolean | Sends the Router request metrics. | true |
enableBatchRouterMetric | Boolean | Sends the Batch Router request metrics. | true |
enableDestinationFailuresMetric | Boolean | Sends the destination failures metrics. | true |
Contact us
For queries on any of the sections covered in this guide, you can contact us or start a conversation in our Slack community.