Data Sources & API

How to send large amounts of data to a data source

Last updated

When you send thousands of records to a data source, group them into batches and send the batches one after another. It's faster than one request per record, and it keeps every request within Sendtric's limits.

This applies to the Sendtric API and to Zapier and n8n data sources.

The limits

LimitValue
Records per request10,000
Request size10 MB
Identifier value255 characters
Any single text value10 KB
Nesting depth20 levels

A request over the record or size limit is rejected with status 413, and nothing in it is saved.

Good batching habits

  • Send arrays, not single records. One request with 50 records does the same work as 50 requests, with far less overhead.
  • Send batches one after another, not all at once in parallel.
  • Pause briefly between batches when you're sending thousands of records.
  • Retry on 503 and network errors. A 503 response includes a Retry-After header with the number of seconds to wait. Then send the same batch again.
  • Don't retry 400, 401, 404 or 422 without fixing the request first. Those mean something in the request needs to change.

Resending a batch is safe. A record with an identifier that already exists replaces the old one, so a retried batch never creates duplicates.

n8n: 1,000 to 5,000 records

For a few thousand records, the HTTP Request node's own batching is enough. Open the node that sends to Sendtric and set:

  1. On the Parameters tab, under Options, click Add option and add:
    • Batching: Items per Batch 50 and Batch Interval (ms) 5000.
    • Timeout: 30000.
  2. On the Settings tab:
    • Turn on Retry On Fail.
    • Max. Tries: 3.
    • Wait Between Tries (ms): 3000.
    • On Error: Stop Workflow.
n8n HTTP Request node options with Items per Batch set to 50, Batch Interval 5000 ms and Timeout 30000
n8n HTTP Request node Settings tab with Retry On Fail on, Max. Tries 3, Wait Between Tries 3000 ms and On Error set to Stop Workflow

With these settings n8n still sends one request per record, 50 at a time, then waits 5 seconds before the next 50.

n8n: more than 5,000 records

For larger data sets, use our ready-made workflow. It groups every 50 records into one request and sends the groups one after another, so 10,000 records become 200 requests instead of 10,000.

The workflow has four nodes:

  1. Google Sheets Trigger reads your sheet. It checks for changes every minute.
  2. Loop Over Items splits the rows into batches of 50.
  3. Code in JavaScript wraps each batch into a single request body.
  4. HTTP Request sends the batch to Sendtric, retries if it fails, then hands back to the loop for the next batch.

Set it up

  1. In Sendtric, create an n8n data source and copy its webhook endpoint and API key. See how to send records from Zapier or n8n.
  2. Copy the workflow JSON below.
  3. In n8n, create a new workflow and paste the JSON onto the empty canvas (or save it as a .json file and use Import from File).
  4. Open Google Sheets Trigger, connect your Google account, then choose your spreadsheet and sheet.
  5. Open HTTP Request:
    • Replace the URL with your Sendtric n8n webhook endpoint.
    • In the Authorization header, replace YOUR_API_KEY with your key. Keep the word Bearer and the space before it.
  6. Run the workflow once to test it, check the records on the data source's Records tab in Sendtric, then activate the workflow.

Your sheet needs a column with the same name as the data source's unique identifier field, filled in on every row.

Need data from somewhere other than Google Sheets? Replace the trigger with any node that outputs your records, and connect it to Loop Over Items.

Workflow JSON

{
  "name": "Google Sheets to Sendtric data source",
  "nodes": [
    {
      "parameters": {
        "pollTimes": { "item": [{ "mode": "everyMinute" }] },
        "documentId": { "__rl": true, "value": "", "mode": "list" },
        "sheetName": { "__rl": true, "value": "", "mode": "list" },
        "options": {}
      },
      "type": "n8n-nodes-base.googleSheetsTrigger",
      "typeVersion": 1,
      "position": [0, 16],
      "id": "e8a0600f-7814-44c9-9960-22b9eb6dc708",
      "name": "Google Sheets Trigger"
    },
    {
      "parameters": {
        "batchSize": 50,
        "options": { "reset": false }
      },
      "type": "n8n-nodes-base.splitInBatches",
      "typeVersion": 3,
      "position": [240, 16],
      "id": "05d065b5-a82b-49ea-8f0b-bd5e3116c694",
      "name": "Loop Over Items"
    },
    {
      "parameters": {
        "jsCode": "// Collect every record in this batch\nconst batchItems = $input.all().map(item => item.json);\n\n// Send the batch as one request body\nreturn [{\n  json: {\n    payload: batchItems\n  }\n}];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [464, 16],
      "id": "1c93cf1d-6636-4d30-b732-df749cbcb9bc",
      "name": "Code in JavaScript"
    },
    {
      "parameters": {
        "method": "POST",
        "url": "PASTE_YOUR_SENDTRIC_N8N_WEBHOOK_ENDPOINT",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [{ "name": "Authorization", "value": "Bearer YOUR_API_KEY" }]
        },
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify($json) }}",
        "options": {}
      },
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.5,
      "position": [688, 0],
      "id": "26a6141b-faa8-457c-927f-6ab5ef94a6ec",
      "name": "HTTP Request",
      "retryOnFail": true,
      "waitBetweenTries": 5000
    }
  ],
  "connections": {
    "Google Sheets Trigger": {
      "main": [[{ "node": "Loop Over Items", "type": "main", "index": 0 }]]
    },
    "Loop Over Items": {
      "main": [[], [{ "node": "Code in JavaScript", "type": "main", "index": 0 }]]
    },
    "Code in JavaScript": {
      "main": [[{ "node": "HTTP Request", "type": "main", "index": 0 }]]
    },
    "HTTP Request": {
      "main": [[{ "node": "Loop Over Items", "type": "main", "index": 0 }]]
    }
  },
  "settings": { "executionOrder": "v1" }
}

Check the result

Records are usually available within about a minute of being accepted. Open the data source and use the Records tab to search for a few identifiers and confirm the values look right.

If a response reports skipped records, the X-Skipped-Records header says how many. They're usually rows missing the identifier field, or with a value over 10 KB.

Related

Was this article helpful?

Click the image to zoom in. Drag or scroll to look around.