Send messages and large syncs
How to send chat and email, and how to bring in everything you already have.
In short
Send each message as its own document, and link replies to their thread. For a first big upload, send everything, retry freely, and watch the counts until nothing is left waiting.
One message, one document
Use the id your source already gives each message as its name. Threads are carried by links between documents, so never join several messages into one.
| Chat | ||
|---|---|---|
doc_id | Channel id and message timestamp | The Message-ID header, URL-encoded |
channel | chat | email |
occurred_at | The message timestamp | The Date header |
participants | The author, as author | From, To and Cc, as from, to and cc |
relations | For a reply: thread and in_reply_to, both naming the thread's first message | in_reply_to, and one thread naming the first message |
| An edit | Send the new text under the same name | Does not apply |
| A deletion | Delete the document | Delete the document |
Getting the most from messages
Each message is read on its own words, so a few habits help:
- State the account on every message you can, replies included. A short reply often has nothing to read, and without a stated account it is filed nowhere.
- Remove quoted history from email. You are already sending those earlier messages, and quoted text can be read as if the sender had just said it.
- Keep the links short. One
in_reply_toand onethreadare enough. A document can hold 8 relations and 24 participants, and more is refused with a422, so send the sender, direct recipients and people on the account.
Your first large sync
There is no batch route, so a sync is many ordinary uploads, one for each document.
Name documents from your source's own ids
Running the sync again then updates documents instead of duplicating them.
Send oldest first where you can
A newer value then replaces the older one and keeps it as history. If an older document arrives after a newer one, the newer value still leads, but the older one stays beside it as "Also on record". A document can link to one you haven't sent yet.
Retry anything that fails or times out
Sending the same contents to the same name never queues the document twice.
Watch progress
GET /agents/{agent_id}/documents:count gives the count in each state, and completed_last_10_minutes shows how fast the sync is being read.
Finish by listing what didn't make it
Add ?state=failed&state=paused to the document list.
Only a limited number of documents are read at once, so a large sync will sit in queued for a while. That is normal. If completed_last_10_minutes is above zero, the sync is moving.
Reading draws on your spend limit, so check it before a large sync. Documents that hit the limit show as paused.