Watched pages

In short

Give a web address and a schedule, and the pages found there are collected and stored as documents, so your agents can answer from them. A page that has changed updates its document on the next run.

Switched on per account

Watched pages are off until they are switched on for your account. Until then every route here answers 403 with URI browser ingest is off for this tenant.

Start watching a page

Save a watch with a web address, a schedule and a short note saying which pages matter. Required scope: agents:write.

curl -X POST "https://api.hyperstruck.com/agents/<AGENT_ID>/ingest-sources" \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "seed_url": "https://wenlow-timber.example/price-list",
    "cadence": "daily",
    "meta_prompt": "The price list and any page it links to about delivery charges.",
    "max_pages": 10
  }'

It answers 201 with the saved watch, including its source_id and next_run_at. The first run comes one cadence after saving, or call run-now to start sooner. Use source_id in every other route.

To check what a watch would do before saving it, send the same body to POST /agents/{agent_id}/ingest-sources:compile. It returns the settings the watch would run with, and saves nothing.

Field	What it does
`seed_url` (required)	The page to start from. It must be `https`: an IP address or an `http` address is refused when you save. A host that turns out to be private fails the run instead. Campaign tracking parameters are removed.
`cadence` (required)	How often to run: `15m`, `1h`, `6h`, `daily` or `weekly`.
`meta_prompt` (required)	Which pages matter, up to 8,000 characters. It chooses among the links found. Include `this page only` and only the page at `seed_url` is collected.
`max_pages`	Pages per run, 1 to 25. Defaults to 10.
`respect_robots`	Follow the site's robots rules. Defaults to `true`.
`include_pdf_documents`	Also collect PDFs the pages link to, up to 10 per run. Defaults to `false`.
`timezone`	Defaults to `UTC`.
`source_id`	Your own name for the watch: lowercase letters, digits and hyphens, 2 to 63 characters. Made from the address when you leave it out.

A watch follows links only on the host you gave it and its www twin. Pages behind a login can't be watched: needs_session: true is refused.

Find the pages it collected

Each page becomes an ordinary document, so it appears in the document list, its claims cite it, and a claim can link back to the page. Its doc_id is made from its address (url:https/wenlow-timber.example/price-list), so a later run updates the same document instead of adding another. A very long address gets a source_ref: id in its place.

Check on it

GET /agents/{agent_id}/ingest-sources/{source_id}/runs lists recent runs, newest first. POST /agents/{agent_id}/ingest-sources/{source_id}/run-now starts one straight away and answers 202 with its run_id. While that run is still waiting to start, calling it again returns the same run_id and starts nothing. Once the run is collecting, calling it again answers 409.

A 202 doesn't mean the pages can be answered from yet. They are read like any other document, so check them with state.

Run field	What it tells you
`status`	`pending`, `processing`, `completed`, `cancelled` or `dead`. A `dead` run has stopped trying, and `last_error` says why.
`page_count`	Pages collected.
`document_count`	PDFs stored. A run that skipped as unchanged stores none.
`document_shortfall`	PDFs it collected but couldn't store.
`skipped_unchanged`	`true` when nothing had changed since the last run, so nothing new was sent.
`last_error`	Why the run stopped, such as your spend limit.

Change, pause or stop

To	Call	What happens
Change settings	`PATCH /agents/{agent_id}/ingest-sources/{source_id}`	Send only what changes: `seed_url`, `cadence`, `meta_prompt`, `max_pages`, `respect_robots`, `include_pdf_documents` or `timezone`.
Pause or resume	The same `PATCH`, with `status` set to `paused` or `active`	A paused watch doesn't run, and pausing cancels a live run.
Stop the current run	`POST /agents/{agent_id}/ingest-sources/{source_id}/stop`	The watch stays active. Safe to repeat when nothing is running.
Remove the watch	`DELETE /agents/{agent_id}/ingest-sources/{source_id}`	Live runs are canceled. Documents already stored stay, and you delete them like any other.

If a run fails to collect three times it ends as dead and the watch's status becomes failing. A failing watch doesn't run until you PATCH its status back to active.

GET /agents/{agent_id}/ingest-sources lists the agent's watches. Reading needs agents:read.

What it costs

Each run holds an amount against your spend limit when it starts, sized by max_pages. Stopping a run that hasn't started releases the hold, and stopping one already collecting counts the hold as used. Reading the collected pages then draws on your spend limit like any other document.

If your limit is reached, the run stops and last_error says so.

Errors

Status	When
`403`	Watched pages are off for your account.
`404`	`Ingest source not found`, or `No such agent`.
`409`	A watch with that `source_id` already exists, or `run-now` was called on a watch that isn't active or on one with a run already collecting.
`422`	A setting was refused, and the message says which: an address that isn't allowed, a schedule outside the list, or a blank note.
`503`	Page collection isn't available right now.

Documents

What happens to each page once it is collected.

Documents API

List, fetch and erase what a watch stored.