Scanning in batches: rate limits, cost, and what gets kept

Discovery routinely produces more candidates than it is sensible to read in one pass. Search is cheap and finds hundreds of pages; enrichment fetches and analyzes each one, and both the fetching and the analysis are rate-limited by providers that will not be hurried.

There are two ways to handle that. Run until you hit the limit and fail, or process a bounded slice and finish cleanly. Only one of them produces a usable product.

A completed slice is a real result

Enrichment takes a provider-safe number of candidates per pass, ordered by eligibility so the strongest evidence gets read first. When that slice finishes, the run is complete — even though candidates remain.

This is a deliberate framing and not a euphemism for stopping early. A run that enriched its best candidates and qualified the results has produced exactly what it set out to produce. The remaining candidates are not lost work; they are the next batch, sitting in score order, waiting.

A provider's rate limit affects throughput, not how many opportunities a workspace can retain over multiple batches. Those are different constraints and should not be confused.

Scan next batch

Monthly workspaces get an explicit Scan next batch control. It does exactly two things:

1. 01

   Enrich the next highest-scored slice

   The next candidates in eligibility order are fetched, read, and qualified — the same pipeline, on the pages that were just below the previous cut.

2. 02

   Rerank everything for the site

   Not just the new arrivals. Every active opportunity for that site is rescored and reordered together, so a new opportunity that beats an existing one appears above it rather than being appended to the bottom.

The second step is the one that makes batching honest. Appending new results to the end of an existing list would produce an ordering that is silently wrong: the top of the list ranked against one dataset, the bottom against another. Reranking globally is the only way a batched pipeline still yields a list you can work from top to bottom.

Saved is not the same as displayed

This is the distinction worth understanding about plan limits, because it is the opposite of how most tools behave.

Qualification persists every qualified opportunity it produces. The trial then displays the top 30. It does not qualify only 30 and discard the rest — the rest are stored, ranked, and complete with their evidence.

The consequence is that upgrading reveals intelligence that already exists, immediately. There is no rerun, no second bill for search and analysis you already paid for, and no waiting. The work was done; the plan was governing the view.

Why the work runs in the background

A discovery run can take a long time — long enough that no web request should be waiting on it. Starting one therefore creates a job and returns immediately, and the work runs as a durable background task with its own retries and schedule.

Progress is pushed to the run screen as each phase advances, so the wait is legible rather than a spinner. And because the jobs are durable rather than tied to a request, a transient provider failure in the middle of a long enrichment pass results in a retry, not a lost run.

What to do with the batch you have

The practical advice is the same regardless of plan: work the top of the list before scanning more. A batch produces a ranked set with reasons and contacts attached, and the value in it is realized by sending careful emails — not by accumulating a longer list of pages nobody has pitched.

When the top of the list is genuinely worked through, scan the next batch. The ranking will fold the new results into the old ones, and the next page you open will be the best one you have not yet acted on.