Read public paginated pages with a stopping rule

Define the collection before opening pages

Write down the allowed source, record type, maximum pages and completion condition before collecting a paginated public view. The fact that a website is reachable does not establish permission to ignore its access rules or reuse restrictions. Prefer a documented feed, export or API when it meets the task. Browser collection is useful when the rendered public page is the available evidence surface. Keep the scope narrow enough that the agent can explain which pages it actually observed and why it stopped. An unbounded instruction to scrape everything is not a completion criterion.

Use page identity rather than click count

After each navigation or next-page action, record the observed URL, visible page marker and retrieval time. A click can fail, redirect or leave the same page visible. Counting clicks without checking page identity can produce duplicate records while appearing to advance. Lupin returns a bounded text snapshot, so a single page can also exceed the available observation. Distinguish pagination completeness from extraction completeness within each page. If the source exposes stable record identifiers, use them for deduplication; otherwise choose an explicit composite key and record the ambiguity it leaves.

Make each mutation a deliberate step

Navigate and click require stable Idempotency-Key values. Preserve a key across retries of the same step, and do not generate a new one merely because the response was lost. Read the current page to determine whether pagination advanced before choosing another action. Pass tab_id explicitly and keep one controller on that page. If the source opens a new tab for each detail view, maintain a small working set and close completed detail tabs. The fixed action surface limits its own new-tab creation, so a collecting agent should not rely on unlimited open pages.

{"action":"click","tab_id":"TARGET_ID_FROM_TABS","selector":"a[rel=next]"}

Bound traffic and runtime

Choose a conservative page budget and leave time for saving evidence and ending the browser. A finite runtime lease, inactivity timeout and target-site behavior all constrain the job. A blocked page, verification challenge or access-denied response is a condition to report, not an invitation to bypass the site's controls. Avoid fetching the same page repeatedly while a loading indicator is unchanged. If the task cannot finish within its approved budget, preserve the exact last observed page and completed record set in the task's storage. That record supports a planned later attempt; it does not preserve browser cookies or a closed session.

Report coverage explicitly

A useful collection result states the pages read, records retained, duplicates removed, failed pages and stopping reason. Distinguish reaching the site's final page from reaching the user's maximum-page budget. When a page snapshot was truncated, identify which fields or records could be incomplete. Link conclusions back to their source pages rather than presenting the collected text as a new authoritative dataset. End the session and confirm cleanup independently of the collection outcome. This approach produces a bounded, reviewable public-data task without promising complete coverage that the browser observations cannot establish.

Sources & verification

Reviewed against the current Lupin route, provider adapter and shared input contracts, plus the linked official references. Examples were not run against live services.

• dom.spec.whatwg.org ↗Checked 2026-11-05
• chromedevtools.github.io · devtools-protocol · tot · Page ↗Checked 2026-11-05

Prepared with AI assistance by Lupin editorial. How we research and maintain these guides.