Extending Browserless to Capture Initial XHRs with Listener-First Scripts

The first version of my VPS crawl engine could render pages. The missing piece was subtle: scripts that listen for XHRs need to install their listeners before navigation starts. Here is how I extended the relay to make that possible.

The crawl engine already had the basics: an authenticated HTTP relay, proxy-level routing, Browserless for JavaScript rendering, and Cloudflare Tunnel in front of the VPS. That is enough for static pages and normal screenshots. It is not enough when the data you need arrives in the first wave of network requests.

If the page has already started loading before your script registers page.on('response'), the most important XHR may already be gone.

The bug: render-first was too late

The original render flow was simple: receive a URL, call Browserless, navigate with page.goto(url), then return HTML. Later I added a custom script option so callers could run Puppeteer code after the page loaded.

That worked for DOM extraction, but it failed for initial-load APIs. Modern sites often request feed data, search results, or product JSON immediately during navigation. If the relay navigates before the custom script runs, the script cannot observe those first responses.

Old flow	Problem
Relay calls page.goto(url)	Network traffic begins immediately
Custom script starts after navigation	Listeners are registered too late
Script waits for matching XHR	The target response may already have completed

The fix: make custom scripts own navigation

The extension was to split Browserless into two modes:

• Default mode: no script is provided, so the relay auto-navigates and returns HTML as before.
• Script mode: a script is provided, so the relay does not navigate. It opens a blank page, injects context, and lets the script register listeners before calling page.goto.

This one branch changes the behavior from “render this URL” to “give me a programmable browser with safe context.”

if (!context.userScript) {
  await page.goto(context.url, {
    waitUntil: context.waitUntil || 'networkidle2',
    timeout: context.gotoTimeout || 45000
  });
}

// If userScript exists, leave the page at about:blank.
// The script is now responsible for navigation.

Design rule: listener-first scripts must call await page.goto(context.url, ...) themselves. That is intentional. It lets the script install page.on('request'), page.on('response'), console hooks, or route interception before the first byte of the real page load.

The POST shape

The public API stayed small. A caller posts JSON to /browse with a URL, an optional proxy level, and a stringified Puppeteer function body.

POST /browse
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "url": "https://example.com/feed",
  "level": 2,
  "wait_until": "domcontentloaded",
  "script": "async ({ page, context }) => { /* puppeteer */ }",
  "output_type": "json"
}

The relay passes safe parameters into context: url, waitUntil, waitMs, and gotoTimeout. It does not pass secrets, proxy credentials, or internal file paths into the script.

A listener-first script

The caller can now register a response listener, then navigate, then scroll or wait until the interesting API response appears.

async ({ page, context }) => {
  const matches = [];

  page.on('response', async (response) => {
    const url = response.url();
    if (!url.includes('/api/') || !url.includes('item')) return;

    try {
      const body = await response.text();
      matches.push({ url, status: response.status(), body: body.slice(0, 5000) });
    } catch (err) {
      matches.push({ url, status: response.status(), error: String(err) });
    }
  });

  await page.setUserAgent('Mozilla/5.0 Chrome/124 Safari/537.36');
  await page.goto(context.url, {
    waitUntil: context.waitUntil || 'domcontentloaded',
    timeout: context.gotoTimeout || 45000
  });

  await page.waitForTimeout(context.waitMs || 5000);
  await page.evaluate(() => window.scrollBy(0, document.body.scrollHeight));
  await page.waitForTimeout(3000);

  return { finalUrl: page.url(), captured: matches.length, matches };
}

Returning real JSON, not double-encoded text

Another small but important change: script results should return as objects when they are objects. The relay should not always JSON.stringify inside Browserless and then try to parse strings later. That creates awkward double-encoding and brittle response handling.

The cleaner contract is:

• If the script returns an object, respond with application/json.
• If the script returns a string, respond with plain text unless the caller requested JSON.
• If Browserless throws, wrap the error with status and a short message.

Keeping it safe

This kind of endpoint is powerful, so I kept it private-by-default:

• Require a bearer token on every request.
• Bind the relay to localhost and expose it only through Cloudflare Tunnel.
• Put short timeouts on navigation and script execution.
• Strip forwarding headers when making outbound fetches.
• Use proxy levels instead of letting callers provide raw proxy credentials.
• Plan SSRF blocking so internal IP ranges cannot be targeted by user-provided URLs.

In other words, callers choose a policy such as level: 0, level: 1, or level: 2. They do not receive the underlying proxy secrets.

The final architecture

The extension turns the relay into a small programmable browser runner:

Layer	Role
Client or Worker	Calls /browse with URL, level, and optional script
Cloudflare Tunnel	Public HTTPS entrypoint without exposing the VPS port
Flask/Gunicorn relay	Auth, validation, proxy-level selection, Browserless request shaping
Browserless	Runs Puppeteer and returns script output or rendered HTML
Proxy provider	Optional private or residential egress selected by policy

The result is a crawl engine that can still do boring HTML rendering, but can also run targeted browser instrumentation when a page hides its useful data in early XHRs.

---

The key lesson: when scraping dynamic sites, navigation timing is part of the API. If your abstraction hides navigation inside the server, you may accidentally hide the only moment when the data is observable. Listener-first mode gives that control back to the script while keeping credentials, egress policy, and browser infrastructure centralized.