Start here: Web Scraping API · API Center

Scraper Debugging Solution

Reproduce a 403 or empty DOM in minutes — change one variable at a time. This is a troubleshooting page, not a place to buy every proxy SKU.

Recommended: paste the failing URL in Web Scraping API Center. If production uses extract endpoints, compare the same curl across one residential and one datacenter sample — then stop.

Open API Center OpenAPI Center →
Who it's for

Engineers stuck on a single failing URL

Backend devs

Works on laptop, fails in prod — isolate IP vs render vs headers.

QA

Canary URLs before a crawl ship.

Data platform

Decide residential vs API with evidence, not Slack opinions.

Consultants

Attach HTML proof to a client ticket.

Pain Points

Stop shotgun-buying products

The failure is usually one of: datacenter ASN, missing render, wrong geo, or cookie rotation. You do not need mobile + business + custom to find out.

🐛

Heisenbugs

Laptop IP ≠ prod IP.

🚫

403 with no body

Cannot tell ASN vs WAF vs auth.

🌍

Silent geo

Wrong catalog saved as truth.

⚙️

Too many knobs

Changing IP, UA, and render together proves nothing.

🔐

Cookie loss

Rotation mid-login looks like a code bug.

💸

Largest pack first

You can diagnose on trial credit.

What to buy

A debug matrix, not a shopping list

Recommended: Web Scraping API Center (toggle render/geo). Alternative: one residential extract curl if production is proxy-based. Do not buy the other seven SKUs to debug.

Product Billing Use it when…
Web Scraping API Recommended Credits · pay per success Recommended — fastest repro: render, super, geoCode, no local browser.
Residential · Rotating Pay per GB Alternative — production uses extract endpoints; confirm residential vs your current IP.
Recommended path

How to start (one recommended path)

Path A — API Center bisect (recommended)

One URL, one flag at a time.

  1. Paste the failing URL.
  2. Fetch with defaults; note length and title.
  3. Toggle only render=true; compare.
  4. Then geoCode; then super. Copy the first working curl into the ticket.
API Center →

Path B — Production is proxy extract

Match the prod protocol.

  1. Generate one residential rotating endpoint.
  2. Run the same curl as prod.
  3. If residential works and DC fails, the bug is ASN — switch the job, do not rewrite parsers.
  4. If both fail, it is markup/headers — try Path A render.
OpenAPI Center →
Decision guide

Pick one — do not mix these on the same job

Use API Center when

  • You need HTML fast.
  • You suspect JS or anti-bot.
  • You do not want to touch prod credentials.

Use a residential sample when

  • Prod already speaks HTTP/SOCKS extract.
  • You must match prod headers exactly.
  • API success would not prove the proxy path.
Getting started

First-week checklist

FAQ

Common questions

Is this remote localhost tunneling?

No. Outbound scrape/proxy QA only.

403 on everything?

Try API render, then one residential curl. If both fail, it is likely headers or ToS — not a missing SKU.

Empty 200?

Enable render. Do not buy more GB.

Need all nine proxies?

No. Two tests diagnose most tickets.

CI?

Five canary URLs in API or extract — not a new product line.

Resources

Go deeper

API Center OpenAPI Center Data scraping Help Center

Reproduce the failing URL once

API Center first. One residential curl if prod is proxy. Then buy only the SKU that actually worked.

Open API Center OpenAPI Center →