# Web context

> Scrape, map, and crawl public web pages.

Source page: https://docs.replynodes.com/docs/guides/web

# Web context

The `web` tag in OpenAPI exposes three read operations:

- `GET /v1/webcontext/scrape` accepts `url` plus optional selector filters and returns page content.
- `GET /v1/webcontext/map` accepts `url` and discovers same-site URLs.
- `GET /v1/webcontext/crawl` accepts `url`, `max_pages`, and `max_depth` for bounded same-origin crawling.

Use the [API Reference](https://docs.replynodes.com/reference) for each operation's parameter and response schema.

## Limiting scrape output

The optional `max_chars` parameter is an integer from 1 through 1048576. It counts Unicode code points, not bytes or model tokens; as a rough guide, there are 4 code points per English token. If a longer page is cut, it is cut at the last line break in the second half of the budget; otherwise it is cut at the last whitespace there; otherwise it is cut exactly at the budget. Trailing whitespace is trimmed and nothing is appended. `data.truncated` is a bool, `data.markdown_chars` is the length of the full page Markdown, and `data.links` / `data.images` are listed from the full page rather than the excerpt. The same price applies with or without `max_chars`. A non-integer or out-of-range value returns `400 invalid_request` before any fetch and is not charged.

```sh
curl -sS -H "Authorization: Bearer $REPLYNODES_API_KEY" "https://api.replynodes.com/v1/webcontext/scrape?url=https://example.com&max_chars=2000"
```

## Free markdown by URL (no API key)

A no-signup, zero-auth endpoint returns the content of one public page as markdown:

```sh
curl -sS https://md.replynodes.com/stripe.com
```

The request is a bare host with an optional path: `https://md.replynodes.com/{domain}` returns the home page and `https://md.replynodes.com/{domain}/{path}` returns a page inside it. The response is `text/markdown; charset=utf-8` — the page's text with its headings, links, and lists kept — so a client can read or store it without an HTML parser.

Response headers carry the cache and quota state of the request: `X-Cache` (`hit` when the URL was served from cache, `miss` when the host had to fetch it — the query string is part of the cache key, so repeating a URL with a new query parameter produced a `miss` in testing), `X-RateLimit-Limit` (100), `X-RateLimit-Remaining`, `X-RateLimit-Reset`, and `X-Request-Id`.

One request is one page: this endpoint does not crawl, follow links, or paginate, so issue a request per page and use the `webcontext` operations above when you need crawl or map behaviour. A request it cannot serve is answered as the standard JSON error envelope instead of markdown — a host that does not resolve returns `400` `invalid_request` with a `request_id` — so check the status and the content type before storing a response.

**`HEAD` describes the same page a `GET` returns.** `curl -I https://md.replynodes.com/stripe.com` answers `200` with the same `content-length` and `content-type` the `GET` returns and no body, so a size or freshness preflight is safe. The quota still treats it as a request: every call, `HEAD` included, spends one unit of the rate limit.

### Output formats

Choose the response format with headers; the query string belongs to the target URL.

| Request headers | Response `Content-Type` | Body |
| --- | --- | --- |
| none, `Accept: */*`, `Accept: text/markdown`, `Accept: text/plain` | `text/markdown; charset=utf-8` | Markdown (unchanged) |
| `X-Format: frontmatter` | `text/markdown; charset=utf-8` | YAML front matter + Markdown |
| `Accept: application/json` (and no `X-Format`) | `application/json` | `{data, meta}` envelope |
| `X-Format: markdown` | as the default row | Markdown |
| `X-Format` with any other value | `application/json` | `400 invalid_request` |
| `Accept: application/json` **and** `X-Format: frontmatter` | `application/json` | `400 invalid_request` (conflicting) |

```sh
curl -sS -H 'Accept: application/json' https://md.replynodes.com/stripe.com
curl -sS -H 'X-Format: frontmatter' https://md.replynodes.com/stripe.com
```

Each format spends one quota unit. Successful responses set `Vary: Accept, X-Format`.

The logo counterpart of this endpoint is documented under [Brand intelligence](https://docs.replynodes.com/docs/guides/brand-intelligence); the product overview for this one is [replynodes.com/markdown-api](https://replynodes.com/markdown-api/).

### Selectors

Filter the page with request headers:

| Request header | Meaning | Example |
| --- | --- | --- |
| `X-Include-Selectors` | Keep only matching elements | `article,.post-body` |
| `X-Exclude-Selectors` | Remove matching elements | `nav,footer,.sidebar` |

Selectors are comma-separated, with at most 10 entries and 512 bytes per header. Duplicates are removed, and control characters or `{` `}` `;` `<` `>` are rejected. Violations return `400 invalid_request` with `reason: bad_option` and spend no quota. A page's main heading may remain in the output even when it matches an exclude selector.

```sh
curl -sS -H 'X-Exclude-Selectors: nav,footer' https://md.replynodes.com/stripe.com
```

Filtered and unfiltered pages are cached separately. If a page returns an access-check page with selectors set, the response is `502 upstream_blocked` because the crawl fallback is not used with selectors.

### Errors

Every error is the standard JSON envelope with unchanged `error.code` and HTTP status, while two extra fields help a program decide what to do: `error.reason` is a stable value from the table, and `error.retryable` is true when sending the identical request again later can succeed without changing anything; it says nothing about how soon, which `X-RateLimit-Reset` / `Retry-After` provide.

| HTTP | `error.code` | `error.reason` | Retryable | What to do |
| --- | --- | --- | --- | --- |
| 400 | `invalid_request` | `empty_target` | `false` | Provide a target URL. |
| 400 | `invalid_request` | `bad_scheme` | `false` | Use `http` or `https`. |
| 400 | `invalid_request` | `malformed_url` | `false` | Fix the URL. |
| 400 | `invalid_request` | `target_too_long` | `false` | Shorten the URL. |
| 400 | `invalid_request` | `target_not_allowed` | `false` | Use a public target. |
| 400 | `invalid_request` | `target_rejected` | `false` | The fetcher rejected the URL; check that it is a valid public page. |
| 400 | `invalid_request` | `bad_option` | `false` | Fix the option request headers. |
| 404 | `not_found` | `route_disabled` | `false` | The free Markdown route is not enabled on this host; use `/v1/webcontext/scrape`. |
| 404 | `not_found` | `target_not_found` | `false` | The page does not exist; check the URL. |
| 405 | `method_not_allowed` | `method_not_allowed` | `false` | Use `GET` or `HEAD`. |
| 429 | `anonymous_limit_reached` | `daily_limit` | `false` | Wait for `X-RateLimit-Reset` or use an API key. |
| 451 | `source_excluded` | `publisher_excluded` | `false` | The publisher excludes automated access; contact them for licensing. |
| 502 | `upstream_blocked` | `access_check` | `false` | Try another page or contact the publisher. |
| 502 | `upstream_unavailable` | `target_unreachable` | `true` | Check the URL or try again later. |
| 502 | `upstream_unavailable` | `empty_content` | `false` | Try a text-based page; scanned PDFs and image-only pages have no text. |
| 502 | `upstream_unavailable` | `content_too_large` | `false` | Try a smaller page. |
| 502 | `upstream_unavailable` | `provider_unavailable` | `true` | Try again later. |
| 503 | `degraded` | `service_degraded` | `true` | Try again later. |
| 499 | `gateway_timeout` | `client_closed` | `true` | Your client closed the connection; keep it open for up to 33 seconds. |
| 500 | `internal_error` | `internal_error` | `true` | Try again later; report the `request_id` if it persists. |
| 504 | `gateway_timeout` | `target_timeout` | `true` | Try again later. |

```json
{ "error": { "code": "upstream_unavailable", "reason": "empty_content", "retryable": false,
             "message": "The page was fetched but has no extractable text, …", "request_id": "…" } }
```

Reason values are a closed set; new values may be added, but existing ones are never renamed or removed.

## Cross-discovery and returning-caller CTA

Successful free responses include `X-ReplyNodes-Discovery: brand` and a
`Link` relation to `https://brand.replynodes.com/{domain}` when the target
host is known. A valid returning `rn_anon_caller_id` cookie also exposes
`X-ReplyNodes-CTA: api-key` and a `Link` relation to
`https://platform.replynodes.com/sign-up`. These headers are informational:
the cookie is analytics-only, and the first free request remains keyless.