We will wire a minimal fetch-to-Markdown step with AnyCrawler’s free endpoint (no API key), then drop the Markdown into a tiny RAG-style ingest function. I work on AnyCrawler; that is disclosed again at the end.

The problem: RAG wants documents, scrapers want farms

Classic web scraping and RAG ingest pull in opposite directions:

• Scrapers optimize for tables and selectors. RAG usually wants readable prose, headings, and stable chunk boundaries.
• Homegrown fetch + BeautifulSoup / Readability works until the next docs theme, SPA shell, or cookie wall breaks your cleaner.
• Playwright / Puppeteer farms are correct for JS-heavy apps, but they are expensive and brittle as the default path for “index this public docs URL.”

For many agent and RAG demos, the job is narrower: one public URL → title + clean Markdown → chunks with source URL metadata. That is a fetch layer, not a scraper farm.

The minimal solution: free URL → Markdown GET

AnyCrawler’s free crawl endpoint:

GET https://api.anycrawler.com/free/v1/crawl?url={url}

What you get:

• No API key on this free path
• One public page per request
• JSON with results.markdown (plus title / description / status_code when available)
• Successful responses may be cached up to 24 hours

Try the same flow in the browser first: free URL→Markdown playground.

Step 1 — Prove the page becomes Markdown

curl -sS "https://api.anycrawler.com/free/v1/crawl?url=https://example.com" \
  | python3 -m json.tool

Example shape:

{
  "ok": true,
  "requested_url": "https://example.com/",
  "results": {
    "description": null,
    "markdown": "This domain is for use in documentation examples...",
    "title": "Example Domain"
  },
  "status_code": 200,
  "total_ms": 519,
  "timestamp": "2026-11-05T10:38:26.962Z"
}

For ingest, keep results.markdown as the document body and requested_url as citation metadata.

Step 2 — Python ingest helper (fetch → Markdown → chunks)

import requests
from typing import Any

FREE_ENDPOINT = "https://api.anycrawler.com/free/v1/crawl"


def fetch_markdown(page_url: str, timeout: int = 60) -> dict[str, Any]:
    resp = requests.get(FREE_ENDPOINT, params={"url": page_url}, timeout=timeout)
    resp.raise_for_status()
    data = resp.json()
    if not data.get("ok"):
        raise RuntimeError(f"crawl failed: {data}")

    results = data.get("results") or {}
    return {
        "source_url": data.get("requested_url") or page_url,
        "title": results.get("title"),
        "description": results.get("description"),
        "markdown": results.get("markdown") or "",
        "status_code": data.get("status_code"),
    }


def naive_chunks(markdown: str, max_chars: int = 800) -> list[str]:
    """Split on blank lines, then pack paragraphs up to max_chars."""
    paras = [p.strip() for p in markdown.split("\n\n") if p.strip()]
    chunks: list[str] = []
    buf = ""
    for p in paras:
        candidate = f"{buf}\n\n{p}".strip() if buf else p
        if len(candidate) <= max_chars:
            buf = candidate
        else:
            if buf:
                chunks.append(buf)
            buf = p if len(p) <= max_chars else p[:max_chars]
    if buf:
        chunks.append(buf)
    return chunks


def ingest_public_url(page_url: str) -> list[dict[str, Any]]:
    doc = fetch_markdown(page_url)
    out = []
    for i, text in enumerate(naive_chunks(doc["markdown"])):
        out.append(
            {
                "id": f"{doc['source_url']}#chunk-{i}",
                "text": text,
                "metadata": {
                    "source_url": doc["source_url"],
                    "title": doc["title"],
                    "chunk_index": i,
                },
            }
        )
    return out


# Example: replace with your vector store upsert
docs = ingest_public_url("https://example.com")
print(len(docs), "chunks")
print(docs[0]["metadata"] if docs else None)
print((docs[0]["text"][:200] + "...") if docs else None)

Swap naive_chunks for your real splitter (tiktoken, LlamaIndex, LangChain) and upsert into your store. The important part is that the source is already Markdown, so you skip a second HTML-cleaning stage for many public pages.

Step 3 — JavaScript version for Node agents

const FREE_ENDPOINT = "https://api.anycrawler.com/free/v1/crawl";

async function fetchMarkdown(pageUrl) {
  const endpoint = new URL(FREE_ENDPOINT);
  endpoint.searchParams.set("url", pageUrl);

  const res = await fetch(endpoint, { method: "GET" });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);

  const data = await res.json();
  if (!data.ok) throw new Error(`crawl failed: ${JSON.stringify(data)}`);

  return {
    sourceUrl: data.requested_url || pageUrl,
    title: data.results?.title ?? null,
    description: data.results?.description ?? null,
    markdown: data.results?.markdown ?? "",
    statusCode: data.status_code,
  };
}

function naiveChunks(markdown, maxChars = 800) {
  const paras = markdown.split(/\n\n+/).map((p) => p.trim()).filter(Boolean);
  const chunks = [];
  let buf = "";
  for (const p of paras) {
    const candidate = buf ? `${buf}\n\n${p}` : p;
    if (candidate.length <= maxChars) {
      buf = candidate;
    } else {
      if (buf) chunks.push(buf);
      buf = p.length <= maxChars ? p : p.slice(0, maxChars);
    }
  }
  if (buf) chunks.push(buf);
  return chunks;
}

async function ingestPublicUrl(pageUrl) {
  const doc = await fetchMarkdown(pageUrl);
  return naiveChunks(doc.markdown).map((text, i) => ({
    id: `${doc.sourceUrl}#chunk-${i}`,
    text,
    metadata: {
      sourceUrl: doc.sourceUrl,
      title: doc.title,
      chunkIndex: i,
    },
  }));
}

const docs = await ingestPublicUrl("https://example.com");
console.log(docs.length, "chunks");
console.log(docs[0]?.metadata);

In an agent tool loop, expose fetchMarkdown (or ingestPublicUrl) as one tool: “Fetch a public page and return clean Markdown for RAG.” Keep the tool description honest—this is read one public URL, not full-site crawl or logged-in browsing.

Step 4 — Wire it into a RAG query path

A practical pattern:

1. Planner or user supplies a public docs/article URL (or a search step picks one).
2. Ingest helper calls the free crawl endpoint.
3. You chunk results.markdown, embed, and store source_url / title as metadata.
4. At answer time, retrieve chunks and cite the source URL.

Cache awareness: free successful results may be reused for up to 24 hours. That is fine for prototypes. If your index must recrawl sooner, or you need explicit fetch vs browser-render modes, move to AnyCrawler’s authenticated production crawl API.

When a scraper farm is still the wrong default

Stay on a fetch-to-Markdown layer when you:

• Index public docs, blog posts, or changelogs for RAG
• Prototype agent tools that “read this URL”
• Want Markdown chunk boundaries instead of HTML DOM noise

Reach for browsers / site-specific scrapers when you:

• Need authenticated sessions or multi-step UI flows
• Extract structured tables with brittle selectors as the product
• Must interact (click, paginate, fill forms) rather than read

Most early RAG and agent stacks over-buy the second category. Start with Markdown fetch; graduate only when a page proves it needs a render mode or a real scraper.

Short FAQ (related searches)

How do I build RAG ingest without maintaining scrapers?

Call a crawl API that returns cleaned Markdown in JSON, then chunk and embed that Markdown. The free AnyCrawler endpoint above does one public URL per request with no API key—enough for many demos and internal tools.

Is fetch-to-Markdown enough for JS-heavy sites?

Often for content-oriented public pages; not always for app shells. If Markdown is empty or useless, try the production API’s render mode, or treat that URL as a scraper exception—not a reason to scrap the whole pipeline.

Are free URL→Markdown results cached?

Yes. Successful free results may be served from cache for up to 24 hours. Plan refresh policy accordingly.

What URLs work for RAG ingest?

Complete, publicly reachable http/https URLs. Login walls, private networks, and unreachable pages are not suitable for the free tool.

Try the free fetch layer, then grow

1. Paste a public docs URL into the AnyCrawler free URL→Markdown tool.
2. Drop the Python or JS ingest helper into your RAG prototype.
3. Move to the authenticated crawl API when you need fetch/render modes, usage records, and production limits.

Disclosure: I work on the AnyCrawler team. Questions: marketing@anycrawler.com.