MCP Marketplace
Sign inSign up
Back to Browse

Crawler MCP Server

by Mark1russell7
Developer ToolsLow Risk10.0MCP RegistryLocal
Free

Server data from the Official MCP Registry

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

About

A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.

Security Report

10.0
Low Risk10.0Low Risk

Valid MCP server (1 strong, 1 medium validity signals). No known CVEs in dependencies. ⚠️ Package registry links to a different repository than scanned source. Imported from the Official MCP Registry. Trust signals: trusted author (3/3 approved); 4 highly-trusted packages. 1 finding(s) downgraded by scanner intelligence.

9 files analyzed · 1 issue found

Security scores are indicators to help you make informed decisions, not guarantees. Always review permissions before connecting any MCP server.

Permissions Required

This plugin requests these system permissions. Most are normal for its category.

file_system

Check that this permission is expected for this type of plugin.

env_vars

Check that this permission is expected for this type of plugin.

database

Check that this permission is expected for this type of plugin.

HTTP Network Access

Connects to external APIs or services over the internet.

What You'll Need

Set these up before or after installing:

The folder of the store. The default is ~/.polite-crawler.Optional

Environment variable: POLITE_CRAWLER_DIR

An email address or URL for the From header of each request.Optional

Environment variable: POLITE_CRAWLER_CONTACT

Set it to 1 to let the crawler fetch loopback, private and link-local addresses. The default refuses them.Optional

Environment variable: POLITE_CRAWLER_ALLOW_PRIVATE

Set it to 1 to let user_provided read a page that refuses AI input when the client cannot ask the user to confirm.Optional

Environment variable: POLITE_CRAWLER_ALLOW_USER_PROVIDED

The file of an Ed25519 private key (PKCS#8 PEM or JWK). The crawler signs each request with it (Web Bot Auth). It needs POLITE_CRAWLER_SIGNATURE_AGENT.Optional

Environment variable: POLITE_CRAWLER_SIGN_KEY

Where a site finds the public key of POLITE_CRAWLER_SIGN_KEY: the https origin of the key directory, or the URL of a JSON Web Key Set.Optional

Environment variable: POLITE_CRAWLER_SIGNATURE_AGENT

How to Install

Add this to your MCP configuration file:

{
  "mcpServers": {
    "io-github-mark1russell7-polite-crawler": {
      "env": {
        "POLITE_CRAWLER_DIR": "your-polite-crawler-dir-here",
        "POLITE_CRAWLER_CONTACT": "your-polite-crawler-contact-here",
        "POLITE_CRAWLER_SIGN_KEY": "your-polite-crawler-sign-key-here",
        "POLITE_CRAWLER_ALLOW_PRIVATE": "your-polite-crawler-allow-private-here",
        "POLITE_CRAWLER_SIGNATURE_AGENT": "your-polite-crawler-signature-agent-here",
        "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "your-polite-crawler-allow-user-provided-here"
      },
      "args": [
        "-y",
        "polite-crawler-mcp"
      ],
      "command": "npx"
    }
  }
}

Documentation

View on GitHub

From the project's GitHub README.

crawler

npm CI Pages license

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp

What the crawler does for you and for the sites:

  • It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
  • By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
  • The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
  • The library is polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

License

MIT

Reviews

No reviews yet

Be the first to review this server!

More Developer Tools MCP Servers

Rea

Free

by Morluto · Developer Tools

Reverse engineer anything from your terminal or agent with one CLI and MCP server.

418.4K
Stars
8
Installs
10.0
Security
No ratings yet
Local

Paperclip

Free

by Paperclipai · Developer Tools

Trending hip-hop artist momentum scores across four cultural dimensions.

103.4K
Stars
5
Installs
10.0
Security
No ratings yet
Remote

Toleno

Free

by Toleno · Developer Tools

Toleno Network MCP Server — Manage your Toleno mining account with Claude AI using natural language.

258
Stars
898
Installs
8.0
Security
4.8
Local

MCP Marketplace

Free

by mcp-marketplace · Developer Tools

Search and install MCP servers from inside your AI client.

-
Stars
189
Installs
10.0
Security
5.0
Remote

mcp-creator-python

Free

by mcp-marketplace · Developer Tools

Create, build, and publish Python MCP servers to PyPI — conversationally.

-
Stars
151
Installs
10.0
Security
4.7
Local

WhatsApp

Free

by Lharries · Communication

Read, search, and send WhatsApp messages through your AI assistant

6.8K
Stars
68
Installs
9.0
Security
No ratings yet
Local