Web Crawler MCP Server

The Web Crawler MCP server is InsightfulPipe's own tool, running on InsightfulPipe's crawling service. It is not affiliated with Google, Microsoft or any AI company whose crawlers it checks.

Any AI client that supports remote MCP servers. The setup guide on this page covers Claude, ChatGPT, Claude Code, Cursor, Gemini CLI and Codex. You can also run the same checks from the terminal with the InsightfulPipe CLI.

Yes. In Claude, add https://crawler.insightfulmcp.com/ as a custom connector and authorize your InsightfulPipe account. You can also add InsightfulPipe from the Claude Directory, which connects all of your linked accounts and tools at once.

Yes. Add the server URL as a custom connector in ChatGPT Developer mode, or add InsightfulPipe from the ChatGPT app directory to connect all of your linked accounts and tools. Developer mode availability depends on your ChatGPT plan.

Thirty page checks, including SEO audits, meta tags, headings, canonicals, indexability, status codes, hreflang, structured data, social tags, internal and broken links, images, content quality, URL structure, mobile setup, page speed and weight, SSL, security headers, AMP, favicons, encoding and deprecated HTML. It also checks robots.txt, AI crawler access and llms.txt, generates sitemaps and robots.txt files, and extracts links and brand design details.

Each check analyzes one URL, but your assistant can run checks across several pages and compare the results. The sitemap generator crawls a whole domain to list its URLs, which you can then audit page by page.

It does not enforce robots.txt rules on the pages you ask it to check, so only analyze sites you own or have permission to audit. To see what a site allows, run the robots.txt checker and the AI crawler checker.

The server is read-only: it fetches public pages and returns findings, and it cannot change any website. Owners, admins and operators can use it by default, and restricted team members can use it only if they are granted access. Requests to private network addresses are blocked.

Yes. There is no site to connect, so you can check any public URL, including competitor pages, and compare results across sites in one conversation.

There is no third-party account or API key. Your AI client signs in to the MCP server with your InsightfulPipe account and receives an InsightfulPipe access token, and checks run on InsightfulPipe's service.

Checks work on public pages one URL at a time, so pages behind a login cannot be analyzed. Each request counts toward a monthly allowance that depends on your plan, and the server runs checks on request rather than monitoring sites on a schedule. For search traffic and index data, use the Google Search Console or Bing Webmaster Tools MCP servers.

The Web Crawler MCP server is included in every InsightfulPipe plan, together with all other MCP servers and the CLI. Plans start at $29.99/month with a 7-day free trial. See insightfulpipe.com/pricing for current plans.