#### #### ##### #### #### ####### #### ### #### ######### #### ###### #### ###### #### ###################### ##### #### ####### ####### ##### #### ####### ####### # #### ###################### #### #### ###### #### #### #### ## #### #### #### #### ################## #### #### ################## ================================================== # SiteOne Crawler, v2.1.0.20260317 # # Author: jan.reges@siteone.cz # ================================================== Detected terminal width 138 < 140 chars - compact mode activated. URL | Status | Type | Time | Size | Cache | Access. | Best pr. ------------------------------------------------------------------------------------------------------------------------------------ / | 308 | Redirect | 264 ms | 91 B | 0 s | | /introduction | 200 | HTML | 144 ms | 1 MB | 0 s | 2/2 | 1/6 /advanced-scraping-guide | 200 | HTML | 96 ms | 1 MB | 0 s | 2/2 | 2/5 /use-cases/lead-enrichment | 200 | HTML | 90 ms | 743 kB | 0 s | 2/2 | 1/6 /features/interact | 200 | HTML | 174 ms | 1 MB | 0 s | 2/2 | 1/6 /features/map | 200 | HTML | 98 ms | 966 kB | 0 s | 2/2 | 1/6 /features/scrape | 200 | HTML | 147 ms | 2 MB | 0 s | 2/2 | 1/6 /use-cases/overview | 200 | HTML | 146 ms | 710 kB | 0 s | 2/2 | 1/6 /sdks/cli | 200 | HTML | 90 ms | 1 MB | 0 s | 2/2 | 1/6 /webhooks/security | 200 | HTML | 111 ms | 929 kB | 0 s | 2/2 | 1/6 /sdks/overview | 200 | HTML | 85 ms | 682 kB | 0 s | 2/2 | 1/6 /developer-guides/openclaw | 200 | HTML | 103 ms | 799 kB | 0 s | 2/2 | 1/6 /developer-guides/examples | 200 | HTML | 116 ms | 728 kB | 0 s | 2/2 | 1/6 /contributing/guide | 200 | HTML | 110 ms | 829 kB | 0 s | 2/2 | 1/6 /use-cases/seo-platforms | 200 | HTML | 91 ms | 732 kB | 0 s | 2/2 | 1/6 /dashboard | 200 | HTML | 89 ms | 742 kB | 0 s | 2/2 | 1/6 /mcp-server | 200 | HTML | 101 ms | 1 MB | 0 s | 2/2 | 1/6 /api-reference/v2-introduction | 200 | HTML | 85 ms | 775 kB | 0 s | 2/2 | 1/6 /contributing/open-source-or-cloud | 200 | HTML | 153 ms | 724 kB | 0 s | 2/2 | 1/6 /features/search | 200 | HTML | 111 ms | 1 MB | 0 s | 2/2 | 1/6 /use-cases/ai-platforms | 200 | HTML | 156 ms | 730 kB | 0 s | 2/2 | 1/6 /use-cases/deep-research | 200 | HTML | 72 ms | 822 kB | 0 s | 2/2 | 1/6 /contributing/self-host | 200 | HTML | 162 ms | 843 kB | 0 s | 2/2 | 1/6 /webhooks/events | 200 | HTML | 91 ms | 1 MB | 0 s | 2/2 | 1/6 /webhooks/testing | 200 | HTML | 84 ms | 765 kB | 0 s | 2/2 | 1/6 /webhooks/overview | 200 | HTML | 117 ms | 779 kB | 0 s | 2/2 | 1/6 /features/crawl | 200 | HTML | 107 ms | 1 MB | 0 s | 2/2 | 1/6 /api-reference/endpoint/map | 200 | HTML | 100 ms | 775 kB | 0 s | 2/2 | 1/6 /billing | 200 | HTML | 91 ms | 792 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/scrape | 200 | HTML | 119 ms | 954 kB | 0 s | 2/2 | 1/6 /use-cases/investment-finance | 200 | HTML | 149 ms | 735 kB | 0 s | 2/2 | 1/6 /use-cases/competitive-intelligence | 200 | HTML | 141 ms | 745 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/scrape-browser-delete | 200 | HTML | 167 ms | 773 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/scrape-execute | 200 | HTML | 97 ms | 917 kB | 0 s | 2/2 | 1/6 /features/document-parsing | 200 | HTML | 85 ms | 791 kB | 0 s | 2/2 | 1/6 /features/browser | 200 | HTML | 122 ms | 2 MB | 0 s | 2/2 | 1/6 /features/proxies | 200 | HTML | 112 ms | 802 kB | 0 s | 2/2 | 1/6 /features/change-tracking | 200 | HTML | 91 ms | 2 MB | 0 s | 2/2 | 1/6 /features/enhanced-mode | 200 | HTML | 77 ms | 772 kB | 0 s | 2/2 | 1/6 /features/batch-scrape | 200 | HTML | 93 ms | 1 MB | 0 s | 2/2 | 1/6 /features/llm-extract | 200 | HTML | 85 ms | 1 MB | 0 s | 2/2 | 1/6 /features/fast-scraping | 200 | HTML | 93 ms | 871 kB | 0 s | 2/2 | 1/6 /sdks/python | 200 | HTML | 119 ms | 1 MB | 0 s | 2/2 | 1/6 /sdks/rust | 200 | HTML | 302 ms | 1 MB | 0 s | 2/2 | 1/6 /sdks/java | 200 | HTML | 173 ms | 2 MB | 0 s | 2/2 | 1/6 /sdks/node | 200 | HTML | 174 ms | 1 MB | 0 s | 2/2 | 1/6 /features/models | 200 | HTML | 139 ms | 829 kB | 0 s | 2/2 | 1/6 /features/agent | 200 | HTML | 143 ms | 1 MB | 0 s | 2/2 | 1/6 /sdks/go | 200 | HTML | 342 ms | 903 kB | 0 s | 2/2 | 1/6 /developer-guides/usage-guides/choosing-the-data-extractor | 200 | HTML | 110 ms | 1 MB | 0 s | 2/2 | 1/6 /use-cases/content-generation | 200 | HTML | 125 ms | 741 kB | 0 s | 2/2 | 1/6 /use-cases/observability | 200 | HTML | 77 ms | 740 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/agent-get | 200 | HTML | 101 ms | 746 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/extract | 200 | HTML | 81 ms | 805 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-delete | 200 | HTML | 103 ms | 734 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/browser-execute | 200 | HTML | 98 ms | 821 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/agent | 200 | HTML | 115 ms | 755 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/token-usage-historical | 200 | HTML | 71 ms | 739 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-params-preview | 200 | HTML | 117 ms | 755 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/browser-list | 200 | HTML | 98 ms | 805 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/batch-scrape | 200 | HTML | 79 ms | 880 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/batch-scrape-get | 200 | HTML | 83 ms | 775 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/extract-get | 200 | HTML | 123 ms | 740 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/queue-status | 200 | HTML | 107 ms | 742 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/token-usage | 200 | HTML | 144 ms | 742 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-get-errors | 200 | HTML | 103 ms | 745 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/agent-delete | 200 | HTML | 111 ms | 727 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/batch-scrape-get-errors | 200 | HTML | 105 ms | 746 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-get | 200 | HTML | 94 ms | 774 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-post | 200 | HTML | 394 ms | 839 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/crawl-active | 200 | HTML | 104 ms | 785 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/search | 200 | HTML | 83 ms | 947 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/credit-usage-historical | 200 | HTML | 213 ms | 741 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/browser-create | 200 | HTML | 145 ms | 821 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/browser-delete | 200 | HTML | 95 ms | 774 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/credit-usage | 200 | HTML | 132 ms | 739 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/batch-scrape-delete | 200 | HTML | 88 ms | 738 kB | 0 s | 2/2 | 1/6 /rate-limits | 200 | HTML | 115 ms | 765 kB | 0 s | 2/2 | 1/6 /use-cases/developers-mcp | 200 | HTML | 94 ms | 751 kB | 0 s | 2/2 | 1/6 /use-cases/product-ecommerce | 200 | HTML | 94 ms | 737 kB | 0 s | 2/2 | 1/6 /developer-guides/workflow-automation/dify | 200 | HTML | 141 ms | 789 kB | 0 s | 2/2 | 1/6 /use-cases/data-migration | 200 | HTML | 152 ms | 737 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/vercel-ai-sdk | 200 | HTML | 82 ms | 1 MB | 0 s | 2/2 | 1/6 /pricing | 404 | HTML | 269 ms | 496 kB | 0 s | | 1/2 /developer-guides/workflow-automation/n8n | 200 | HTML | 92 ms | 1 MB | 0 s | 2/2 | 1/6 /features/extract | 200 | HTML | 128 ms | 1 MB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/openai | 200 | HTML | 85 ms | 1 MB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-crawl-started | 200 | HTML | 89 ms | 731 kB | 0 s | 2/2 | 1/6 /developer-guides/workflow-automation/zapier | 200 | HTML | 117 ms | 796 kB | 0 s | 2/2 | 1/6 /developer-guides/workflow-automation/make | 200 | HTML | 76 ms | 813 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/langgraph | 200 | HTML | 93 ms | 992 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/langchain | 200 | HTML | 96 ms | 1 MB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/mastra | 200 | HTML | 76 ms | 1023 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/anthropic | 200 | HTML | 84 ms | 996 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/llamaindex | 200 | HTML | 100 ms | 813 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/google-adk | 200 | HTML | 83 ms | 873 kB | 0 s | 2/2 | 1/6 /developer-guides/llm-sdks-and-frameworks/gemini | 200 | HTML | 103 ms | 933 kB | 0 s | 2/2 | 1/6 /developer-guides/common-sites/wikipedia | 200 | HTML | 79 ms | 914 kB | 0 s | 2/2 | 1/6 /v1/api-reference/introduction | 200 | HTML | 80 ms | 721 kB | 0 s | 2/2 | 1/6 /agents/fire-1 | 200 | HTML | 80 ms | 826 kB | 0 s | 2/2 | 1/6 /migrate-to-v2 | 200 | HTML | 119 ms | 920 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-crawl-page | 200 | HTML | 111 ms | 742 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-crawl-completed | 200 | HTML | 89 ms | 731 kB | 0 s | 2/2 | 1/6 /developer-guides/cookbooks/ai-research-assistant-cookbook | 200 | HTML | 126 ms | 2 MB | 0 s | 2/2 | 1/6 /developer-guides/common-sites/github | 200 | HTML | 163 ms | 974 kB | 0 s | 2/2 | 1/6 /developer-guides/common-sites/etsy | 200 | HTML | 116 ms | 936 kB | 0 s | 2/2 | 1/6 /developer-guides/common-sites/amazon | 200 | HTML | 144 ms | 941 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/crawl-delete | 200 | HTML | 75 ms | 719 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/crawl-post | 200 | HTML | 105 ms | 823 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/extract-get | 200 | HTML | 76 ms | 723 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/batch-scrape-get-errors | 200 | HTML | 99 ms | 733 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/crawl-active | 200 | HTML | 79 ms | 766 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/batch-scrape-get | 200 | HTML | 102 ms | 762 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/extract | 200 | HTML | 103 ms | 785 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/token-usage | 200 | HTML | 166 ms | 726 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/credit-usage | 200 | HTML | 126 ms | 726 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/queue-status | 200 | HTML | 75 ms | 728 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/batch-scrape | 200 | HTML | 76 ms | 836 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/crawl-get-errors | 200 | HTML | 77 ms | 732 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/map | 200 | HTML | 75 ms | 754 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/search | 200 | HTML | 89 ms | 804 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/credit-usage-historical | 200 | HTML | 99 ms | 727 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/batch-scrape-delete | 200 | HTML | 102 ms | 725 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/crawl-get | 200 | HTML | 108 ms | 761 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/scrape | 200 | HTML | 267 ms | 861 kB | 0 s | 2/2 | 1/6 /api-reference/v1-endpoint/token-usage-historical | 200 | HTML | 165 ms | 727 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-batch-scrape-started | 200 | HTML | 119 ms | 731 kB | 0 s | 2/2 | 1/6 /developer-guides/cookbooks/brand-style-gu…-generator-cookbook | 200 | HTML | 161 ms | 1 MB | 0 s | 2/2 | 1/6 /developer-guides/mcp-setup-guides/factory-ai | 200 | HTML | 181 ms | 750 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-batch-scrape-page | 200 | HTML | 83 ms | 735 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-batch-scrape-completed | 200 | HTML | 124 ms | 731 kB | 0 s | 2/2 | 1/6 /developer-guides/mcp-setup-guides/chatgpt | 200 | HTML | 105 ms | 779 kB | 0 s | 2/2 | 1/6 /developer-guides/mcp-setup-guides/cursor | 200 | HTML | 81 ms | 778 kB | 0 s | 2/2 | 1/6 /developer-guides/mcp-setup-guides/windsurf | 200 | HTML | 77 ms | 741 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-agent-started | 200 | HTML | 75 ms | 732 kB | 0 s | 2/2 | 1/6 /developer-guides/mcp-setup-guides/claude-code | 200 | HTML | 200 ms | 738 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-agent-cancelled | 200 | HTML | 87 ms | 734 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-agent-failed | 200 | HTML | 90 ms | 738 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-agent-completed | 200 | HTML | 83 ms | 737 kB | 0 s | 2/2 | 1/6 /api-reference/endpoint/webhook-agent-action | 200 | HTML | 96 ms | 737 kB | 0 s | 2/2 | 1/6 Skipped URLs Summary -------------------- Reason | Domain | Unique URLs ----------------------------------------------------------- Not allowed host | github.com | 164 Not allowed host | www.firecrawl.dev | 20 Not allowed host | zapier.com | 14 Not allowed host | firecrawl.dev | 6 Not allowed host | ai-sdk.dev | 6 Not allowed host | insiders.vscode.dev | 2 Not allowed host | localhost | 2 Not allowed host | platform.openai.com | 2 Not allowed host | docs.n8n.io | 2 Not allowed host | docs.anthropic.com | 1 Not allowed host | smithery.ai | 1 Not allowed host | json-schema.org | 1 Not allowed host | nodejs.org | 1 Not allowed host | www.make.com | 1 Not allowed host | playwright.dev | 1 Not allowed host | mcp.firecrawl.dev | 1 Not allowed host | langchain-ai.github.io | 1 Not allowed host | chatgpt.com | 1 Not allowed host | mastra.ai | 1 Not allowed host | www.docker.com | 1 Not allowed host | redis.io | 1 Not allowed host | ai.google.dev | 1 Not allowed host | vercel.com | 1 Not allowed host | docs.cursor.com | 1 Not allowed host | js.langchain.com | 1 Not allowed host | x.com | 1 Not allowed host | help.openai.com | 1 Not allowed host | discord.gg | 1 Not allowed host | ts.llamaindex.ai | 1 Not allowed host | pnpm.io | 1 Not allowed host | docs.docker.com | 1 Not allowed host | marketplace.dify.ai | 1 Not allowed host | community.n8n.io | 1 Not allowed host | www.linkedin.com | 1 Not allowed host | docs.factory.ai | 1 Skipped URLs ------------ Reason | Skipped URL | Source | Found at URL ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Not allowed host | http://localhost:3000/ | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | http://localhost:3000/v2/mcp | | /mcp-server Not allowed host | https://ai-sdk.dev/docs | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://ai-sdk.dev/docs/foundations/providers-and-models | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://ai-sdk.dev/docs/foundations/tools | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://ai-sdk.dev/elements/overview | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://ai-sdk.dev/providers/ai-sdk-providers | | /developer-guides/llm-sdks-and-frameworks/vercel-ai-sdk Not allowed host | https://ai.google.dev/docs | | /developer-guides/llm-sdks-and-frameworks/gemini Not allowed host | https://chatgpt.com/ | | /developer-guides/mcp-setup-guides/chatgpt Not allowed host | https://community.n8n.io/ | | /developer-guides/workflow-automation/n8n Not allowed host | https://discord.gg/firecrawl | | /introduction Not allowed host | https://docs.anthropic.com/claude/docs | | /developer-guides/llm-sdks-and-frameworks/anthropic Not allowed host | https://docs.cursor.com/context/model-context-protocol | | /mcp-server Not allowed host | https://docs.docker.com/get-docker/ | | /contributing/self-host Not allowed host | https://docs.factory.ai/cli/getting-started/quickstart | | /developer-guides/mcp-setup-guides/factory-ai Not allowed host | https://docs.n8n.io/ | | /developer-guides/workflow-automation/n8n Not allowed host | https://docs.n8n.io/hosting/ | | /developer-guides/workflow-automation/n8n Not allowed host | https://firecrawl.dev/app | | /developer-guides/mcp-setup-guides/factory-ai Not allowed host | https://firecrawl.dev/blog | | /introduction Not allowed host | https://firecrawl.dev/changelog | | /introduction Not allowed host | https://firecrawl.dev/enterprise | | /rate-limits Not allowed host | https://firecrawl.dev/playground | | /introduction Not allowed host | https://firecrawl.dev/pricing | | /developer-guides/usage-guides/choosing-the-data-extractor Not allowed host | https://github.com/cloudflare/cloudflared/releases | | /webhooks/testing Not allowed host | https://github.com/directus/directus/issues/25906 | | /mcp-server Not allowed host | https://github.com/firecrawl/ai-ready-website | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/cli | | /sdks/cli Not allowed host | https://github.com/firecrawl/fire-enrich | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/firecrawl | | /introduction Not allowed host | https://github.com/firecrawl/firecrawl-d…nced-scraping-guide | | /advanced-scraping-guide Not allowed host | https://github.com/firecrawl/firecrawl-d…h:%20/agents/fire-1 | | /agents/fire-1 Not allowed host | https://github.com/firecrawl/firecrawl-d…ence/endpoint/agent | | /api-reference/endpoint/agent Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/agent-delete | | /api-reference/endpoint/agent-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…/endpoint/agent-get | | /api-reference/endpoint/agent-get Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/batch-scrape | | /api-reference/endpoint/batch-scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…batch-scrape-delete | | /api-reference/endpoint/batch-scrape-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…nt/batch-scrape-get | | /api-reference/endpoint/batch-scrape-get Not allowed host | https://github.com/firecrawl/firecrawl-d…h-scrape-get-errors | | /api-reference/endpoint/batch-scrape-get-errors Not allowed host | https://github.com/firecrawl/firecrawl-d…oint/browser-create | | /api-reference/endpoint/browser-create Not allowed host | https://github.com/firecrawl/firecrawl-d…oint/browser-delete | | /api-reference/endpoint/browser-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…int/browser-execute | | /api-reference/endpoint/browser-execute Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/browser-list | | /api-reference/endpoint/browser-list Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/crawl-active | | /api-reference/endpoint/crawl-active Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/crawl-delete | | /api-reference/endpoint/crawl-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…/endpoint/crawl-get | | /api-reference/endpoint/crawl-get Not allowed host | https://github.com/firecrawl/firecrawl-d…nt/crawl-get-errors | | /api-reference/endpoint/crawl-get-errors Not allowed host | https://github.com/firecrawl/firecrawl-d…rawl-params-preview | | /api-reference/endpoint/crawl-params-preview Not allowed host | https://github.com/firecrawl/firecrawl-d…endpoint/crawl-post | | /api-reference/endpoint/crawl-post Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/credit-usage | | /api-reference/endpoint/credit-usage Not allowed host | https://github.com/firecrawl/firecrawl-d…it-usage-historical | | /api-reference/endpoint/credit-usage-historical Not allowed host | https://github.com/firecrawl/firecrawl-d…ce/endpoint/extract | | /api-reference/endpoint/extract Not allowed host | https://github.com/firecrawl/firecrawl-d…ndpoint/extract-get | | /api-reference/endpoint/extract-get Not allowed host | https://github.com/firecrawl/firecrawl-d…erence/endpoint/map | | /api-reference/endpoint/map Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/queue-status | | /api-reference/endpoint/queue-status Not allowed host | https://github.com/firecrawl/firecrawl-d…nce/endpoint/scrape | | /api-reference/endpoint/scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…rape-browser-delete | | /api-reference/endpoint/scrape-browser-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…oint/scrape-execute | | /api-reference/endpoint/scrape-execute Not allowed host | https://github.com/firecrawl/firecrawl-d…nce/endpoint/search | | /api-reference/endpoint/search Not allowed host | https://github.com/firecrawl/firecrawl-d…ndpoint/token-usage | | /api-reference/endpoint/token-usage Not allowed host | https://github.com/firecrawl/firecrawl-d…en-usage-historical | | /api-reference/endpoint/token-usage-historical Not allowed host | https://github.com/firecrawl/firecrawl-d…ebhook-agent-action | | /api-reference/endpoint/webhook-agent-action Not allowed host | https://github.com/firecrawl/firecrawl-d…ook-agent-cancelled | | /api-reference/endpoint/webhook-agent-cancelled Not allowed host | https://github.com/firecrawl/firecrawl-d…ook-agent-completed | | /api-reference/endpoint/webhook-agent-completed Not allowed host | https://github.com/firecrawl/firecrawl-d…ebhook-agent-failed | | /api-reference/endpoint/webhook-agent-failed Not allowed host | https://github.com/firecrawl/firecrawl-d…bhook-agent-started | | /api-reference/endpoint/webhook-agent-started Not allowed host | https://github.com/firecrawl/firecrawl-d…ch-scrape-completed | | /api-reference/endpoint/webhook-batch-scrape-completed Not allowed host | https://github.com/firecrawl/firecrawl-d…k-batch-scrape-page | | /api-reference/endpoint/webhook-batch-scrape-page Not allowed host | https://github.com/firecrawl/firecrawl-d…atch-scrape-started | | /api-reference/endpoint/webhook-batch-scrape-started Not allowed host | https://github.com/firecrawl/firecrawl-d…ook-crawl-completed | | /api-reference/endpoint/webhook-crawl-completed Not allowed host | https://github.com/firecrawl/firecrawl-d…/webhook-crawl-page | | /api-reference/endpoint/webhook-crawl-page Not allowed host | https://github.com/firecrawl/firecrawl-d…bhook-crawl-started | | /api-reference/endpoint/webhook-crawl-started Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/batch-scrape | | /api-reference/v1-endpoint/batch-scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…batch-scrape-delete | | /api-reference/v1-endpoint/batch-scrape-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…nt/batch-scrape-get | | /api-reference/v1-endpoint/batch-scrape-get Not allowed host | https://github.com/firecrawl/firecrawl-d…h-scrape-get-errors | | /api-reference/v1-endpoint/batch-scrape-get-errors Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/crawl-active | | /api-reference/v1-endpoint/crawl-active Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/crawl-delete | | /api-reference/v1-endpoint/crawl-delete Not allowed host | https://github.com/firecrawl/firecrawl-d…-endpoint/crawl-get | | /api-reference/v1-endpoint/crawl-get Not allowed host | https://github.com/firecrawl/firecrawl-d…nt/crawl-get-errors | | /api-reference/v1-endpoint/crawl-get-errors Not allowed host | https://github.com/firecrawl/firecrawl-d…endpoint/crawl-post | | /api-reference/v1-endpoint/crawl-post Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/credit-usage | | /api-reference/v1-endpoint/credit-usage Not allowed host | https://github.com/firecrawl/firecrawl-d…it-usage-historical | | /api-reference/v1-endpoint/credit-usage-historical Not allowed host | https://github.com/firecrawl/firecrawl-d…v1-endpoint/extract | | /api-reference/v1-endpoint/extract Not allowed host | https://github.com/firecrawl/firecrawl-d…ndpoint/extract-get | | /api-reference/v1-endpoint/extract-get Not allowed host | https://github.com/firecrawl/firecrawl-d…nce/v1-endpoint/map | | /api-reference/v1-endpoint/map Not allowed host | https://github.com/firecrawl/firecrawl-d…dpoint/queue-status | | /api-reference/v1-endpoint/queue-status Not allowed host | https://github.com/firecrawl/firecrawl-d…/v1-endpoint/scrape | | /api-reference/v1-endpoint/scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…/v1-endpoint/search | | /api-reference/v1-endpoint/search Not allowed host | https://github.com/firecrawl/firecrawl-d…ndpoint/token-usage | | /api-reference/v1-endpoint/token-usage Not allowed host | https://github.com/firecrawl/firecrawl-d…en-usage-historical | | /api-reference/v1-endpoint/token-usage-historical Not allowed host | https://github.com/firecrawl/firecrawl-d…nce/v2-introduction | | /api-reference/v2-introduction Not allowed host | https://github.com/firecrawl/firecrawl-d…dy=Path:%20/billing | | /billing Not allowed host | https://github.com/firecrawl/firecrawl-d…/contributing/guide | | /contributing/guide Not allowed host | https://github.com/firecrawl/firecrawl-d…pen-source-or-cloud | | /contributing/open-source-or-cloud Not allowed host | https://github.com/firecrawl/firecrawl-d…tributing/self-host | | /contributing/self-host Not allowed host | https://github.com/firecrawl/firecrawl-d…=Path:%20/dashboard | | /dashboard Not allowed host | https://github.com/firecrawl/firecrawl-d…common-sites/amazon | | /developer-guides/common-sites/amazon Not allowed host | https://github.com/firecrawl/firecrawl-d…s/common-sites/etsy | | /developer-guides/common-sites/etsy Not allowed host | https://github.com/firecrawl/firecrawl-d…common-sites/github | | /developer-guides/common-sites/github Not allowed host | https://github.com/firecrawl/firecrawl-d…mon-sites/wikipedia | | /developer-guides/common-sites/wikipedia Not allowed host | https://github.com/firecrawl/firecrawl-d…-assistant-cookbook | | /developer-guides/cookbooks/ai-research-assistant-cookbook Not allowed host | https://github.com/firecrawl/firecrawl-d…-generator-cookbook | | /developer-guides/cookbooks/brand-style-…-generator-cookbook Not allowed host | https://github.com/firecrawl/firecrawl-d…per-guides/examples | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/firecrawl-d…rameworks/anthropic | | /developer-guides/llm-sdks-and-frameworks/anthropic Not allowed host | https://github.com/firecrawl/firecrawl-d…d-frameworks/gemini | | /developer-guides/llm-sdks-and-frameworks/gemini Not allowed host | https://github.com/firecrawl/firecrawl-d…ameworks/google-adk | | /developer-guides/llm-sdks-and-frameworks/google-adk Not allowed host | https://github.com/firecrawl/firecrawl-d…rameworks/langchain | | /developer-guides/llm-sdks-and-frameworks/langchain Not allowed host | https://github.com/firecrawl/firecrawl-d…rameworks/langgraph | | /developer-guides/llm-sdks-and-frameworks/langgraph Not allowed host | https://github.com/firecrawl/firecrawl-d…ameworks/llamaindex | | /developer-guides/llm-sdks-and-frameworks/llamaindex Not allowed host | https://github.com/firecrawl/firecrawl-d…d-frameworks/mastra | | /developer-guides/llm-sdks-and-frameworks/mastra Not allowed host | https://github.com/firecrawl/firecrawl-d…d-frameworks/openai | | /developer-guides/llm-sdks-and-frameworks/openai Not allowed host | https://github.com/firecrawl/firecrawl-d…works/vercel-ai-sdk | | /developer-guides/llm-sdks-and-frameworks/vercel-ai-sdk Not allowed host | https://github.com/firecrawl/firecrawl-d…etup-guides/chatgpt | | /developer-guides/mcp-setup-guides/chatgpt Not allowed host | https://github.com/firecrawl/firecrawl-d…-guides/claude-code | | /developer-guides/mcp-setup-guides/claude-code Not allowed host | https://github.com/firecrawl/firecrawl-d…setup-guides/cursor | | /developer-guides/mcp-setup-guides/cursor Not allowed host | https://github.com/firecrawl/firecrawl-d…p-guides/factory-ai | | /developer-guides/mcp-setup-guides/factory-ai Not allowed host | https://github.com/firecrawl/firecrawl-d…tup-guides/windsurf | | /developer-guides/mcp-setup-guides/windsurf Not allowed host | https://github.com/firecrawl/firecrawl-d…per-guides/openclaw | | /developer-guides/openclaw Not allowed host | https://github.com/firecrawl/firecrawl-d…-the-data-extractor | | /developer-guides/usage-guides/choosing-the-data-extractor Not allowed host | https://github.com/firecrawl/firecrawl-d…low-automation/dify | | /developer-guides/workflow-automation/dify Not allowed host | https://github.com/firecrawl/firecrawl-d…low-automation/make | | /developer-guides/workflow-automation/make Not allowed host | https://github.com/firecrawl/firecrawl-d…flow-automation/n8n | | /developer-guides/workflow-automation/n8n Not allowed host | https://github.com/firecrawl/firecrawl-d…w-automation/zapier | | /developer-guides/workflow-automation/zapier Not allowed host | https://github.com/firecrawl/firecrawl-d…:%20/features/agent | | /features/agent Not allowed host | https://github.com/firecrawl/firecrawl-d…atures/batch-scrape | | /features/batch-scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…20/features/browser | | /features/browser Not allowed host | https://github.com/firecrawl/firecrawl-d…res/change-tracking | | /features/change-tracking Not allowed host | https://github.com/firecrawl/firecrawl-d…:%20/features/crawl | | /features/crawl Not allowed host | https://github.com/firecrawl/firecrawl-d…es/document-parsing | | /features/document-parsing Not allowed host | https://github.com/firecrawl/firecrawl-d…tures/enhanced-mode | | /features/enhanced-mode Not allowed host | https://github.com/firecrawl/firecrawl-d…20/features/extract | | /features/extract Not allowed host | https://github.com/firecrawl/firecrawl-d…tures/fast-scraping | | /features/fast-scraping Not allowed host | https://github.com/firecrawl/firecrawl-d…0/features/interact | | /features/interact Not allowed host | https://github.com/firecrawl/firecrawl-d…eatures/llm-extract | | /features/llm-extract Not allowed host | https://github.com/firecrawl/firecrawl-d…th:%20/features/map | | /features/map Not allowed host | https://github.com/firecrawl/firecrawl-d…%20/features/models | | /features/models Not allowed host | https://github.com/firecrawl/firecrawl-d…20/features/proxies | | /features/proxies Not allowed host | https://github.com/firecrawl/firecrawl-d…%20/features/scrape | | /features/scrape Not allowed host | https://github.com/firecrawl/firecrawl-d…%20/features/search | | /features/search Not allowed host | https://github.com/firecrawl/firecrawl-d…th:%20/introduction | | /introduction Not allowed host | https://github.com/firecrawl/firecrawl-d…Path:%20/mcp-server | | /mcp-server Not allowed host | https://github.com/firecrawl/firecrawl-d…h:%20/migrate-to-v2 | | /migrate-to-v2 Not allowed host | https://github.com/firecrawl/firecrawl-d…ath:%20/rate-limits | | /rate-limits Not allowed host | https://github.com/firecrawl/firecrawl-d…y=Path:%20/sdks/cli | | /sdks/cli Not allowed host | https://github.com/firecrawl/firecrawl-d…dy=Path:%20/sdks/go | | /sdks/go Not allowed host | https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/java | | /sdks/java Not allowed host | https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/node | | /sdks/node Not allowed host | https://github.com/firecrawl/firecrawl-d…h:%20/sdks/overview | | /sdks/overview Not allowed host | https://github.com/firecrawl/firecrawl-d…ath:%20/sdks/python | | /sdks/python Not allowed host | https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/rust | | /sdks/rust Not allowed host | https://github.com/firecrawl/firecrawl-d…-cases/ai-platforms | | /use-cases/ai-platforms Not allowed host | https://github.com/firecrawl/firecrawl-d…titive-intelligence | | /use-cases/competitive-intelligence Not allowed host | https://github.com/firecrawl/firecrawl-d…/content-generation | | /use-cases/content-generation Not allowed host | https://github.com/firecrawl/firecrawl-d…ases/data-migration | | /use-cases/data-migration Not allowed host | https://github.com/firecrawl/firecrawl-d…cases/deep-research | | /use-cases/deep-research Not allowed host | https://github.com/firecrawl/firecrawl-d…ases/developers-mcp | | /use-cases/developers-mcp Not allowed host | https://github.com/firecrawl/firecrawl-d…/investment-finance | | /use-cases/investment-finance Not allowed host | https://github.com/firecrawl/firecrawl-d…ses/lead-enrichment | | /use-cases/lead-enrichment Not allowed host | https://github.com/firecrawl/firecrawl-d…cases/observability | | /use-cases/observability Not allowed host | https://github.com/firecrawl/firecrawl-d…/use-cases/overview | | /use-cases/overview Not allowed host | https://github.com/firecrawl/firecrawl-d…s/product-ecommerce | | /use-cases/product-ecommerce Not allowed host | https://github.com/firecrawl/firecrawl-d…cases/seo-platforms | | /use-cases/seo-platforms Not allowed host | https://github.com/firecrawl/firecrawl-d…erence/introduction | | /v1/api-reference/introduction Not allowed host | https://github.com/firecrawl/firecrawl-d…%20/webhooks/events | | /webhooks/events Not allowed host | https://github.com/firecrawl/firecrawl-d…0/webhooks/overview | | /webhooks/overview Not allowed host | https://github.com/firecrawl/firecrawl-d…0/webhooks/security | | /webhooks/security Not allowed host | https://github.com/firecrawl/firecrawl-d…20/webhooks/testing | | /webhooks/testing Not allowed host | https://github.com/firecrawl/firecrawl-mcp-server | | /mcp-server Not allowed host | https://github.com/firecrawl/firecrawl/blob/main/LICENSE | | /contributing/open-source-or-cloud Not allowed host | https://github.com/firecrawl/firecrawl/issues/new/choose | | /contributing/self-host Not allowed host | https://github.com/firecrawl/firecrawl/pull/586/ | | /contributing/guide Not allowed host | https://github.com/firecrawl/firecrawl/t…/main/apps/java-sdk | | /sdks/java Not allowed host | https://github.com/firecrawl/firecrawl/t…tes/cluster-install | | /contributing/self-host Not allowed host | https://github.com/firecrawl/firegeo | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/fireplexity | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/firesearch | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/firestarter | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/open-agent-builder | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/open-lovable | | /developer-guides/examples Not allowed host | https://github.com/firecrawl/open-researcher | | /developer-guides/examples Not allowed host | https://github.com/gstarwd | | /mcp-server Not allowed host | https://github.com/mendableai/firecrawl/issues | | /contributing/guide Not allowed host | https://github.com/microsoft/vscode/issues/155379 | | /mcp-server Not allowed host | https://github.com/vercel-labs/agent-browser | | /features/interact Not allowed host | https://github.com/vrknetha | | /mcp-server Not allowed host | https://github.com/xiangkaiz | | /mcp-server Not allowed host | https://github.com/zihaolin96 | | /mcp-server Not allowed host | https://help.openai.com/en/articles/1258…ors-in-chatgpt-beta | | /developer-guides/mcp-setup-guides/chatgpt Not allowed host | https://insiders.vscode.dev/redirect/mcp…AapiKey%7D%22%7D%7D | | /mcp-server Not allowed host | https://insiders.vscode.dev/redirect/mcp…7D&quality=insiders | | /mcp-server Not allowed host | https://js.langchain.com/docs | | /developer-guides/llm-sdks-and-frameworks/langchain Not allowed host | https://json-schema.org/ | | /api-reference/endpoint/extract Not allowed host | https://langchain-ai.github.io/langgraphjs/ | | /developer-guides/llm-sdks-and-frameworks/langgraph Not allowed host | https://marketplace.dify.ai/plugins/langgenius/firecrawl | | /developer-guides/workflow-automation/dify Not allowed host | https://mastra.ai/docs | | /developer-guides/llm-sdks-and-frameworks/mastra Not allowed host | https://mcp.firecrawl.dev/%7BFIRECRAWL_API_KEY%7D/v2/mcp | | /mcp-server Not allowed host | https://nodejs.org/en/learn/getting-star…w-to-install-nodejs | | /contributing/guide Not allowed host | https://platform.openai.com/api-keys | | /developer-guides/workflow-automation/n8n Not allowed host | https://platform.openai.com/docs | | /developer-guides/llm-sdks-and-frameworks/openai External URLs ------------- External URL | Pages | Found on URL (max 5) ------------------------------------------------------------------------------------------------------------------------------------- http://localhost:3000/ | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook http://localhost:3000/v2/mcp | 1 | /mcp-server https://ai-sdk.dev/docs | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://ai-sdk.dev/docs/foundations/providers-and-models | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://ai-sdk.dev/docs/foundations/tools | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://ai-sdk.dev/elements/overview | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://ai-sdk.dev/providers/ai-sdk-providers | 1 | /developer-guides/llm-sdks-and-frameworks/vercel-ai-sdk https://ai.google.dev/docs | 1 | /developer-guides/llm-sdks-and-frameworks/gemini https://chatgpt.com/ | 1 | /developer-guides/mcp-setup-guides/chatgpt https://community.n8n.io/ | 1 | /developer-guides/workflow-automation/n8n https://discord.gg/firecrawl | 1 | /introduction https://docs.anthropic.com/claude/docs | 1 | /developer-guides/llm-sdks-and-frameworks/anthropic https://docs.cursor.com/context/model-context-protocol | 1 | /mcp-server https://docs.docker.com/get-docker/ | 1 | /contributing/self-host https://docs.factory.ai/cli/getting-started/quickstart | 1 | /developer-guides/mcp-setup-guides/factory-ai https://docs.n8n.io/ | 1 | /developer-guides/workflow-automation/n8n https://docs.n8n.io/hosting/ | 1 | /developer-guides/workflow-automation/n8n https://firecrawl.dev/app | 1 | /developer-guides/mcp-setup-guides/factory-ai https://firecrawl.dev/blog | 1 | /introduction https://firecrawl.dev/changelog | 1 | /introduction https://firecrawl.dev/enterprise | 1 | /rate-limits https://firecrawl.dev/playground | 1 | /introduction https://firecrawl.dev/pricing | 1 | /developer-guides/usage-guides/choosing-the-data-extractor https://github.com/cloudflare/cloudflared/releases | 1 | /webhooks/testing https://github.com/directus/directus/issues/25906 | 1 | /mcp-server https://github.com/firecrawl/ai-ready-website | 1 | /developer-guides/examples https://github.com/firecrawl/cli | 1 | /sdks/cli https://github.com/firecrawl/fire-enrich | 1 | /developer-guides/examples https://github.com/firecrawl/firecrawl | 1 | /introduction https://github.com/firecrawl/firecrawl-d…nced-scraping-guide | 1 | /advanced-scraping-guide https://github.com/firecrawl/firecrawl-d…h:%20/agents/fire-1 | 1 | /agents/fire-1 https://github.com/firecrawl/firecrawl-d…ence/endpoint/agent | 1 | /api-reference/endpoint/agent https://github.com/firecrawl/firecrawl-d…dpoint/agent-delete | 1 | /api-reference/endpoint/agent-delete https://github.com/firecrawl/firecrawl-d…/endpoint/agent-get | 1 | /api-reference/endpoint/agent-get https://github.com/firecrawl/firecrawl-d…dpoint/batch-scrape | 1 | /api-reference/endpoint/batch-scrape https://github.com/firecrawl/firecrawl-d…batch-scrape-delete | 1 | /api-reference/endpoint/batch-scrape-delete https://github.com/firecrawl/firecrawl-d…nt/batch-scrape-get | 1 | /api-reference/endpoint/batch-scrape-get https://github.com/firecrawl/firecrawl-d…h-scrape-get-errors | 1 | /api-reference/endpoint/batch-scrape-get-errors https://github.com/firecrawl/firecrawl-d…oint/browser-create | 1 | /api-reference/endpoint/browser-create https://github.com/firecrawl/firecrawl-d…oint/browser-delete | 1 | /api-reference/endpoint/browser-delete https://github.com/firecrawl/firecrawl-d…int/browser-execute | 1 | /api-reference/endpoint/browser-execute https://github.com/firecrawl/firecrawl-d…dpoint/browser-list | 1 | /api-reference/endpoint/browser-list https://github.com/firecrawl/firecrawl-d…dpoint/crawl-active | 1 | /api-reference/endpoint/crawl-active https://github.com/firecrawl/firecrawl-d…dpoint/crawl-delete | 1 | /api-reference/endpoint/crawl-delete https://github.com/firecrawl/firecrawl-d…/endpoint/crawl-get | 1 | /api-reference/endpoint/crawl-get https://github.com/firecrawl/firecrawl-d…nt/crawl-get-errors | 1 | /api-reference/endpoint/crawl-get-errors https://github.com/firecrawl/firecrawl-d…rawl-params-preview | 1 | /api-reference/endpoint/crawl-params-preview https://github.com/firecrawl/firecrawl-d…endpoint/crawl-post | 1 | /api-reference/endpoint/crawl-post https://github.com/firecrawl/firecrawl-d…dpoint/credit-usage | 1 | /api-reference/endpoint/credit-usage https://github.com/firecrawl/firecrawl-d…it-usage-historical | 1 | /api-reference/endpoint/credit-usage-historical https://github.com/firecrawl/firecrawl-d…ce/endpoint/extract | 1 | /api-reference/endpoint/extract https://github.com/firecrawl/firecrawl-d…ndpoint/extract-get | 1 | /api-reference/endpoint/extract-get https://github.com/firecrawl/firecrawl-d…erence/endpoint/map | 1 | /api-reference/endpoint/map https://github.com/firecrawl/firecrawl-d…dpoint/queue-status | 1 | /api-reference/endpoint/queue-status https://github.com/firecrawl/firecrawl-d…nce/endpoint/scrape | 1 | /api-reference/endpoint/scrape https://github.com/firecrawl/firecrawl-d…rape-browser-delete | 1 | /api-reference/endpoint/scrape-browser-delete https://github.com/firecrawl/firecrawl-d…oint/scrape-execute | 1 | /api-reference/endpoint/scrape-execute https://github.com/firecrawl/firecrawl-d…nce/endpoint/search | 1 | /api-reference/endpoint/search https://github.com/firecrawl/firecrawl-d…ndpoint/token-usage | 1 | /api-reference/endpoint/token-usage https://github.com/firecrawl/firecrawl-d…en-usage-historical | 1 | /api-reference/endpoint/token-usage-historical https://github.com/firecrawl/firecrawl-d…ebhook-agent-action | 1 | /api-reference/endpoint/webhook-agent-action https://github.com/firecrawl/firecrawl-d…ook-agent-cancelled | 1 | /api-reference/endpoint/webhook-agent-cancelled https://github.com/firecrawl/firecrawl-d…ook-agent-completed | 1 | /api-reference/endpoint/webhook-agent-completed https://github.com/firecrawl/firecrawl-d…ebhook-agent-failed | 1 | /api-reference/endpoint/webhook-agent-failed https://github.com/firecrawl/firecrawl-d…bhook-agent-started | 1 | /api-reference/endpoint/webhook-agent-started https://github.com/firecrawl/firecrawl-d…ch-scrape-completed | 1 | /api-reference/endpoint/webhook-batch-scrape-completed https://github.com/firecrawl/firecrawl-d…k-batch-scrape-page | 1 | /api-reference/endpoint/webhook-batch-scrape-page https://github.com/firecrawl/firecrawl-d…atch-scrape-started | 1 | /api-reference/endpoint/webhook-batch-scrape-started https://github.com/firecrawl/firecrawl-d…ook-crawl-completed | 1 | /api-reference/endpoint/webhook-crawl-completed https://github.com/firecrawl/firecrawl-d…/webhook-crawl-page | 1 | /api-reference/endpoint/webhook-crawl-page https://github.com/firecrawl/firecrawl-d…bhook-crawl-started | 1 | /api-reference/endpoint/webhook-crawl-started https://github.com/firecrawl/firecrawl-d…dpoint/batch-scrape | 1 | /api-reference/v1-endpoint/batch-scrape https://github.com/firecrawl/firecrawl-d…batch-scrape-delete | 1 | /api-reference/v1-endpoint/batch-scrape-delete https://github.com/firecrawl/firecrawl-d…nt/batch-scrape-get | 1 | /api-reference/v1-endpoint/batch-scrape-get https://github.com/firecrawl/firecrawl-d…h-scrape-get-errors | 1 | /api-reference/v1-endpoint/batch-scrape-get-errors https://github.com/firecrawl/firecrawl-d…dpoint/crawl-active | 1 | /api-reference/v1-endpoint/crawl-active https://github.com/firecrawl/firecrawl-d…dpoint/crawl-delete | 1 | /api-reference/v1-endpoint/crawl-delete https://github.com/firecrawl/firecrawl-d…-endpoint/crawl-get | 1 | /api-reference/v1-endpoint/crawl-get https://github.com/firecrawl/firecrawl-d…nt/crawl-get-errors | 1 | /api-reference/v1-endpoint/crawl-get-errors https://github.com/firecrawl/firecrawl-d…endpoint/crawl-post | 1 | /api-reference/v1-endpoint/crawl-post https://github.com/firecrawl/firecrawl-d…dpoint/credit-usage | 1 | /api-reference/v1-endpoint/credit-usage https://github.com/firecrawl/firecrawl-d…it-usage-historical | 1 | /api-reference/v1-endpoint/credit-usage-historical https://github.com/firecrawl/firecrawl-d…v1-endpoint/extract | 1 | /api-reference/v1-endpoint/extract https://github.com/firecrawl/firecrawl-d…ndpoint/extract-get | 1 | /api-reference/v1-endpoint/extract-get https://github.com/firecrawl/firecrawl-d…nce/v1-endpoint/map | 1 | /api-reference/v1-endpoint/map https://github.com/firecrawl/firecrawl-d…dpoint/queue-status | 1 | /api-reference/v1-endpoint/queue-status https://github.com/firecrawl/firecrawl-d…/v1-endpoint/scrape | 1 | /api-reference/v1-endpoint/scrape https://github.com/firecrawl/firecrawl-d…/v1-endpoint/search | 1 | /api-reference/v1-endpoint/search https://github.com/firecrawl/firecrawl-d…ndpoint/token-usage | 1 | /api-reference/v1-endpoint/token-usage https://github.com/firecrawl/firecrawl-d…en-usage-historical | 1 | /api-reference/v1-endpoint/token-usage-historical https://github.com/firecrawl/firecrawl-d…nce/v2-introduction | 1 | /api-reference/v2-introduction https://github.com/firecrawl/firecrawl-d…dy=Path:%20/billing | 1 | /billing https://github.com/firecrawl/firecrawl-d…/contributing/guide | 1 | /contributing/guide https://github.com/firecrawl/firecrawl-d…pen-source-or-cloud | 1 | /contributing/open-source-or-cloud https://github.com/firecrawl/firecrawl-d…tributing/self-host | 1 | /contributing/self-host https://github.com/firecrawl/firecrawl-d…=Path:%20/dashboard | 1 | /dashboard https://github.com/firecrawl/firecrawl-d…common-sites/amazon | 1 | /developer-guides/common-sites/amazon https://github.com/firecrawl/firecrawl-d…s/common-sites/etsy | 1 | /developer-guides/common-sites/etsy https://github.com/firecrawl/firecrawl-d…common-sites/github | 1 | /developer-guides/common-sites/github https://github.com/firecrawl/firecrawl-d…mon-sites/wikipedia | 1 | /developer-guides/common-sites/wikipedia https://github.com/firecrawl/firecrawl-d…-assistant-cookbook | 1 | /developer-guides/cookbooks/ai-research-assistant-cookbook https://github.com/firecrawl/firecrawl-d…-generator-cookbook | 1 | /developer-guides/cookbooks/brand-style-…-generator-cookbook https://github.com/firecrawl/firecrawl-d…per-guides/examples | 1 | /developer-guides/examples https://github.com/firecrawl/firecrawl-d…rameworks/anthropic | 1 | /developer-guides/llm-sdks-and-frameworks/anthropic https://github.com/firecrawl/firecrawl-d…d-frameworks/gemini | 1 | /developer-guides/llm-sdks-and-frameworks/gemini https://github.com/firecrawl/firecrawl-d…ameworks/google-adk | 1 | /developer-guides/llm-sdks-and-frameworks/google-adk https://github.com/firecrawl/firecrawl-d…rameworks/langchain | 1 | /developer-guides/llm-sdks-and-frameworks/langchain https://github.com/firecrawl/firecrawl-d…rameworks/langgraph | 1 | /developer-guides/llm-sdks-and-frameworks/langgraph https://github.com/firecrawl/firecrawl-d…ameworks/llamaindex | 1 | /developer-guides/llm-sdks-and-frameworks/llamaindex https://github.com/firecrawl/firecrawl-d…d-frameworks/mastra | 1 | /developer-guides/llm-sdks-and-frameworks/mastra https://github.com/firecrawl/firecrawl-d…d-frameworks/openai | 1 | /developer-guides/llm-sdks-and-frameworks/openai https://github.com/firecrawl/firecrawl-d…works/vercel-ai-sdk | 1 | /developer-guides/llm-sdks-and-frameworks/vercel-ai-sdk https://github.com/firecrawl/firecrawl-d…etup-guides/chatgpt | 1 | /developer-guides/mcp-setup-guides/chatgpt https://github.com/firecrawl/firecrawl-d…-guides/claude-code | 1 | /developer-guides/mcp-setup-guides/claude-code https://github.com/firecrawl/firecrawl-d…setup-guides/cursor | 1 | /developer-guides/mcp-setup-guides/cursor https://github.com/firecrawl/firecrawl-d…p-guides/factory-ai | 1 | /developer-guides/mcp-setup-guides/factory-ai https://github.com/firecrawl/firecrawl-d…tup-guides/windsurf | 1 | /developer-guides/mcp-setup-guides/windsurf https://github.com/firecrawl/firecrawl-d…per-guides/openclaw | 1 | /developer-guides/openclaw https://github.com/firecrawl/firecrawl-d…-the-data-extractor | 1 | /developer-guides/usage-guides/choosing-the-data-extractor https://github.com/firecrawl/firecrawl-d…low-automation/dify | 1 | /developer-guides/workflow-automation/dify https://github.com/firecrawl/firecrawl-d…low-automation/make | 1 | /developer-guides/workflow-automation/make https://github.com/firecrawl/firecrawl-d…flow-automation/n8n | 1 | /developer-guides/workflow-automation/n8n https://github.com/firecrawl/firecrawl-d…w-automation/zapier | 1 | /developer-guides/workflow-automation/zapier https://github.com/firecrawl/firecrawl-d…:%20/features/agent | 1 | /features/agent https://github.com/firecrawl/firecrawl-d…atures/batch-scrape | 1 | /features/batch-scrape https://github.com/firecrawl/firecrawl-d…20/features/browser | 1 | /features/browser https://github.com/firecrawl/firecrawl-d…res/change-tracking | 1 | /features/change-tracking https://github.com/firecrawl/firecrawl-d…:%20/features/crawl | 1 | /features/crawl https://github.com/firecrawl/firecrawl-d…es/document-parsing | 1 | /features/document-parsing https://github.com/firecrawl/firecrawl-d…tures/enhanced-mode | 1 | /features/enhanced-mode https://github.com/firecrawl/firecrawl-d…20/features/extract | 1 | /features/extract https://github.com/firecrawl/firecrawl-d…tures/fast-scraping | 1 | /features/fast-scraping https://github.com/firecrawl/firecrawl-d…0/features/interact | 1 | /features/interact https://github.com/firecrawl/firecrawl-d…eatures/llm-extract | 1 | /features/llm-extract https://github.com/firecrawl/firecrawl-d…th:%20/features/map | 1 | /features/map https://github.com/firecrawl/firecrawl-d…%20/features/models | 1 | /features/models https://github.com/firecrawl/firecrawl-d…20/features/proxies | 1 | /features/proxies https://github.com/firecrawl/firecrawl-d…%20/features/scrape | 1 | /features/scrape https://github.com/firecrawl/firecrawl-d…%20/features/search | 1 | /features/search https://github.com/firecrawl/firecrawl-d…th:%20/introduction | 1 | /introduction https://github.com/firecrawl/firecrawl-d…Path:%20/mcp-server | 1 | /mcp-server https://github.com/firecrawl/firecrawl-d…h:%20/migrate-to-v2 | 1 | /migrate-to-v2 https://github.com/firecrawl/firecrawl-d…ath:%20/rate-limits | 1 | /rate-limits https://github.com/firecrawl/firecrawl-d…y=Path:%20/sdks/cli | 1 | /sdks/cli https://github.com/firecrawl/firecrawl-d…dy=Path:%20/sdks/go | 1 | /sdks/go https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/java | 1 | /sdks/java https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/node | 1 | /sdks/node https://github.com/firecrawl/firecrawl-d…h:%20/sdks/overview | 1 | /sdks/overview https://github.com/firecrawl/firecrawl-d…ath:%20/sdks/python | 1 | /sdks/python https://github.com/firecrawl/firecrawl-d…=Path:%20/sdks/rust | 1 | /sdks/rust https://github.com/firecrawl/firecrawl-d…-cases/ai-platforms | 1 | /use-cases/ai-platforms https://github.com/firecrawl/firecrawl-d…titive-intelligence | 1 | /use-cases/competitive-intelligence https://github.com/firecrawl/firecrawl-d…/content-generation | 1 | /use-cases/content-generation https://github.com/firecrawl/firecrawl-d…ases/data-migration | 1 | /use-cases/data-migration https://github.com/firecrawl/firecrawl-d…cases/deep-research | 1 | /use-cases/deep-research https://github.com/firecrawl/firecrawl-d…ases/developers-mcp | 1 | /use-cases/developers-mcp https://github.com/firecrawl/firecrawl-d…/investment-finance | 1 | /use-cases/investment-finance https://github.com/firecrawl/firecrawl-d…ses/lead-enrichment | 1 | /use-cases/lead-enrichment https://github.com/firecrawl/firecrawl-d…cases/observability | 1 | /use-cases/observability https://github.com/firecrawl/firecrawl-d…/use-cases/overview | 1 | /use-cases/overview https://github.com/firecrawl/firecrawl-d…s/product-ecommerce | 1 | /use-cases/product-ecommerce https://github.com/firecrawl/firecrawl-d…cases/seo-platforms | 1 | /use-cases/seo-platforms https://github.com/firecrawl/firecrawl-d…erence/introduction | 1 | /v1/api-reference/introduction https://github.com/firecrawl/firecrawl-d…%20/webhooks/events | 1 | /webhooks/events https://github.com/firecrawl/firecrawl-d…0/webhooks/overview | 1 | /webhooks/overview https://github.com/firecrawl/firecrawl-d…0/webhooks/security | 1 | /webhooks/security https://github.com/firecrawl/firecrawl-d…20/webhooks/testing | 1 | /webhooks/testing https://github.com/firecrawl/firecrawl-mcp-server | 1 | /mcp-server https://github.com/firecrawl/firecrawl/blob/main/LICENSE | 1 | /contributing/open-source-or-cloud https://github.com/firecrawl/firecrawl/issues/new/choose | 1 | /contributing/self-host https://github.com/firecrawl/firecrawl/pull/586/ | 1 | /contributing/guide https://github.com/firecrawl/firecrawl/t…/main/apps/java-sdk | 1 | /sdks/java https://github.com/firecrawl/firecrawl/t…tes/cluster-install | 1 | /contributing/self-host https://github.com/firecrawl/firegeo | 1 | /developer-guides/examples https://github.com/firecrawl/fireplexity | 1 | /developer-guides/examples https://github.com/firecrawl/firesearch | 1 | /developer-guides/examples https://github.com/firecrawl/firestarter | 1 | /developer-guides/examples https://github.com/firecrawl/open-agent-builder | 1 | /developer-guides/examples https://github.com/firecrawl/open-lovable | 1 | /developer-guides/examples https://github.com/firecrawl/open-researcher | 1 | /developer-guides/examples https://github.com/gstarwd | 1 | /mcp-server https://github.com/mendableai/firecrawl/issues | 1 | /contributing/guide https://github.com/microsoft/vscode/issues/155379 | 1 | /mcp-server https://github.com/vercel-labs/agent-browser | 1 | /features/interact https://github.com/vrknetha | 1 | /mcp-server https://github.com/xiangkaiz | 1 | /mcp-server https://github.com/zihaolin96 | 1 | /mcp-server https://help.openai.com/en/articles/1258…ors-in-chatgpt-beta | 1 | /developer-guides/mcp-setup-guides/chatgpt https://insiders.vscode.dev/redirect/mcp…AapiKey%7D%22%7D%7D | 1 | /mcp-server https://insiders.vscode.dev/redirect/mcp…7D&quality=insiders | 1 | /mcp-server https://js.langchain.com/docs | 1 | /developer-guides/llm-sdks-and-frameworks/langchain https://json-schema.org/ | 1 | /api-reference/endpoint/extract https://langchain-ai.github.io/langgraphjs/ | 1 | /developer-guides/llm-sdks-and-frameworks/langgraph https://marketplace.dify.ai/plugins/langgenius/firecrawl | 1 | /developer-guides/workflow-automation/dify https://mastra.ai/docs | 1 | /developer-guides/llm-sdks-and-frameworks/mastra https://mcp.firecrawl.dev/%7BFIRECRAWL_API_KEY%7D/v2/mcp | 1 | /mcp-server https://nodejs.org/en/learn/getting-star…w-to-install-nodejs | 1 | /contributing/guide https://platform.openai.com/api-keys | 1 | /developer-guides/workflow-automation/n8n https://platform.openai.com/docs | 1 | /developer-guides/llm-sdks-and-frameworks/openai Redirected URLs --------------- Status | Redirected URL | Target URL | Found at URL -------------------------------------------------------------------------------------------------------------------------------------- 308 | / | /introduction | 404 URLs -------- Status | URL 404 | Found at URL ---------------------------------------------------------------------------------------------------------------------------------------- 404 | /pricing | /features/document-parsing SSL/TLS info ------------ Info | Text --------------------------------------------------------------------------------------------------------------------------------------- Issuer | C = US, O = Let's Encrypt, CN = R12 Subject | CN = docs.firecrawl.dev Valid from | Feb 28 05:37:34 2026 GMT (VALID already 24.4 day(s)) Valid to | May 29 05:37:33 2026 GMT (VALID still for 65.6 day(s)) Supported protocols | TLSv1.2, TLSv1.3 RAW certificate output | Certificate: Data: Version: 3 (0x2) Serial Number:…6:bc:a4:91:0a: f9:92:38:1a RAW protocols output | === ssl2 === s_client: Unknown option: -ssl2 s_client: Use -help for su…Max Early Data: 0 --- read R BLOCK TOP fastest URLs ---------------- Time | Status | Fast URL ------------------------------------------------------------------------------------------------------------------------------------------ 71 ms | 200 | /api-reference/endpoint/token-usage-historical 72 ms | 200 | /use-cases/deep-research 75 ms | 200 | /api-reference/endpoint/webhook-agent-started 75 ms | 200 | /api-reference/v1-endpoint/queue-status 75 ms | 200 | /api-reference/v1-endpoint/crawl-delete 75 ms | 200 | /api-reference/v1-endpoint/map 76 ms | 200 | /api-reference/v1-endpoint/extract-get 76 ms | 200 | /api-reference/v1-endpoint/batch-scrape 76 ms | 200 | /developer-guides/workflow-automation/make 76 ms | 200 | /developer-guides/llm-sdks-and-frameworks/mastra 77 ms | 200 | /developer-guides/mcp-setup-guides/windsurf 77 ms | 200 | /features/enhanced-mode 77 ms | 200 | /api-reference/v1-endpoint/crawl-get-errors 77 ms | 200 | /use-cases/observability 79 ms | 200 | /api-reference/v1-endpoint/crawl-active 79 ms | 200 | /developer-guides/common-sites/wikipedia 79 ms | 200 | /api-reference/endpoint/batch-scrape 80 ms | 200 | /agents/fire-1 80 ms | 200 | /v1/api-reference/introduction 81 ms | 200 | /developer-guides/mcp-setup-guides/cursor TOP slowest URLs ---------------- Time | Status | Slow URL ------------------------------------------------------------------------------------------------------------------------------------- 394 ms | 200 | /api-reference/endpoint/crawl-post 342 ms | 200 | /sdks/go 302 ms | 200 | /sdks/rust 269 ms | 404 | /pricing 267 ms | 200 | /api-reference/v1-endpoint/scrape 213 ms | 200 | /api-reference/endpoint/credit-usage-historical 200 ms | 200 | /developer-guides/mcp-setup-guides/claude-code 181 ms | 200 | /developer-guides/mcp-setup-guides/factory-ai 174 ms | 200 | /features/interact 174 ms | 200 | /sdks/node 173 ms | 200 | /sdks/java 167 ms | 200 | /api-reference/endpoint/scrape-browser-delete 166 ms | 200 | /api-reference/v1-endpoint/token-usage 165 ms | 200 | /api-reference/v1-endpoint/token-usage-historical 163 ms | 200 | /developer-guides/common-sites/github 162 ms | 200 | /contributing/self-host 162 ms | 200 | /developer-guides/cookbooks/brand-style-guide-generator-cookbook 156 ms | 200 | /use-cases/ai-platforms 153 ms | 200 | /contributing/open-source-or-cloud 152 ms | 200 | /use-cases/data-migration SEO metadata ------------ This table contains large data and shows max 10 rows. To see them all, use output to HTML using `--output-html-report=tmp/myreport.html`. URL | Indexing | Title | H1 | Description | Keywords --------------------------------------------------------------------------------------------------------------------------------------- /advanced-scraping-guide | Allowed | Advanced…awl | Advanced…ide | Configur…ce. | /agents/fire-1 | Allowed | FIRE-1 A…awl | FIRE-1 A…ta) | AI agent…ges | /api-reference/endpoint/agent | Allowed | Agent -…ocs | Agent | | /api-reference/endpoint/agent-delete | Allowed | Cancel A…ocs | Cancel Agent | | /api-reference/endpoint/agent-get | Allowed | Get Agen…ocs | Get Agen…tus | | /api-reference/endpoint/batch-scrape | Allowed | Batch Sc…ocs | Batch Scrape | | /api-reference/endpoint/batch-scrape-delete | Allowed | Cancel B…ocs | Cancel B…ape | | /api-reference/endpoint/batch-scrape-get | Allowed | Get Batc…ocs | Get Batc…tus | | /api-reference/endpoint/batch-scrape-get-errors | Allowed | Get Batc…ocs | Get Batc…ors | | /api-reference/endpoint/browser-create | Allowed | Create B…ocs | Create B…ion | Launch a…nt. | OpenGraph metadata ------------------ This table contains large data and shows max 10 rows. To see them all, use output to HTML using `--output-html-report=tmp/myreport.html`. URL | OG Title | OG Description | OG Image | Twitter Title | Twitter Description | Twitter Image -------------------------------------------------------------------------------------------------------------------------------------------------- /advanced-scraping-guide | Advance…wl | Configu…e. | https://fire…q=100 | Advance…wl | Configu…e. | https://fire…q=100 /agents/fire-1 | FIRE-1…wl | AI agen…es | https://fire…q=100 | FIRE-1…wl | AI agen…es | https://fire…q=100 /api-reference/endpoint/agent | Agent -…cs | | https://fire…q=100 | Agent -…cs | | https://fire…q=100 /api-reference/endpoint/agent-delete | Cancel…cs | | https://fire…q=100 | Cancel…cs | | https://fire…q=100 /api-reference/endpoint/agent-get | Get Age…cs | | https://fire…q=100 | Get Age…cs | | https://fire…q=100 /api-reference/endpoint/batch-scrape | Batch S…cs | | https://fire…q=100 | Batch S…cs | | https://fire…q=100 /api-reference/endpoint/batch-scrape-delete | Cancel…cs | | https://fire…q=100 | Cancel…cs | | https://fire…q=100 /api-reference/endpoint/batch-scrape-get | Get Bat…cs | | https://fire…q=100 | Get Bat…cs | | https://fire…q=100 /api-reference/endpoint/batch-scrape-get-errors | Get Bat…cs | | https://fire…q=100 | Get Bat…cs | | https://fire…q=100 /api-reference/endpoint/browser-create | Create…cs | Launch…t. | https://fire…q=100 | Create…cs | Launch…t. | https://fire…q=100 Heading structure ----------------- This table contains large data and shows max 10 rows. To see them all, use output to HTML using `--output-html-report=tmp/myreport.html`. Heading structure | Count | Errors | URL ----------------------------------------------------------------------------------------------------------------------------------------

FIRE-1 Agent (Beta) [#page-title]

​What FIRE-1…​Rate limits [#rate-limits] | 5 | 1 | /agents/fire-1

Quickstart [#page-title]

​Scrape your first we…en Source

Integrations | 24 | 0 | /introduction

Advanced Scraping Guide [#page-title]

​Basic s…-to-call-the-firecrawl-api] | 25 | 0 | /advanced-scraping-guide

Lead Enrichment [#page-title]

​Start with a Te…Cases [#related-use-cases] | 15 | 0 | /use-cases/lead-enrichment

Interact after scraping [#page-title]

AI promp…

​Response [#response] | 24 | 0 | /features/interact

Map [#page-title]

​Introducing /map [#introduc…derations [#considerations] | 12 | 0 | /features/map

Scrape [#page-title]

Try it in the Playground…[#zero-data-retention-zdr] | 30 | 0 | /features/scrape

Use Cases [#page-title]

AI Platforms

Lead…igration

Observability | 12 | 0 | /use-cases/overview

Skill + CLI [#page-title]

​Installation [#inst…​Open Source [#open-source] | 30 | 0 | /sdks/cli

Security [#page-title]

​Secret Key [#secret-ke…h3> ​Use HTTPS [#use-https] | 9 | 0 | /webhooks/security HTTP headers ------------ Header | Occurs | Unique | Values preview | Min value | Max value -------------------------------------------------------------------------------------------------------------------------- Age | 137 | - | [ignored generic values] | 10.3 hour(s) | 11.5 hour(s) Cache-Control | 140 | 1 | no-store, no-cache, must-revalid…date, max-age=0 | | Cf-Cache-Status | 140 | 2 | HIT (138) / MISS (2) | | Cf-Ray | 140 | - | [ignored generic values] | | Content-Security-Policy | 140 | 1 | worker-src * blob: data: 'unsafe…codesandbox.io; | | Content-Type | 140 | 2 | text/html; charset=utf-8 (139) / text/html (1) | | Date | 140 | - | [ignored generic values] | 2026-03-24 | 2026-03-24 Expires | 140 | - | [ignored generic values] | | Link | 140 | 1 | ; rel="llms-txt", ; rel="llms-txt", ; rel="llms-full-txt" Location | 1 | /introduction Pragma | 140 | no-cache Server | 140 | Vercel Strict-Transport-Security | 140 | max-age=63072000 Vary | 139 | rsc, next-router-state-tree, next-router-prefetch, next…-prefetch, Accept-Encoding Vary | 1 | rsc, next-router-state-tree, next-router-prefetch, next…-prefetch, accept-encoding X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/sdks/overview#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/contribu…-source-or-cloud#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/introduction#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/contributing/guide#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/advanced-scraping-guide#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/sdks/cli#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/features/search#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/use-cases/overview#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/webhooks/security#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/dashboard#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/developer-guides/examples#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/features/interact#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/features/scrape#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/use-cases/lead-enrichment#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/developer-guides/openclaw#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/features/map#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/use-cases/seo-platforms#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/mcp-server#html=html X-Cache-Key | 1 | firecrawl/491/dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2/api-refe…/v2-introduction#html=html X-Frame-Options | 140 | DENY X-Llms-Txt | 140 | /llms.txt X-Matched-Path | 140 | /_sites/[subdomain]/[[...slug]] X-Mint-Proxy-Version | 140 | 1.0.0-prod X-Mintlify-Client-Version | 140 | 0.0.2698 X-Nextjs-Prerender | 140 | 1 X-Nextjs-Stale-Time | 140 | 60 X-Served-Version | 140 | dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2 X-Vercel-Cache | 137 | MISS X-Vercel-Cache | 3 | HIT X-Vercel-Id | 1 | arn1:cle1:iad1::iad1::k88sj-1774362870770-f021b7a0cd10 X-Vercel-Id | 1 | arn1:iad1::iad1::4x4xm-1774362872067-97e115368499 X-Vercel-Id | 1 | arn1:iad1::iad1::dxhhq-1774362871758-854811a703b0 X-Vercel-Id | 1 | arn1:iad1::iad1::mdjvm-1774362871256-dbf62b9ebefe X-Vercel-Id | 1 | arn1:iad1::iad1::wwn2q-1774362871957-8811960de8a6 X-Vercel-Id | 1 | arn1:iad1::iad1::ffxww-1774362872256-3a0f0e6b13e7 X-Vercel-Id | 1 | arn1:iad1::iad1::k88sj-1774362871694-55fea7b466b3 X-Vercel-Id | 1 | arn1:iad1::iad1::wkjgd-1774362872208-f579b5239a1c X-Vercel-Id | 1 | arn1:iad1::iad1::wwn2q-1774362872854-a4aae6956d6a X-Vercel-Id | 1 | arn1:iad1::iad1::gjrxr-1774362872857-49e54dc7ea2f X-Vercel-Id | 1 | arn1:iad1::iad1::dxhhq-1774362871033-b824ae12fb3c X-Vercel-Id | 1 | arn1:iad1::iad1::dxhhq-1774362871436-af8d8bb347c0 X-Vercel-Id | 1 | arn1:iad1::iad1::ffxww-1774362871597-69aeefbf692b X-Vercel-Id | 1 | arn1:iad1::iad1::ffxww-1774362872592-32dc3597d975 X-Vercel-Id | 1 | arn1:iad1::iad1::4x4xm-1774362873011-85d00b785d05 X-Vercel-Id | 1 | arn1:iad1::iad1::bq9ct-1774362872657-8fb19b8c2eaa X-Vercel-Id | 1 | arn1:iad1::iad1::4x4xm-1774362872370-4acb3f38cac2 X-Vercel-Id | 1 | arn1:iad1::iad1::dxhhq-1774362872459-02f499274b49 X-Vercel-Id | 1 | arn1:iad1::iad1::gjrxr-1774362871902-4d4625f8bb7e X-Vercel-Id | 1 | arn1:iad1::iad1::4x4xm-1774362871458-ec9c79f63785 X-Vercel-Project-Id | 140 | prj_3kakCEKDVpOxnQIJmKyTWs83RXEa X-Version | 140 | dpl_Cy2pZ1YHL7un8yN36DStGn7XHFL2 HTTP Caching by content type (only from crawlable domains) ---------------------------------------------------------- Content type | Cache type | URLs | AVG lifetime | MIN lifetime | MAX lifetime ---------------------------------------------------------------------------- HTML | Cache-Control | 139 | 0 s | 0 s | 0 s Redirect | Cache-Control | 1 | 0 s | 0 s | 0 s HTTP Caching by domain ---------------------- Domain | Cache type | URLs | AVG lifetime | MIN lifetime | MAX lifetime ------------------------------------------------------------------------------------ docs.firecrawl.dev | Cache-Control | 140 | 0 s | 0 s | 0 s HTTP Caching by domain and content type --------------------------------------- Domain | Content type | Cache type | URLs | AVG lifetime | MIN lifetime | MAX lifetime --------------------------------------------------------------------------------------------------- docs.firecrawl.dev | HTML | Cache-Control | 139 | 0 s | 0 s | 0 s docs.firecrawl.dev | Redirect | Cache-Control | 1 | 0 s | 0 s | 0 s TOP non-unique titles --------------------- Count | Title ------------------------------------------------------------------------------------------------------------------------------------------ 2 | Queue Status - Firecrawl Docs 2 | Cancel Batch Scrape - Firecrawl Docs 2 | Get Active Crawls - Firecrawl Docs 2 | Get Crawl Status - Firecrawl Docs 2 | Get Batch Scrape Status - Firecrawl Docs 2 | Crawl - Firecrawl Docs 2 | Scrape - Firecrawl Docs 2 | Historical Credit Usage - Firecrawl Docs 2 | Get Crawl Errors - Firecrawl Docs 2 | Batch Scrape - Firecrawl Docs TOP non-unique descriptions --------------------------- Count | Description ------------------------------------------------------------------------------------------------------------------------------------------ 53 | Best practices -------------- Analysis name | OK | Notice | Warning | Critical -------------------------------------------------------------------------------- Heading structure | 138 | 1 | 193 | 0 Duplicate inline SVGs (> 5 and > 1024 B) | 167 | 0 | 0 | 0 Invalid inline SVGs | 167 | 0 | 0 | 0 Non-clickable phone numbers | 11 | 0 | 1 | 0 DOM depth (> 30) | 139 | 0 | 0 | 0 Large inline SVGs (> 5120 B) | 167 | 0 | 0 | 0 Title uniqueness (> 10%) | 118 | 0 | 0 | 0 Description uniqueness (> 10%) | 85 | 0 | 1 | 0 Brotli support | 0 | 0 | 138 | 0 WebP support | 0 | 0 | 1 | 0 AVIF support | 0 | 0 | 1 | 0 Accessibility ------------- Analysis name | OK | Notice | Warning | Critical -------------------------------------------------------------------- Missing aria labels | 36 | 0 | 209 | 0 Missing image alt attributes | 58 | 0 | 0 | 0 Missing html lang attribute | 1 | 0 | 0 | 0 Missing roles | 0 | 0 | 3 | 0 Source domains -------------- Domain | Totals | HTML | Redirect ------------------------------------------------------------------ docs.firecrawl.dev | 140/121MB/16s | 139/121MB/16s | 1/91B/264ms Content types ------------- Content type | URLs | Total size | Total time | Avg time | Status 20x | Status 30x | Status 40x -------------------------------------------------------------------------------------------------- HTML | 139 | 121 MB | 16 s | 116 ms | 138 | 0 | 1 Redirect | 1 | 91 B | 264 ms | 264 ms | 0 | 1 | 0 Content types (MIME types) -------------------------- Content type | URLs | Total size | Total time | Avg time | Status 20x | Status 30x | Status 40x ---------------------------------------------------------------------------------------------------------------- text/html; charset=utf-8 | 139 | 121 MB | 16 s | 116 ms | 138 | 0 | 1 text/html | 1 | 91 B | 264 ms | 264 ms | 0 | 1 | 0 DNS info -------- DNS resolving tree ------------------------------------------------------------------------ docs.firecrawl.dev cname.vercel-dns.com IPv4: cname.vercel-dns.com. IPv4: 76.76.21.22 IPv4: 66.33.60.129 DNS server: 127.0.0.53 Security -------- Header | OK | Notice | Warning | Critical | Recommendation ----------------------------------------------------------------------------------------------------------------------------------------- X-Content-Type-Options | 0 | 0 | 139 | 0 | X-Content-Type-Options header is not set. It s…content type attacks. Referrer-Policy | 0 | 0 | 139 | 0 | Referrer-Policy header is not set. It controls…privacy and security. Feature-Policy | 0 | 0 | 139 | 0 | Feature-Policy header is not set. It allows en…ssions-Policy is set. Permissions-Policy | 0 | 0 | 139 | 0 | Permissions-Policy header is not set. It allow…eatures for security. Server | 0 | 139 | 0 | 0 | Server header is set to 'Vercel'. It is better…al used technologies. Strict-Transport-Security | 139 | 0 | 0 | 0 | X-Frame-Options | 139 | 0 | 0 | 0 | X-XSS-Protection | 139 | 0 | 0 | 0 | Content-Security-Policy | 139 | 0 | 0 | 0 | Analysis stats -------------- Class::method | Exec time | Exec count ------------------------------------------------------------------------------- BestPracticeAnalyzer::checkNonClickablePhoneNumbers | 977 ms | 139 BestPracticeAnalyzer::checkHeadingStructure | 955 ms | 139 AccessibilityAnalyzer::checkMissingLabels | 585 ms | 138 AccessibilityAnalyzer::checkMissingAriaLabels | 543 ms | 138 AccessibilityAnalyzer::checkMissingRoles | 455 ms | 138 AccessibilityAnalyzer::checkMissingLang | 401 ms | 138 BestPracticeAnalyzer::checkMaxDOMDepth | 383 ms | 139 SslTlsAnalyzer::getTLSandSSLCertificateInfo | 330 ms | 1 BestPracticeAnalyzer::checkInlineSvg | 107 ms | 139 BestPracticeAnalyzer::checkMissingQuotesOnAttributes | 44 ms | 139 SeoAndOpenGraphAnalyzer::analyzeHeadings | 16 ms | 1 AccessibilityAnalyzer::checkImageAltAttributes | 15 ms | 138 SecurityAnalyzer::checkHtmlSecurity | 14 ms | 139 SecurityAnalyzer::checkHeaders | 3 ms | 139 SeoAndOpenGraphAnalyzer::analyzeSeo | 0 ms | 1 SeoAndOpenGraphAnalyzer::analyzeOpenGraph | 0 ms | 1 BestPracticeAnalyzer::checkTitleUniqueness | 0 ms | 1 BestPracticeAnalyzer::checkMetaDescriptionUniqueness | 0 ms | 1 BestPracticeAnalyzer::checkBrotliSupport | 0 ms | 1 BestPracticeAnalyzer::checkWebpSupport | 0 ms | 1 BestPracticeAnalyzer::checkAvifSupport | 0 ms | 1 Content processor stats ----------------------- Class::method | Exec time | Exec count ----------------------------------------------------------------------------------- NextJsProcessor::applyContentChangesBeforeUrlParsing | 1.1 s | 139 JavaScriptProcessor::findUrls | 596 ms | 139 HtmlProcessor::findUrls | 337 ms | 140 CssProcessor::findUrls | 32 ms | 139 AstroProcessor::findUrls | 12 ms | 139 AstroProcessor::applyContentChangesBeforeUrlParsing | 0 ms | 139 NextJsProcessor::findUrls | 0 ms | 139 JavaScriptProcessor::applyContentChangesBeforeUrlParsing | 0 ms | 139 SvelteProcessor::applyContentChangesBeforeUrlParsing | 0 ms | 139 SvelteProcessor::findUrls | 0 ms | 139 HtmlProcessor::applyContentChangesBeforeUrlParsing | 0 ms | 140 CssProcessor::applyContentChangesBeforeUrlParsing | 0 ms | 139