Connect AI agents (MCP): Claude, ChatGPT, Cursor, VS Code

Claude

claude.ai, Claude Desktop and the mobile apps

1Open Settings → Connectors and click Add custom connector. On Team and Enterprise plans an owner adds it first, in the organization's connector settings.
2Name it Crawley Cloud and paste the URL. Leave the OAuth client ID and secret empty.
3Click Connect, sign in to Crawley Cloud and click Allow.
4In a chat, turn it on from the tools menu. It follows you to Desktop and mobile.

Claude Code

In your terminal

1Add the server:

claude mcp add --transport http crawley https://crawleycloud.com/mcp

2Run /mcp in Claude Code, pick crawley, choose Authenticate and click Allow in the browser.

ChatGPT

Plans with developer mode for connectors

1Open Settings → Apps & Connectors → Advanced settings and turn on Developer mode.
2Back in Apps & Connectors, click Create. Name it Crawley Cloud, paste the URL as the MCP server URL and choose OAuth authentication.
3Sign in to Crawley Cloud when it opens and click Allow. Then pick it in a chat.

Cursor

Cursor Settings → MCP, or ~/.cursor/mcp.json

1Add a new MCP server, or put this in mcp.json:

{
  "mcpServers": {
    "crawley": { "url": "https://crawleycloud.com/mcp" }
  }
}

2Cursor shows the server as needing sign-in: click Connect (or Login), sign in and click Allow.

VS Code

GitHub Copilot agent mode

1Run MCP: Add Server… from the Command Palette, choose HTTP and paste the URL, or add this to .vscode/mcp.json:

{
  "servers": {
    "crawley": { "type": "http", "url": "https://crawleycloud.com/mcp" }
  }
}

2Start the server, accept the sign-in prompt and click Allow. Its tools appear in agent mode.

Any MCP client

Windsurf, the MCP Inspector, your own agent…

Add a Streamable HTTP server with the URL above. Clients with OAuth find everything on their own: discovery, dynamic client registration and PKCE, then the same Allow page.

No OAuth? Create a token in Dashboard → AI agents (shown once, revocable) and send it as a header:

Authorization: Bearer cwmcp_…

Projects and recipes

Find, create, copy and organize scraping projects, or start one from a ready-made recipe.

List projectslist_projects(Reads)
Project detailsget_project(Reads)
Create a projectcreate_project(Changes)
Update a projectupdate_project(Changes)
Duplicate a projectclone_project(Changes)
Delete a projectdelete_project(Confirms)
Browse recipeslist_recipes(Reads)
Create a project from a recipecreate_from_recipe(Changes)
Publish a project as a recipepublish_recipe(Confirms)

Crawlers and fields

Point a crawler at the pages, detect and test the fields to extract, and repair selectors when a site changes.

Configure crawlingconfigure_crawler(Confirms)
Set proxies, site login or headersset_crawler_credentials(Confirms)
Test the site logintest_site_login(Changes)
Detect fields on a pagedetect_fields(Reads)
Preview extracted rowspreview_fields(Reads)
Ask AI for a selectorsuggest_selector(Changes)
Set the fields to extractset_fields(Changes)
Change one fieldupdate_field(Changes)
Remove one fielddelete_field(Confirms)
Create or change a page typeset_page_type(Changes)
Delete a page typedelete_page_type(Confirms)
Restore saved fieldsrestore_field_version(Changes)

Runs and schedules

Start and watch runs, see what the crawler saw, and keep projects running on a schedule.

Start a runstart_run(Confirms)
Cancel a runcancel_run(Changes)
List runslist_runs(Reads)
Run detailsget_run(Reads)
What a run sawget_run_snapshot(Reads)
List scheduleslist_schedules(Reads)
Create or change a scheduleset_schedule(Confirms)
Delete a scheduledelete_schedule(Confirms)
Run a schedule nowrun_schedule(Confirms)

Data, changes and exports

Read and filter the rows, compare runs (new, removed and changed values, like prices), export or send them to Google Sheets.

Read scraped rowsget_data(Reads)
What changed between runsget_changes(Reads)
Download link for a tableexport_data(Reads)
Delete scraped rowsclear_data(Confirms)
Fix stored rowsupdate_row(Confirms)
Delete stored rowsdelete_row(Confirms)
Rename a tablerename_table(Changes)
Send a table to Google Sheetspush_to_sheet(Confirms)

Scrape a page now

Turn one page into Markdown or rows without creating a project, or pull items out with AI.

Scrape one page nowscrape_page(Reads)
Extract from a page with AIai_extract_page(Changes)

Alerts and integrations

Email, Slack and webhook notifications, field alerts, Zapier/Make/n8n hooks and deliveries to SQL, S3, Airtable or Sheets.

Change a project's notificationsupdate_alerts(Confirms)
Create or change a field alertset_field_alert(Confirms)
Delete a field alertdelete_field_alert(Confirms)
Create or change a REST hookset_rest_hook(Confirms)
Delete a REST hookdelete_rest_hook(Confirms)
Send data somewhere after each runset_delivery(Confirms)
Delete a data deliverydelete_delivery(Confirms)
Check a delivery's connectiontest_delivery_connection(Reads)
Test a webhook, alert or REST hooktest_integration(Confirms)

Team

Share projects with members and viewers, and accept invitations.

Share a projectshare_project(Confirms)
Remove someone from a projectunshare_project(Confirms)
Accept a project invitationaccept_invite(Changes)

Account, plan and access

Plan and quota, analytics and the activity log, connected apps and API keys. Plan changes are a Stripe link you open yourself.

Account overviewget_account(Reads)
Usage analyticsget_analytics(Reads)
Recent account activityget_activity(Reads)
Plans and priceslist_plans(Reads)
Upgrade or switch planchange_plan(Confirms)
Manage the subscriptionbilling_portal(Confirms)
API keys and connected appslist_access(Reads)
Revoke an API key or disconnect an apprevoke_access(Confirms)

Troubleshooting and the REST API

Find out what needs attention and how to fix it, and reach anything else in the Crawley Cloud REST API.

What needs attentiondiagnose(Reads)
Read from the REST APIapi_read(Reads)
Create with the REST API (POST)api_create(Confirms)
Update with the REST API (PATCH/PUT)api_update(Confirms)
Delete with the REST APIapi_delete(Confirms)

Things to ask

Plain language works. The agent picks the tools, and asks you before anything that matters.

“Track the prices on books.toscrape.com every morning and email me when one drops.”
“Is anything wrong with my projects? Fix what you can and tell me what you changed.”
“Why did my last run come back empty?”
“What changed in my competitor data this week? Summarize the biggest price moves.”
“Turn this page into Markdown: https://example.com/pricing”
“Send the rows of my Books project to this Google Sheet after every run.”

Ready-made prompts

Clients that show MCP prompts (like Claude's “+” menu) offer these:

Monitor a competitor's prices

Build a scraper for a product list, run it, and set up a schedule and price-change alerts.

Fix a broken crawler

Find out why a project fails, scrapes nothing or lost fields, repair it and prove it with a run.

Weekly data review

What changed in the scraped data this week, what broke, and what to do about it.

Signs in as you, with OAuth

You approve the connection on a Crawley Cloud page. The app gets a token, never your password, and acts with your role on each project: owners do everything, members edit and run, viewers only read.

Reads run, changes ask

21 tools only read. Every other tool is marked as one that changes something, so Claude, ChatGPT and other clients ask you before each call.

A preview before anything big

Deleting, a large crawl, a schedule that spends quota, sending data or messages outside your account (webhooks, Slack, someone else's email, Google Sheets, invitations) and public recipes: the first call only says what would happen. Nothing happens until you confirm.

It can't pay or move money

Upgrading returns a Stripe Checkout link that you open and pay yourself; the subscription is managed on Stripe's page. Billing and payment endpoints are out of the agent's reach.

Secrets stay write-only

Proxies, site logins, extra headers, webhook URLs and database or storage keys can be set by the agent but are never shown back to it. API keys show only their last 4 characters.

Scraped content is data, never instructions

Page text and HTML, scraped rows, recipe texts, logs and run errors come from websites and other people, so the server hands them to the agent as untrusted data. A page can't give your agent orders or confirm an action for you.

Revoke any time, see everything

Disconnect an app in Dashboard → AI agents. Changing your password disconnects every app. Every change an agent makes shows in Security → Recent account activity as “AI agent: …”.

On other websites, it can…

Fetch the pages you point it at, and crawl them from our servers with your project's settings: robots.txt honored unless you turn it off, polite delays and rate limits.
Run browser steps (click, scroll, type in a search box) to reach the data, and log in to a site with credentials you give it.
Use your plan's item quota for every run, like a run you start yourself.

…and it can't

Use your own browser, cookies or signed-in sessions on other sites: crawls run on our servers.
Read back site passwords, proxies or headers saved in a project.
Pay, buy or move money anywhere, or go past your plan's limits.
Take orders from the pages it reads: what a site says is data for your agent, not instructions.

What an agent scrapes on your behalf falls under our Acceptable Use Policy, just like your own scraping: collect only what you're allowed to, and respect each site's terms.

Connect your AI agent to Crawley Cloud

Set it up in a minute