--- name: webcrawlerapi description: "Scrape a single page or crawl a full website using WebCrawlerAPI. Trigger for: fetching page content, getting markdown from a URL, scraping a page, crawling a website/domain, websearch, webfetch." --- # WebCrawlerAPI Skill Use WebCrawlerAPI to get page content as markdown (single page scrape) or crawl entire websites (multi-page). ## Setup — API Key The API key must be set as an environment variable before running any curl commands: ```bash export WEBCRAWLERAPI_API_KEY="your_api_key" ``` Get your key: 1. Go to https://webcrawlerapi.com/ 2. Sign up at https://dash.webcrawlerapi.com/ 3. Visit https://dash.webcrawlerapi.com/access 4. Copy your API key If `WEBCRAWLERAPI_API_KEY` is not set, stop and ask the user to set it before proceeding. ## Decision: Scrape vs Crawl **Scrape** (single page) — default when user asks for: - Content/markdown of a specific page or URL - "Get me this page", "scrape this URL", "what does this page say" - No mention of "website", "full site", "all pages", "crawl" **Crawl** (multi-page) — when user asks for: - "Crawl this website", "get all pages from", "full website content" - Mentions a domain broadly (not a specific path) - Wants multiple pages --- ## Scrape — Single Page Use `POST /v2/scrape`. Synchronous — result is returned immediately. ```bash curl --fail --silent --show-error \ --request POST \ --url "https://api.webcrawlerapi.com/v2/scrape" \ --header "Authorization: Bearer ${WEBCRAWLERAPI_API_KEY}" \ --header "Content-Type: application/json" \ --data '{ "url": "", "output_formats": ["markdown"] }' ``` ### Scrape Response The response contains `markdown` field directly — no polling needed: ```json { "success": true, "status": "done", "markdown": "## Page Title\n\nPage content...", "page_status_code": 200, "page_title": "Page Title" } ``` ### On success Output the markdown content directly to the user. No need to save to files for scrape. ### On failure If `success` is `false`, show the `error_code` and `error_message` to the user. --- ## Crawl — Full Website Use `POST /v1/crawl`. Asynchronous — returns a job ID, then poll for results. ### Step 1: Start the crawl ```bash curl --fail --silent --show-error \ --request POST \ --url "https://api.webcrawlerapi.com/v1/crawl" \ --header "Authorization: Bearer ${WEBCRAWLERAPI_API_KEY}" \ --header "Content-Type: application/json" \ --data '{ "url": "", "items_limit": 25, "output_formats": ["markdown"] }' ``` Response: ```json { "id": "" } ``` ### Step 2: Poll job status (background loop) Use a background Bash job to poll every 10 seconds until `status` is `done` or `error`: ```bash JOB_ID="" while true; do RESULT=$(curl --fail --silent --show-error \ --request GET \ --url "https://api.webcrawlerapi.com/v1/job/${JOB_ID}" \ --header "Authorization: Bearer ${WEBCRAWLERAPI_API_KEY}") STATUS=$(echo "$RESULT" | python3 -c "import sys,json; print(json.load(sys.stdin)['status'])") echo "Job status: $STATUS" if [ "$STATUS" = "done" ] || [ "$STATUS" = "error" ]; then echo "$RESULT" break fi sleep 10 done ``` ### Step 3: Download and save results When job is `done`, for each `job_item` with `status: done`: - Fetch the content from `markdown_content_url` - Save to `.webcrawlerapi//.md` ```bash mkdir -p ".webcrawlerapi/" # For each job_item, fetch markdown_content_url and save: curl --silent "" \ --output ".webcrawlerapi//.md" ``` Sanitize filenames: replace `://`, `/`, `?`, `#`, `:` with `_`. Trim leading underscores. ### Step 4: Report to user After saving, tell the user: - Total pages crawled - How many succeeded vs failed - Where files were saved: `.webcrawlerapi//` - List the saved files --- ## Notes - Default `items_limit` for crawl: 25 (ask user if they want more) - For scrape, just output the markdown — don't save to disk - For crawl, always save to `.webcrawlerapi/` directory in current working dir - If the job returns `error` status, show `last_error` from job items and the job-level error if present - Never hardcode the API key — always use `${WEBCRAWLERAPI_API_KEY}`

GetMarkdown

Description

Latest Changes

Install GetMarkdown with One Click