--- name: web-autopilot description: > Record any web app operation once, AI turns it into a reusable automation tool. Use when: (1) automating repetitive tasks on any web application (reports, submissions, data extraction), (2) creating no-code automation for any logged-in web app, (3) building callable tools from recorded browser sessions. Supports REST, GraphQL, form submissions, file uploads, any login method. Task types: query/export (data extraction) and submit (form submissions like expense reports, travel requests, payment requests). --- # Web Autopilot > Record once in any web app, let AI handle it from now on. ## Overview Record → Analyze → **Confirm Fields** → Generate → Test → Register as Tool ``` 🎬 Record User performs the workflow once in a real browser (after login) 🔍 Analyze AI analyzes network traffic, classifies fixed/dynamic/session fields ✅ Confirm Fields [Required for submit tasks] User confirms field classifications 📝 Generate Generates reusable TS script + field mapping 🧪 Test Iterative test loop, up to 5 rounds of auto-fix 🔧 Register Register as an OpenClaw tool for direct invocation ``` ## Task Types ### 📊 Query / Export Data extraction and report generation. Scripts run and output results automatically — no manual intervention needed. Examples: pull sales reports, extract project data, export revenue details ### 📝 Submit Submit forms such as expense reports, travel requests, payment requests, etc. **Each run requires dynamic parameters.** Examples: submit travel request, submit expense report, submit payment request The key challenge for submit tasks: **correctly distinguishing which fields are fixed vs. which change every time**, and confirming with the user before generating the script. ## Skill Directory ``` ~/.openclaw/rpa/ ├── recordings//recording.json ├── tasks// │ ├── task-meta.json │ ├── run.ts │ └── field-mapping.json └── sessions/.session.json ``` Skill scripts: `/opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/` --- ## Commands ### 1. `record` — Record a workflow Ask user: task name, login URL or app URL. ```bash cd /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot # Option A: Start from login page (SSO, OAuth, username/password, etc.) npx ts-node scripts/record.ts --name "my-task" --sso-url "https://login.example.com" # Option B: Start directly from app (if already logged in or no login needed) npx ts-node scripts/record.ts --name "my-task" --app-url "https://app.example.com" ``` Run in PTY mode (pty: true, background: true). User operates browser, types "done" when finished. Note: `--sso-url` is a legacy parameter name; it works for **any** login URL (SSO, OAuth, plain login page, etc.). ### 2. `analyze` — Analyze the recording (AI does this) Read recording.json, separate login traffic from business traffic, identify core APIs. **Key steps:** 1. Read `~/.openclaw/rpa/recordings//summary.txt` for overview 2. Parse recording.json to extract all API calls to app domain 3. For each POST/PUT/PATCH with meaningful body: - Classify fields: FIXED / DYNAMIC / SESSION / RELATIONAL - Detect protocol: rest-json / graphql / form-urlencoded / multipart 4. Map the complete API sequence (prerequisites → main operation → follow-ups) 5. **Analyze ALL response fields and create field-mapping.json with human-readable labels** 6. Create task-meta.json 7. **[Submit tasks] After analysis, present the field classification confirmation table to the user (see below)** #### Field Classification | Type | Meaning | Handling | |------|---------|----------| | **FIXED** | Same value every submission (approval flow ID, company entity, currency, expense type enums…) | Hardcoded in script | | **DYNAMIC** | Different each submission (amount, date, reason, attachment path…) | Becomes CLI `--parameter` | | **SESSION** | Auth tokens/cookies, auto-managed | Injected by session.ts | | **RELATIONAL** | Requires a lookup from another API to get the ID (e.g., project ID, person ID…) | Auto-queried in script, or exposed as DYNAMIC parameter | #### Field Analysis Rules (MANDATORY) **Every field must have a human-readable label.** Including system-generated field names. Inference priority: 1. **Data value type**: timestamp (10^12-13) / monetary amount (contextual) / enum (fixed values) / URL / JSON object 2. **Field name pattern**: *time/*date/*_at → datetime | *amount/*price/*cost → monetary | *id/*_key → ID | *status/*state → status 3. **Business context**: infer from related fields, API endpoint names 4. If uncertain → annotate as `(unknown meaning: sample value)` #### Field Confirmation Step (MANDATORY for Submit tasks) After analysis, **you must present the following confirmation table to the user and wait for confirmation before generating the script**: ``` 📋 Field Classification Confirmation — ✅ FIXED (hardcoded): - approvalFlowId: "xxx" → Approval Flow ID - companyId: "yyy" → Company Entity - currency: "CNY" → Currency 🔄 DYNAMIC (passed as parameters each run): - amount → Amount (example: --amount 1500) - startDate → Start Date (example: --startDate 2026-03-10) - endDate → End Date (example: --endDate 2026-03-12) - destination → Destination (example: --destination "New York") - reason → Reason (example: --reason "Client visit") - attachments → Attachment path (example: --attachments ~/Desktop/receipt.jpg) 🔗 RELATIONAL (auto-queried): - projectId → Project ID (auto-looked up by project name, --projectName "Project X") ❓ Needs confirmation (AI uncertain): - field_abc123 → Unknown meaning (recorded value: "0"), suggest: FIXED("0") or DYNAMIC? Please confirm the above classification or indicate any fields that need adjustment. ``` **Only proceed to the Generate step after user confirmation.** #### CSV Export Rules (MANDATORY) - **Keep ALL fields**, including hidden fields, dynamic fields, system fields — never crop - Field order: preserve original order from data, **never sort** (sorting causes column misalignment) - JSON/object fields → convert to JSON string for storage - Use `csv.writer` + proper quoting to handle JSON fields containing commas ### 3. `generate` — Generate the task script **Pre-generation checklist (Query/Export tasks)**: - ✅ All fields are in field-mapping.json - ✅ All fields have human-readable labels - ✅ CSV export uses field-mapping.json for column headers - ✅ Field order preserves original order **Pre-generation checklist (Submit tasks)**: - ✅ User has confirmed field classification (FIXED / DYNAMIC / RELATIONAL) - ✅ All DYNAMIC fields converted to CLI parameters (with type, example value, required/optional) - ✅ RELATIONAL fields have auto-query logic or corresponding parameters - ✅ Script has `--dry-run` mode (prints request body without submitting, for testing) - ✅ Script outputs submission result (success/failure + document number/link) **Submit task invocation example** (written to task-meta.json `usage` field after generation): ```bash # Preview (no actual submission) npx ts-node run.ts --dry-run --amount 1500 --startDate 2026-03-10 ... # Submit for real npx ts-node run.ts --amount 1500 --startDate 2026-03-10 --destination "New York" --reason "Client visit" ``` ### 4. `test` — Iterative test loop (max 5 rounds) Run script → check output → if error: diagnose → fix → repeat. | Error | Cause | Fix | |-------|-------|-----| | 401/403 | Session expired / wrong auth | Re-check auth headers, re-login | | 400 | Wrong field name/type | Compare with recording | | 404 | Wrong URL | Check URL exactly | | JSON parse error | Response is HTML | Log resp.raw | ### 5. `run` — Execute a registered task ```bash npx ts-node ~/.openclaw/rpa/tasks//run.ts --param1 value1 ``` ### 6. `list` — List all tasks ```bash npx ts-node /opt/homebrew/lib/node_modules/openclaw/skills/web-autopilot/scripts/run-task.ts --list ``` --- ## Session & Credential Management ### Session (Cookie/Token Storage) Sessions are cookie-based and work with **any login method**: - SSO (OIDC, SAML, CAS, etc.) - OAuth / OAuth2 - Username + password forms - Any browser-based authentication Session files: `~/.openclaw/rpa/sessions/.session.json` ### Credentials (Encrypted Storage) Login credentials are stored **encrypted** (AES-256-GCM) in a separate file — **never stored in plaintext**. File: `~/.openclaw/rpa/credentials.enc` - Encryption key = machine identity (hostname+username) + optional `RPA_CREDENTIAL_KEY` env var - File permissions: 0600 (owner only) - Supports automatic extraction and encrypted storage from recording.json ```bash # Manage credentials npx ts-node scripts/utils/credentials.ts list # List saved domains + usernames npx ts-node scripts/utils/credentials.ts save # Save manually npx ts-node scripts/utils/credentials.ts delete # Delete npx ts-node scripts/utils/credentials.ts extract # Extract from recording ``` ### Auto-Login Flow When a session expires, the auto-login flow kicks in: ``` 1. Read encrypted credentials for the target domain from credentials.enc 2. Select login strategy based on loginFlow.type 3. Launch browser (headless if credentials exist, headed if not) 4. Execute login steps → follow redirects → reach target app 5. Capture cookies/tokens → save new session 6. If all else fails → open headed browser for manual login (fallback) ``` #### Login Flow Types When generating scripts, **you must identify the login type from the recording** and write it to the `loginFlow` field in `task-meta.json`: | type | Scenario | Auto-login method | Example | |------|----------|-------------------|---------| | **`api`** | SSO/app provides a REST login endpoint, single POST completes auth | Call API directly → follow redirects | Enterprise SSO (`POST /api/sso/login`) | | **`form`** | Single-page login form (username + password on same page) | Fill form fields → click submit | Common admin dashboards | | **`multi-step`** | Multi-step login (email → next page → password → next page → possible 2FA) | Execute step sequence | Google, Microsoft, Okta | | **`manual-only`** | Has CAPTCHA/2FA/risk control, cannot be fully automated | Open headed browser directly | Banking systems, strong CAPTCHA sites | #### loginFlow Schema (task-meta.json) ```jsonc { "loginFlow": { "type": "api", // api | form | multi-step | manual-only "loginUrl": "https://sso.example.com", "loginDomain": "sso.example.com", "appDomain": "app.example.com", // ── type=api specific fields ── "loginApiPath": "/api/sso/login", "authType": "passwordAuth", // Optional, auth type field in API body "appId": "1234567890", // Optional, SSO portal app ID (for forward redirect) "appForwardUrl": "...", // Optional, direct redirect URL (alternative to appId) // ── type=form specific fields ── "usernameSelector": "input[name='email']", // Optional, custom selectors "passwordSelector": "input[type='password']", "submitSelector": "button[type='submit']", // ── type=multi-step specific fields ── "steps": [ { "action": "fill", "selector": "input[type=email]", "field": "username" }, { "action": "click", "selector": "#identifierNext" }, { "action": "wait", "selector": "input[type=password]", "timeoutMs": 5000 }, { "action": "fill", "selector": "input[type=password]", "field": "password" }, { "action": "click", "selector": "#passwordNext" } ], // ── Common fields ── "successIndicator": "url_contains:app.example.com", // Condition to detect successful login "postLoginWaitMs": 3000 // Wait time after login success (for cookies to settle) } } ``` #### Login Identification Guide for Analyze Step (MANDATORY) During the analyze step, you **must** complete the following login analysis: 1. **Extract credentials** → `credentials.ts extract ` (auto-detects username/password in POST body) 2. **Identify login type** → Inspect the login flow in the recording: - Has a clear `POST login/auth API` → type = `api` - Has form fill actions (password type input) on the same page → type = `form` - Has multiple form fill actions with page navigations in between → type = `multi-step` - Has CAPTCHA image requests or reCAPTCHA scripts → type = `manual-only` 3. **Document the SSO → app redirect path**: - Does it use an appId forward? - Does it use a redirect_uri callback? - Where is the token — in URL query / response body / cookie? 4. **Write loginFlow** → Write all fields to `task-meta.json` 5. **Sanitize** → Replace passwords in recording.json with `[REDACTED]` **⚠️ If credentials.ts extract cannot extract credentials (e.g., Google multi-step login), prompt the user to save credentials manually:** ```bash npx ts-node scripts/utils/credentials.ts save accounts.google.com user@gmail.com 'password' ``` #### Login Code Templates for Script Generation Choose the auto-login implementation based on `loginFlow.type`: **type=api (REST API login):** ```typescript // API login → follow redirects → navigate to app const resp = await page.evaluate(async (p) => { const r = await fetch(p.url, { method: 'POST', headers: {'Content-Type':'application/json'}, body: JSON.stringify(p.body), credentials: 'include' }); return { status: r.status, ok: r.ok }; }, { url: loginApiUrl, body: { authType, credential: { username, password } } }); ``` **type=form:** ```typescript await page.fill(loginFlow.usernameSelector || 'input[name="username"]', cred.username); await page.fill(loginFlow.passwordSelector || 'input[type="password"]', cred.password); await page.click(loginFlow.submitSelector || 'button[type="submit"]'); ``` **type=multi-step:** ```typescript for (const step of loginFlow.steps) { if (step.action === 'fill') { const value = step.field === 'username' ? cred.username : cred.password; await page.fill(step.selector, value); } else if (step.action === 'click') { await page.click(step.selector); } else if (step.action === 'wait') { await page.waitForSelector(step.selector, { timeout: step.timeoutMs || 10000 }); } } ``` **type=manual-only:** ```typescript // Open headed browser, wait for user to complete login manually const browser = await pw.chromium.launch({ headless: false }); // ... wait for successIndicator ``` **task-meta.json loginFlow example (SSO → enterprise app):** ```json { "loginFlow": { "type": "api", "loginUrl": "https://sso.example.com", "loginDomain": "sso.example.com", "loginApiPath": "/api/sso/login", "authType": "passwordAuth", "appId": "1234567890", "appDomain": "app.example.com", "successIndicator": "url_contains:app.example.com" } } ``` ### ⚠️ Security Rules (MANDATORY) 1. **Passwords in recording.json must be sanitized immediately after analysis** (replace with `[REDACTED]`) 2. **credentials.enc is an encrypted binary file** — do not attempt to read or edit directly 3. **credentials.enc and sessions/ directory must never be committed to version control or shared** 4. **Skill packages (.skill) must not contain any credentials, sessions, or recording data** 5. **Generated task scripts (run.ts) must never hardcode any passwords** --- ## Known Issues & Lessons Learned ### 🔐 Login flow must match app — don't assume one-size-fits-all - Current implemented scripts use `type=api` mode (enterprise SSO → business app) - **Each new app recording must re-identify the login type** — do not reuse login logic from old scripts - Google/Microsoft multi-step logins require `type=multi-step` + steps sequence - Sites with CAPTCHA/2FA can only use `type=manual-only` - Inlining login logic into run.ts (rather than importing external login.ts) is more stable due to Node ESM/CJS compatibility issues ### ⚠️ Node v25 ESM compatibility - Node v25 defaults to ESM, `require()` is unavailable - Solution: place `tsconfig.json` in the task directory to force `"module": "commonjs"` - Dependencies like Playwright need full-path require: `require('/opt/.../node_modules/playwright')` - Cross-directory .ts imports under ts-node are unstable — recommend inlining critical logic into run.ts ### ⚠️ Multi-tab traffic capture (fixed) Some login flows or apps open new tabs. Recorder uses `context.on('request/response')` to capture ALL tabs. ### 📋 CSV must include ALL fields with human-readable labels - Never crop fields — include everything from the API response - System-generated field names (e.g. `field_*`, `attr_*`, `custom_*`) must be analyzed from sample data - Create field-mapping.json for every task - Field order: preserve original order from data, **never sort** - Use proper CSV quoting to handle JSON fields with commas ### 📝 Submit tasks: always confirm field classification before generating - Never skip the field confirmation step — wrong FIXED/DYNAMIC split breaks every future submission - Fields that look fixed (e.g. a hardcoded project ID) might actually need to be dynamic in real use - Always include `--dry-run` in generated scripts so users can verify the request body before committing - RELATIONAL fields (e.g. approver ID looked up by name) should be auto-resolved in script, exposed as human-readable params --- ## File Locations | Item | Path | |------|------| | Recorder | `scripts/record.ts` | | Task runner | `scripts/run-task.ts` | | Session utility | `scripts/utils/session.ts` | | Login helper | `scripts/utils/login.ts` | | Recordings | `~/.openclaw/rpa/recordings//` | | Generated tasks | `~/.openclaw/rpa/tasks//` | | Sessions | `~/.openclaw/rpa/sessions/.session.json` |

Web Autopilot

Description

Latest Changes

Install Web Autopilot with One Click