Skip to content

Add Inspect skill - #30

Merged
rahulbhadja merged 3 commits into
mainfrom
feat/inspect-skill
Oct 20, 2026
Merged

rahulbhadja merged 3 commits into
mainfrom
feat/inspect-skill

Conversation

@rahulbhadja

Copy link
Copy Markdown
Contributor

Adds an inspect skill so agents can use Hookdeck Inspect: create a free webhook test URL, wait for requests, and show what they sent.

  • New skills/inspect/ with SKILL.md and reference files (reading requests, sending test webhooks, signatures, forwarding)
  • The hookdeck router skill sends webhook-testing tasks to inspect
  • README, AGENTS.md, plugin files and changelog updated

Install: npx skills add hookdeck/agent-skills --skill inspect

@rahulbhadja
rahulbhadja requested a review from leggetter October 19, 2026 13:21
Comment thread skills/inspect/references/forwarding.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
Comment thread skills/inspect/SKILL.md Resolved
@rahulbhadja

Copy link
Copy Markdown
Contributor Author

@leggetter addressed the feedback.

@leggetter

Copy link
Copy Markdown
Collaborator

Review: accuracy, clarity and agent usefulness

How I reviewed it: I used the skill the way an agent would, against the live service. I created Sources, sent requests to them, waited for those requests, triggered the error paths, ran the samples.hookdeck.com snippet and ran cmd_hint against a local server. I also checked the claims against the Inspect source in hookdeck/core (clients/apps/inspect/).

Verdict: accurate overall and useful for an agent. Nearly every reply shape, field, limit, error code and UI label matched the live service. I'd ask for fixes 1–3 before merging; 4–6 are small corrections.

Suggested actions

  1. forwarding.md: say that the request path is appended to --path. Posting to ingest_url/webhooks and running cmd_hint with --path /webhooks delivered the request to /webhooks/webhooks locally.
  2. SKILL.md frontmatter: switch allowed-tools: WebFetch to Bash(curl:*) (optionally keeping WebFetch for llms.txt). As written, the frontmatter pre-approves the page-summarising fetch tool that the skill's Gotchas tell agents not to use, and leaves curl unapproved.
  3. Note that X's Source Type is TWITTER, and that types are uppercase constants. {"type":"X"} and {"type":"stripe"} both return 422.
  4. reading-requests.md: correct the body row. Form-encoded bodies come back parsed into an object, and binary bodies come back as null.
  5. Rename "Wait replies only" to "/requests replies". body_raw, body_base64 and already_arrived are also present on reads that don't wait.
  6. Limit the "follow the reply's note" rule, or fix the server: on a 404, the note says to GET read_url, which is the same missing Source and returns 404 again. The skill's own 404 guidance is correct.

Smaller additions: mention the capture lag and the real wall-clock time of a wait, cap how much body to paste into chat, say which ingest URL to give users when the CLI prints a console.hookdeck.com one, add a tiebreaker between event-gateway and inspect for local testing, and add an inspect scenario to scenarios.yaml.

Details: what was verified, and evidence for each suggestion

Verified against the live service and the source

  • Create (POST /): returns 201 with a Location header and exactly the documented fields (ingest_url, inspect_url, requests_url, type, name, cli_key, cmd_hint, note).
    • {"type":"STRIPE"} pins the type; an unsupported type returns 422 listing the valid types.
    • A bad name returns 422 with the pattern it must match.
  • Ingest reply: 200 with status, request_id, inspect_url, requests_url and note. For HEAD, the ID is in the x-hookdeck-request-id header.
  • Waiting (/requests?timeout=25):
    • next_url is always present, and more means more requests are ready now.
    • A timed-out wait returns timed_out, an empty requests, a resume whose r counter goes up each call, and read.
    • already_arrived behaves as described (arrivedBeforeWait in request-wait.ts).
  • Snapshot (?format=json): returns an src_… ID and the 10 newest requests, newest first, without body_raw, body_base64 or already_arrived, plus read_url. Sending Accept: application/json returns the same JSON.
  • Body download (body_url): returns the exact bytes as application/octet-stream.
  • Errors and limits:
    • A bad timeout returns 400 with read_url; an unknown ID returns 404.
    • Rate limit: 60 calls a minute per visitor IP per Source, with Retry-After: 60 (agent-rate-limit.server.ts).
    • Body limits: 1 MB of text, 64 KB of binary, 2 MB of bodies per reply (raw-body.ts).
  • Platforms that check the URL first: Zoom, X, eBay, LinkedIn, Xero, Smartcar and Exact Online exactly match URL_CHECK_BY_SOURCE_TYPE in src/providers/provider-setup.ts. Creating a ZOOM or AIRWALLEX Source through the API without a secret works, and its requests come back with verified: false and rejection_cause: null, as the skill says.
  • UI labels: "Use with a specific platform", "Source URLs" and Forward → Localhost all exist in the UI.
  • samples.hookdeck.com: providers.json has .stripe.latest_version and lists 155 providers ("150+" holds). The per-version file is keyed by event name with headers and body. The Stripe snippet in send-test-webhooks.md runs as written.
  • CLIs:
    • hookdeck-cli 3.1.0 has --cli-key and --path. brew install hookdeck and npm install hookdeck-cli -g match the CLI README.
    • npx skills add … --list exists.
    • cmd_hint (with [port] replaced) forwarded a request to a local server.

Evidence for each suggested action

  1. Path doubling. Running cmd_hint with --path /webhooks and posting to {ingest_url}/webhooks, the CLI logged POST http://localhost:8765/webhooks/webhooks and the local server received /webhooks/webhooks. "Change --path to the path the local handler listens on" only holds when the provider posts to the bare ingest_url. The SKILL.md example request also shows "path": "/webhooks", which nudges agents toward that setup.
  2. allowed-tools. WebFetch is the right choice for the docs-oriented event-gateway and outpost skills. This skill reads payload data, and its own Gotchas say summarisers can drop or reword fields.
  3. TWITTER. The Source Type list has TWITTER and no X. Type matching is case-sensitive.
  4. body. A PUT with curl's default form content type came back with body as an object. A binary POST came back with body: null, body_raw: null and body_base64 set.
  5. Non-waiting reads. GET /requests?limit=2 with no timeout still returned body_raw and already_arrived on each request.
  6. 404 loop. GET /{unknown}/requests returns {"error":"not_found","note":"… To start again, GET read_url.","read_url":"…/{unknown}/requests"}, and following it returns 404 again. The note comes from the not_found branch of sourceRequestsResponse in src/requests/agent-api.server.ts.

Smaller additions

  • Capture lag. A request sent about 1 second before a wait took about 8 seconds to come back. An agent that sends a test and then reads the snapshot or a non-waiting /requests can wrongly conclude nothing arrived. Suggested line: "After sending a test, use the wait call; capture takes a few seconds."
  • Wall-clock time. timeout=5 took 11 seconds end to end, so a timeout=25 wait can run about 31 seconds, despite the note saying each call "returns within 25 seconds". Suggest telling agents to allow about 35 seconds, so a curl -m 25 doesn't cut the wait short.
  • Reporting large bodies. "Show the full body in a code block" is right for typical webhooks, but bodies can be up to 1 MB. Suggest a cap, for example over ~20 KB, show the start and point to body_url.
  • The CLI's printed URL. listen prints Requests to → https://console.hookdeck.com/e/{id}. Tell agents to keep giving users the Inspect ingest_url.
  • Router overlap. The hookdeck router sends "Test and debug webhooks on localhost" to event-gateway, but Inspect also covers forwarding to localhost. Suggested tiebreaker: no account or just need a URL → inspect; building a lasting setup → event-gateway.
  • Test coverage. scenarios.yaml only recognises hookdeck | event-gateway | outpost. Inspect is the easiest skill in the repo to test live (create → send → wait → report).
  • Optional. The OpenAPI says replaying the session cookie from the first create puts more Sources in the same workspace. The skill doesn't need this for v1.

Agent usefulness

Inspect's llms.txt and the note fields on each reply already guide agents well. The skill's extra value is in three places:

  • Routing: it sends "give me a webhook URL" to Inspect.
  • Traps: it warns about reading the ingest URL (which captures a new request), background watchers that die when a headless session ends, and reporting a request with already_arrived: true instead of waiting for a newer one.
  • Recipes: realistic provider samples, checking signatures against the exact body, and platforms that need their secret saved on the Source first.

All of these held up in use, and SKILL.md keeps the core loop short while the reference files cover the side tasks.


Generated by Claude Code

@rahulbhadja
rahulbhadja merged commit efe1e08 into main Oct 20, 2026
8 checks passed
@rahulbhadja
rahulbhadja deleted the feat/inspect-skill branch October 20, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants