diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a24c2126..21d8145ef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ This changelog was generated from the repository Git history and release tags. V ### Fixed - Vision mode can attach more than two files in one task. Each file goes through its own upload hand-off, and a run used to stop after two hand-offs even when both uploads worked: the third file, such as the last zip of a three-asset GitHub release, was refused with "file uploads are not available in vision mode", and the agent asked you to turn Vision mode off. Now only hand-offs that end without attaching a file count toward the limit of two. When that limit is reached, the agent says the earlier attempts attached nothing instead of blaming vision mode. Chrome and Firefox. +- Vision mode attaches each file with the first upload attempt. After a click on an upload control hands the upload to the element tools, the agent uses the file input the click named, but WebBrain's page check refused that first attempt when the input was hidden, as on a GitHub release form, because it had not seen that input itself. The agent then had to list the page's file inputs and send the same upload again, two wasted steps per file. The hand-off now records that exact input before the agent's next step, so the first attempt attaches the file. A file input the agent guessed is still refused. Chrome and Firefox. ## [46.1.0] - 2026-11-05 diff --git a/docs/page-feedback-monitor.md b/docs/page-feedback-monitor.md index 0e96736db..b1ddb6e0b 100644 --- a/docs/page-feedback-monitor.md +++ b/docs/page-feedback-monitor.md @@ -308,7 +308,12 @@ Steer or grant authorization. Enter and Alt+Enter retain their existing behavior attachment. A node the page re-mounts later under the same selector was never listed and keeps the refusal until a fresh inventory lists it, as does any other selector. A blocked native chooser names its input with the - selector the inventory uses. + selector the inventory uses. In a vision run that blocked `click_at` hands + the next steps to the element tools (`docs/vision-mode.md`), and its result + is the model's observation of the input: when the handoff starts, the + monitor keeps the node the result's selector names under a fresh token + (`page_monitor_record_upload_targets`), an inventory of the blocked inputs + only, so the first `upload_file` with that selector binds as above. - Native `summary` controls and authored click/tab-focus targets use the same observed owner and exact action footprint as native and ARIA buttons. Changed handler identity, disclosure state, target/owner identity and user intervention diff --git a/docs/vision-mode.md b/docs/vision-mode.md index d4086259f..2b0629136 100644 --- a/docs/vision-mode.md +++ b/docs/vision-mode.md @@ -17,7 +17,7 @@ A run decides once, when it starts (`_beginVisionRun` snapshots both settings), Three things end vision mode inside a run for good, all proofs that only the accessibility tree can give. A site workflow whose completion is proved against the app-owned element inventory (`requiresLedger` in `adapter-workflow.js`, form and publish jobs of some adapters) cannot be certified from screenshots: once the planner or a live adoption selects one, `resolveVisionMode` answers `workflow_needs_element_inventory`. A request for a whole conversation (a thread to summarize in full, Gmail's `read-complete-thread` job) is answered only after trusted coverage through its last page with every message expanded (`read-completeness.js`): once that requirement is armed, it answers `complete_thread_read`. WhatsApp Web and Telegram's web app confirm who an open chat is with, and bind the chat they read and send in, through the element inventory (a contact check made with `click_ax`, chat reads and sends bound to refs): on their pages it answers `native_chat_needs_element_inventory` (`_visionNativeChatPage`, from the tab's last address). Either way the next step gets the tree tools and the tree prompt (the loop rebuilds the system prompt when the vision state flips). -A file upload leaves vision mode for a few steps only. A native file chooser opens outside the page, where screenshots cannot show or operate it, so a `click_at` that lands on a file input is stopped by the file-chooser guard (Chrome's CDP guard, Firefox's content-script guard, on every dispatch path). The batch pipeline then arms an upload handoff (`_visionNoteUploadHandoff` in `vision-runtime.js`, kept in the run's snapshot) and rewrites the result: the next steps use the element tools, `upload_file` with the blocked input's `selector` (the compact tier's discovery call instead), and screenshots come back after the upload; another file starts with another click on the control. Changes take effect only at step boundaries (`_visionUploadHandoffTick`, at the top of each step, before adapter notes, cookie banners and the step's tools), so the rest of the click's batch and its screen update stay in vision mode, and the upload's batch stays on the tree path. While the handoff is active `resolveVisionMode` answers `file_upload_needs_element_inventory`. It belongs to the page the tab shows when it starts (the click that armed it may itself push a route), and ends after a successful `upload_file` (the file is in the input in any build's success shape, not a discovery listing or an unconfirmed outcome), when the tab's address changes (the fragment aside), or after five decisions without an upload. Each file takes its own handoff, and a handoff that attached its file does not count against the run (a release with three assets takes three). Once two handoffs have ended without an upload (expired, or the page was left), a further chooser click is not handed off, and its result says why (`exhausted`: the earlier attempts attached nothing) rather than that uploads are unavailable in vision mode, which would be false after an upload that worked. Another chooser in the batch that armed a handoff is reported as handed off too: the armed handoff covers its input. The trace notes each handoff as `armed`, `ended` (`uploaded`, `navigated`, `expired`) or `refused` (`exhausted`, `unreachable`, `no_source`). Nested runs keep their own handoff with their snapshot. +A file upload leaves vision mode for a few steps only. A native file chooser opens outside the page, where screenshots cannot show or operate it, so a `click_at` that lands on a file input is stopped by the file-chooser guard (Chrome's CDP guard, Firefox's content-script guard, on every dispatch path). The batch pipeline then arms an upload handoff (`_visionNoteUploadHandoff` in `vision-runtime.js`, kept in the run's snapshot) and rewrites the result: the next steps use the element tools, `upload_file` with the blocked input's `selector` (the compact tier's discovery call instead), and screenshots come back after the upload; another file starts with another click on the control. Changes take effect only at step boundaries (`_visionUploadHandoffTick`, at the top of each step, before adapter notes, cookie banners and the step's tools), so the rest of the click's batch and its screen update stay in vision mode, and the upload's batch stays on the tree path. The click's result is how the model observed the input, which has no ref: when the handoff starts, before its first decision, the page monitor keeps the node each blocked input's `selector` names under a fresh file-inventory token (`_pageFeedbackObserveUploadTargets` in `page-feedback.js`, the monitor's `page_monitor_record_upload_targets`; not on the compact tier, whose `upload_file` takes no selector). The first `upload_file` with that selector is then covered by the step's page capture, or sent as an inventoried input with a `page-feedback-target-binding` note, instead of being refused as `target_uncovered` until the model had read `get_interactive_elements` and sent the same call again (two wasted steps per file). Any other selector, and a node the page renders later under the same selector, keep that refusal. While the handoff is active `resolveVisionMode` answers `file_upload_needs_element_inventory`. It belongs to the page the tab shows when it starts (the click that armed it may itself push a route), and ends after a successful `upload_file` (the file is in the input in any build's success shape, not a discovery listing or an unconfirmed outcome), when the tab's address changes (the fragment aside), or after five decisions without an upload. Each file takes its own handoff, and a handoff that attached its file does not count against the run (a release with three assets takes three). Once two handoffs have ended without an upload (expired, or the page was left), a further chooser click is not handed off, and its result says why (`exhausted`: the earlier attempts attached nothing) rather than that uploads are unavailable in vision mode, which would be false after an upload that worked. Another chooser in the batch that armed a handoff is reported as handed off too: the armed handoff covers its input. The trace notes each handoff as `armed`, `ended` (`uploaded`, `navigated`, `expired`) or `refused` (`exhausted`, `unreachable`, `no_source`). Nested runs keep their own handoff with their snapshot. No handoff is armed when the upload could not succeed on the tree path either, and the click result says why instead: when only Chrome's protocol guard saw the chooser (`protocolOnly`: an input in a frame or a closed shadow root, which `upload_file` cannot reach), and on Chrome's compact tier when the run's message has no attachment (`_visionUploadSourceAvailable`; compact `upload_file` attaches only those). @@ -183,9 +183,10 @@ npm run test:vision-mode ``` - `test/vision-mode.mjs`: sizing, profiles, token formulas, diffs, micro changes, sessions, translation, tool surfaces of every tier in both builds, permission classification, prompt, the page-feedback policy for vision calls and the monitor's vision rebase. -- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation (and a `click_at` in a turn that does not offer it, such as a completion recovery turn, refused before it can run as a click), patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled. -- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch. +- `test/vision-mode-agent.mjs`: the real run loop in both builds (streaming and not) with a scripted vision model and a fake screen: first frame, tool surface, click translation (and a `click_at` in a turn that does not offer it, such as a completion recovery turn, refused before it can run as a click), patches, "no visible change", new frames on navigation, pruning, stale and moved points, native tools (the page monitor prepared with `hover_at`'s and `drag_at`'s CSS points, a wheel with none), fallback to the tree; a refused click stopping its batch, tool and encoding failures, a failed first capture in a later run, a model switch between runs, drag/wheel/repeated-click refusals, `drag_at` on a protected messaging page; the settle wait on a caret or clock versus real motion; page churn before a decision (no tree read, nothing re-sent while the screen stays as seen, a change after the update shown, a person's input always reported); site notes in vision terms; a ledger workflow switching the run to the tree path; a drag after an earlier action of its batch re-checked at both ends; a window resized while the model decides (a fresh full frame first) and one page feedback has not reported yet (the batch's points refused); a `take_screenshot` and `done` in one batch after an update that could not be captured, and a `drag_at` finished in its own batch, both refused as unverified; a zoom after an earlier action of its batch usable beside that batch's patch except where the screen changed after it; in Chrome, a click that opened no tab costing no poll, tabs opened at once or later being rerouted, and a late-tab watch staying armed past an update and rerouting a tab opened while the screen settled; on a hidden file input no page observation covers, the first `upload_file` of a handoff with the blocked click's selector passing the page-action binding check, and a guessed selector refused. +- `test/vision-mode-guards.mjs`: the guards in a vision run, both builds: a vision click deleting the item the request names is bound to the control read at its point (the same point outside a vision run stays unbound), the deletion card, a changed item, a forbidden item and Enter keep their rules, and an authorized deletion is read again at dispatch, sending nothing when the control moved away; loop detection by screen (repeats after a change are progress, on an unchanged screen they stop), vision loop messages, native dropdowns, arrow keys and clicks without DOM-snapshot verdicts, page-focus churn, the delivery checkpoint, one screen update per batch however it ended; whole-thread reads and WhatsApp/Telegram on the tree path, the Gmail counter, observations and refusal wording after an action (a requested frame verifying nothing until captured, a drag opening verification debt), on-screen status texts, the opening frame as evidence, the CAPTCHA root read and the toolbar check keeping the vision dispatch; an upload handoff observing the inputs its batch blocked for the page-action binding check (the first `upload_file` attaches with a note, Strict checks refuse it as for an inventory, a guessed selector, a re-mounted input and another document's answer stay refused, the compact tier observes nothing). - `test/quality/vision-mode-delete-mv3.mjs` (`--vague`): the installed MV3 extension deleting a note with `click_at` on its Delete button and on the site's own confirmation; a request that names the note asks nothing, "delete it" asks once on WebBrain's card, and exactly that note is deleted once. +- `test/quality/vision-mode-upload-mv3.mjs`: the installed MV3 extension clicking an upload control with `click_at`: the file-chooser guard stops the chooser, the next step attaches the user's clip with one `upload_file` and the step after it is back on a fresh frame, both for a visible file input and for a release form whose hidden binary input sits on a page too deep for the page observation to certify (the clip reaches that input, not the Markdown one). - `test/quality/vision-mode-repeat-mv3.mjs`: the installed MV3 extension pressing a "+" button ten times at the same point (its counter is drawn on a canvas, so the DOM text never changes): all ten presses go out, and a button that does nothing is then stopped by the loop guard, in vision terms. - `test/monitor-vision-wheel.mjs`: the real page monitor in Chromium: two wheel notches fenced before their events arrive stay the agent's (a third, unfenced one is a person's), page churn does not void a prepared wheel, a person's click does. - `test/vision-pipeline-browser.mjs`: Chromium at DPR 1/2/3 on a scrolled page: planned size, exact mapping, trusted clicks, diff regions, WebP vs JPEG; the Firefox content input path (caret placement, word and line selection, pointer transitions, wheel routing, HTML5 drag and drop, drags in a same-origin frame, native range inputs (dragged, and clicked on the track), text selection, disabled controls, the file-chooser guard, CSS `:hover` revealed under the pointer (a CSS-only menu, a selector listed beside a `:hover` keeping its own cascade, a later rule of equal specificity still winning, a rule inside `@media`, a shadow root's `:host(:hover)`, a rule added later) and ended by a person's own pointer with the page's selectors put back, pointer capture taken on the press: a thumb with its handlers on itself, a track that captures its thumb, a control that drops the implicit capture, run as from the page and as from a Firefox content script; clicks and drags in a scaled frame and in a frame inside a scaled ancestor, a rotated frame and another origin's frame refused); the DOM probe and the Chrome click submit probe in a scaled, bordered frame; the DOM probe on buttons, labelled text fields, focused number fields, selects, link cards, sliders and same-/cross-origin frames; the click submit probe at a vision point inside open shadow roots and on plain content, in both builds. diff --git a/src/chrome/src/agent/guard-policy.js b/src/chrome/src/agent/guard-policy.js index fe3fdbba5..ca9853443 100644 --- a/src/chrome/src/agent/guard-policy.js +++ b/src/chrome/src/agent/guard-policy.js @@ -328,7 +328,8 @@ export const GUARDS = Object.freeze({ "page-feedback-dispatch-fence": { class: "host_check", family: "tool", decides: "mixed", onDecided: "block", onCannotDecide: "block", summary: "Transport-boundary fence (beforePageAgentDispatch, the content page monitor, the execute_js gate, the trusted Enter path): human input or feedback during preparation, a changed document mapping, focus moved into a child frame, an unprepared or rejecting monitor vetoes the dispatch; after input…" }, "page-feedback-target-binding": { class: "quality", family: "tool", decides: "mixed", onDecided: "advise", onCannotDecide: "advise", - summary: "The selector/ref of a state-changing call cannot be bound to the observed action certificate (target_ambiguous, target_unresolved, target_uncovered, capture budget exhausted) or its original owner context was never certified: nothing is dispatched and a bounded fresh capture is queued for the next…" }, + summary: "The selector/ref of a state-changing call cannot be bound to the observed action certificate (target_ambiguous, target_unresolved, target_uncovered, capture budget exhausted) or its original owner context was never certified: nothing is dispatched and a bounded fresh capture is queued for the next…", + mustNotBlock: "test/vision-mode-agent.mjs#the first upload_file of a handoff, with the blocked click's selector, attaches past the page-action binding check" }, "pdf-viewer-dom-tools": { class: "host_check", family: "tool", decides: "knows", onDecided: "block", onCannotDecide: "block", summary: "A DOM/action tool on Chrome's built-in PDF viewer (read_page is redirected to read_pdf)." }, "preparation-timeout-generic": { class: "quality", family: "tool", decides: "no-evidence", onDecided: "advise", onCannotDecide: "advise", diff --git a/src/chrome/src/agent/page-feedback.js b/src/chrome/src/agent/page-feedback.js index a66ad5875..8f6b13a74 100644 --- a/src/chrome/src/agent/page-feedback.js +++ b/src/chrome/src/agent/page-feedback.js @@ -1501,12 +1501,46 @@ export const pageFeedbackMethods = { const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector && run.actionBindingInventoried === true && uploadSelectorObserved(run, args.selector); if (!observedRef && !observedUpload) return false; - return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload - ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.' - : 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.', + const handoffUpload = observedUpload && run.uploadObservationScope?.source === 'upload_handoff'; + return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', !observedUpload + ? 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.' + : handoffUpload + ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input the blocked file-chooser click named without that check. Read the page to confirm the file landed where you intended.' + : 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.', { tabId, decided: false }); }, + // A vision run's upload handoff (vision-runtime.js): the click_at the + // file-chooser guard stopped told the model to call upload_file with the + // blocked input's selector. That result is the model's observation of the + // input, as get_interactive_elements is on the tree path, so before the + // first decision of the handoff the page monitor keeps the node each + // selector names, and the run's upload scope holds exactly those + // selectors. The first upload_file with one of them is then covered by the + // capture (uploadTargets) or sent as an inventoried input above, instead of + // being refused until the model reads an inventory and sends the same call + // again. Any other selector, and a node the page renders later under the + // same selector, keep their answer. Returns whether the scope was set. + async _pageFeedbackObserveUploadTargets(tabId, selectors = []) { + const run = this._pageFeedbackRuns?.get(tabId); + const documentToken = run?.frames.get(0)?.token, url = run?.url; + const wanted = [...new Set((Array.isArray(selectors) ? selectors : []).filter(selector => + typeof selector === 'string' && selector.length > 0 && selector.length <= 4096))].slice(0, 8); + if (!run || !documentToken || !url || !wanted.length) return false; + const signal = this._runAbortSignal?.(tabId) || null; + const inventoryToken = token(); + let response = null; + try { + response = await untilAborted(apiFor().tabs.sendMessage(tabId, { target: 'content', action: 'page_monitor_record_upload_targets', + params: { runToken: run.token, token: inventoryToken, selectors: wanted } }, { frameId: 0 }), signal); + } catch { this._throwIfAborted?.(signal); return false; } + const recorded = Array.isArray(response?.recorded) ? wanted.filter(selector => response.recorded.includes(selector)) : []; + if (!recorded.length || response.runToken !== run.token || response.documentToken !== documentToken + || this._pageFeedbackRuns?.get(tabId) !== run || run.frames.get(0)?.token !== documentToken || run.url !== url) return false; + run.uploadObservationScope = { documentToken, url, token: inventoryToken, selectors: recorded, source: 'upload_handoff' }; + return true; + }, + _pageFeedbackBlocksCall(tabId, name, args = {}, { focusFollowUp = false } = {}) { const run = this._pageFeedbackRuns?.get(tabId); if ((name === 'scroll' || name === 'scroll_at') && run?.modelState diff --git a/src/chrome/src/agent/vision-runtime.js b/src/chrome/src/agent/vision-runtime.js index ea50b7428..97b82ca24 100644 --- a/src/chrome/src/agent/vision-runtime.js +++ b/src/chrome/src/agent/vision-runtime.js @@ -174,6 +174,18 @@ function uploadAttached(result) { && result.noDispatch !== true && ['input_attached', 'page_consumed'].includes(result.attachmentState); } +// The selector a blocked chooser's result tells the model to pass to +// upload_file, kept with the handoff that covers it until the handoff starts +// (_visionUploadHandoffTick). The compact tier's upload_file names its target +// by its own discovery, never by selector. +function noteHandoffUploadSelector(handoff, result, compact) { + const selector = typeof result?.selector === 'string' ? result.selector : ''; + const selectors = handoff.selectors ??= []; + if (compact || handoff.active || !selector || selector.length > 4096 || selectors.includes(selector) + || selectors.length >= 8) return; + selectors.push(selector); +} + function regionLabel(region) { return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) }; } @@ -381,14 +393,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { // Another chooser in the batch that armed a handoff: that handoff gives // the next steps upload_file for this input too, so the model is told // so, not that uploads are unavailable. - if (!reason && snapshot.uploadHandoff) return visionFilePickerBlockedResult(result, { handoff: true, compact }); + if (!reason && snapshot.uploadHandoff) { + noteHandoffUploadSelector(snapshot.uploadHandoff, result, compact); + return visionFilePickerBlockedResult(result, { handoff: true, compact }); + } if (!reason && snapshot.unsuccessfulUploadHandoffs >= UNSUCCESSFUL_UPLOAD_HANDOFFS_PER_RUN) reason = 'exhausted'; if (reason) { this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'refused', reason }); return visionFilePickerBlockedResult(result, { compact, reason }); } snapshot.uploadHandoffs += 1; - snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '' }; + snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '', selectors: [] }; + noteHandoffUploadSelector(snapshot.uploadHandoff, result, compact); this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'armed', count: snapshot.uploadHandoffs }); return visionFilePickerBlockedResult(result, { handoff: true, compact }); } @@ -404,7 +420,9 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { * after its upload, when that page is left, or after * UPLOAD_HANDOFF_DECISIONS decisions without an upload; the last two count * as unsuccessful. `step` is the loop's count of model calls, which grows - * only when the model decides. + * only when the model decides. Starting runs before the step's page + * capture, so the blocked inputs the click's result named are observed for + * the first upload_file (page-feedback.js _pageFeedbackObserveUploadTargets). */ proto._visionUploadHandoffTick = async function _visionUploadHandoffTick(tabId, step) { const snapshot = this._visionRunSnapshots?.get(tabId); @@ -414,6 +432,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { handoff.active = true; handoff.since = step; handoff.url = pageKey(await this._currentUrl?.(tabId)); + if (handoff.selectors?.length) await this._pageFeedbackObserveUploadTargets?.(tabId, handoff.selectors); return; } let end = handoff.done ? 'uploaded' : ''; diff --git a/src/chrome/src/content/page-monitor.js b/src/chrome/src/content/page-monitor.js index f8865f90b..74bba17eb 100644 --- a/src/chrome/src/content/page-monitor.js +++ b/src/chrome/src/content/page-monitor.js @@ -2036,6 +2036,22 @@ fileInventory = { token, runToken, entries }; return true; } + // A vision run's upload handoff (agent vision-runtime.js): the result of a + // click_at the file-chooser guard stopped names the blocked input by + // selector, and that result is how the model observed it. The node each + // selector names now, before the model decides, is kept as an inventory + // under the agent's token. A selector naming no single file input is left + // out; the answer lists the selectors that were kept. + function recordUploadTargets(params = {}) { + if (!active || params.runToken !== runToken || !Array.isArray(params.selectors)) return []; + const pairs = []; + for (const selector of params.selectors.slice(0, 8)) { + if (typeof selector !== 'string' || !selector || selector.length > 4096) continue; + const target = modelTarget({ tool: 'upload_file', selector }); + if (target instanceof HTMLInputElement && target.type === 'file') pairs.push([target, selector]); + } + return pairs.length && recordFileInventory(params.token, pairs) ? pairs.map(([, selector]) => selector) : []; + } // Whether an upload's target is the very node that inventory named with // this selector. A node the page renders later under the same selector, as // a reopened composer does, was never observed. @@ -4023,6 +4039,9 @@ ...(result.inventoried ? { inventoried: true } : {}), ...(result.reason ? { reason: result.reason } : {}), runToken, documentToken, snapshotToken: params.snapshotToken }); } + else if (msg.action === 'page_monitor_record_upload_targets') { + respond({ recorded: recordUploadTargets(msg.params || {}), runToken, documentToken }); + } else if (msg.action === 'page_monitor_traversal') { // Value-free: the background names the URL it already observed; only // whether page script started this document's latest traversal there diff --git a/src/firefox/src/agent/guard-policy.js b/src/firefox/src/agent/guard-policy.js index fe3fdbba5..ca9853443 100644 --- a/src/firefox/src/agent/guard-policy.js +++ b/src/firefox/src/agent/guard-policy.js @@ -328,7 +328,8 @@ export const GUARDS = Object.freeze({ "page-feedback-dispatch-fence": { class: "host_check", family: "tool", decides: "mixed", onDecided: "block", onCannotDecide: "block", summary: "Transport-boundary fence (beforePageAgentDispatch, the content page monitor, the execute_js gate, the trusted Enter path): human input or feedback during preparation, a changed document mapping, focus moved into a child frame, an unprepared or rejecting monitor vetoes the dispatch; after input…" }, "page-feedback-target-binding": { class: "quality", family: "tool", decides: "mixed", onDecided: "advise", onCannotDecide: "advise", - summary: "The selector/ref of a state-changing call cannot be bound to the observed action certificate (target_ambiguous, target_unresolved, target_uncovered, capture budget exhausted) or its original owner context was never certified: nothing is dispatched and a bounded fresh capture is queued for the next…" }, + summary: "The selector/ref of a state-changing call cannot be bound to the observed action certificate (target_ambiguous, target_unresolved, target_uncovered, capture budget exhausted) or its original owner context was never certified: nothing is dispatched and a bounded fresh capture is queued for the next…", + mustNotBlock: "test/vision-mode-agent.mjs#the first upload_file of a handoff, with the blocked click's selector, attaches past the page-action binding check" }, "pdf-viewer-dom-tools": { class: "host_check", family: "tool", decides: "knows", onDecided: "block", onCannotDecide: "block", summary: "A DOM/action tool on Chrome's built-in PDF viewer (read_page is redirected to read_pdf)." }, "preparation-timeout-generic": { class: "quality", family: "tool", decides: "no-evidence", onDecided: "advise", onCannotDecide: "advise", diff --git a/src/firefox/src/agent/page-feedback.js b/src/firefox/src/agent/page-feedback.js index a66ad5875..8f6b13a74 100644 --- a/src/firefox/src/agent/page-feedback.js +++ b/src/firefox/src/agent/page-feedback.js @@ -1501,12 +1501,46 @@ export const pageFeedbackMethods = { const observedUpload = name === 'upload_file' && args?.ref_id == null && hasSelector && run.actionBindingInventoried === true && uploadSelectorObserved(run, args.selector); if (!observedRef && !observedUpload) return false; - return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', observedUpload - ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.' - : 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.', + const handoffUpload = observedUpload && run.uploadObservationScope?.source === 'upload_handoff'; + return this._guardAdvisesFor(advisories, 'page-feedback-target-binding', !observedUpload + ? 'This target was outside the page observation WebBrain certified for this action, so its binding to what you observed was not checked. It was sent anyway; read the page to confirm what it changed.' + : handoffUpload + ? 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input the blocked file-chooser click named without that check. Read the page to confirm the file landed where you intended.' + : 'WebBrain could not certify the page context of this file input, so the attachment was sent to the exact input your file-input inventory named without that check. Read the page to confirm the file landed where you intended.', { tabId, decided: false }); }, + // A vision run's upload handoff (vision-runtime.js): the click_at the + // file-chooser guard stopped told the model to call upload_file with the + // blocked input's selector. That result is the model's observation of the + // input, as get_interactive_elements is on the tree path, so before the + // first decision of the handoff the page monitor keeps the node each + // selector names, and the run's upload scope holds exactly those + // selectors. The first upload_file with one of them is then covered by the + // capture (uploadTargets) or sent as an inventoried input above, instead of + // being refused until the model reads an inventory and sends the same call + // again. Any other selector, and a node the page renders later under the + // same selector, keep their answer. Returns whether the scope was set. + async _pageFeedbackObserveUploadTargets(tabId, selectors = []) { + const run = this._pageFeedbackRuns?.get(tabId); + const documentToken = run?.frames.get(0)?.token, url = run?.url; + const wanted = [...new Set((Array.isArray(selectors) ? selectors : []).filter(selector => + typeof selector === 'string' && selector.length > 0 && selector.length <= 4096))].slice(0, 8); + if (!run || !documentToken || !url || !wanted.length) return false; + const signal = this._runAbortSignal?.(tabId) || null; + const inventoryToken = token(); + let response = null; + try { + response = await untilAborted(apiFor().tabs.sendMessage(tabId, { target: 'content', action: 'page_monitor_record_upload_targets', + params: { runToken: run.token, token: inventoryToken, selectors: wanted } }, { frameId: 0 }), signal); + } catch { this._throwIfAborted?.(signal); return false; } + const recorded = Array.isArray(response?.recorded) ? wanted.filter(selector => response.recorded.includes(selector)) : []; + if (!recorded.length || response.runToken !== run.token || response.documentToken !== documentToken + || this._pageFeedbackRuns?.get(tabId) !== run || run.frames.get(0)?.token !== documentToken || run.url !== url) return false; + run.uploadObservationScope = { documentToken, url, token: inventoryToken, selectors: recorded, source: 'upload_handoff' }; + return true; + }, + _pageFeedbackBlocksCall(tabId, name, args = {}, { focusFollowUp = false } = {}) { const run = this._pageFeedbackRuns?.get(tabId); if ((name === 'scroll' || name === 'scroll_at') && run?.modelState diff --git a/src/firefox/src/agent/vision-runtime.js b/src/firefox/src/agent/vision-runtime.js index ea50b7428..97b82ca24 100644 --- a/src/firefox/src/agent/vision-runtime.js +++ b/src/firefox/src/agent/vision-runtime.js @@ -174,6 +174,18 @@ function uploadAttached(result) { && result.noDispatch !== true && ['input_attached', 'page_consumed'].includes(result.attachmentState); } +// The selector a blocked chooser's result tells the model to pass to +// upload_file, kept with the handoff that covers it until the handoff starts +// (_visionUploadHandoffTick). The compact tier's upload_file names its target +// by its own discovery, never by selector. +function noteHandoffUploadSelector(handoff, result, compact) { + const selector = typeof result?.selector === 'string' ? result.selector : ''; + const selectors = handoff.selectors ??= []; + if (compact || handoff.active || !selector || selector.length > 4096 || selectors.includes(selector) + || selectors.length >= 8) return; + selectors.push(selector); +} + function regionLabel(region) { return { x: Math.round(region.x), y: Math.round(region.y), width: Math.round(region.width), height: Math.round(region.height) }; } @@ -381,14 +393,18 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { // Another chooser in the batch that armed a handoff: that handoff gives // the next steps upload_file for this input too, so the model is told // so, not that uploads are unavailable. - if (!reason && snapshot.uploadHandoff) return visionFilePickerBlockedResult(result, { handoff: true, compact }); + if (!reason && snapshot.uploadHandoff) { + noteHandoffUploadSelector(snapshot.uploadHandoff, result, compact); + return visionFilePickerBlockedResult(result, { handoff: true, compact }); + } if (!reason && snapshot.unsuccessfulUploadHandoffs >= UNSUCCESSFUL_UPLOAD_HANDOFFS_PER_RUN) reason = 'exhausted'; if (reason) { this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'refused', reason }); return visionFilePickerBlockedResult(result, { compact, reason }); } snapshot.uploadHandoffs += 1; - snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '' }; + snapshot.uploadHandoff = { active: false, done: false, since: 0, url: '', selectors: [] }; + noteHandoffUploadSelector(snapshot.uploadHandoff, result, compact); this._visionTraceNote(tabId, 'vision_upload_handoff', { stage: 'armed', count: snapshot.uploadHandoffs }); return visionFilePickerBlockedResult(result, { handoff: true, compact }); } @@ -404,7 +420,9 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { * after its upload, when that page is left, or after * UPLOAD_HANDOFF_DECISIONS decisions without an upload; the last two count * as unsuccessful. `step` is the loop's count of model calls, which grows - * only when the model decides. + * only when the model decides. Starting runs before the step's page + * capture, so the blocked inputs the click's result named are observed for + * the first upload_file (page-feedback.js _pageFeedbackObserveUploadTargets). */ proto._visionUploadHandoffTick = async function _visionUploadHandoffTick(tabId, step) { const snapshot = this._visionRunSnapshots?.get(tabId); @@ -414,6 +432,7 @@ export function installVisionRuntime(AgentClass, { trace = null } = {}) { handoff.active = true; handoff.since = step; handoff.url = pageKey(await this._currentUrl?.(tabId)); + if (handoff.selectors?.length) await this._pageFeedbackObserveUploadTargets?.(tabId, handoff.selectors); return; } let end = handoff.done ? 'uploaded' : ''; diff --git a/src/firefox/src/content/page-monitor.js b/src/firefox/src/content/page-monitor.js index f8865f90b..74bba17eb 100644 --- a/src/firefox/src/content/page-monitor.js +++ b/src/firefox/src/content/page-monitor.js @@ -2036,6 +2036,22 @@ fileInventory = { token, runToken, entries }; return true; } + // A vision run's upload handoff (agent vision-runtime.js): the result of a + // click_at the file-chooser guard stopped names the blocked input by + // selector, and that result is how the model observed it. The node each + // selector names now, before the model decides, is kept as an inventory + // under the agent's token. A selector naming no single file input is left + // out; the answer lists the selectors that were kept. + function recordUploadTargets(params = {}) { + if (!active || params.runToken !== runToken || !Array.isArray(params.selectors)) return []; + const pairs = []; + for (const selector of params.selectors.slice(0, 8)) { + if (typeof selector !== 'string' || !selector || selector.length > 4096) continue; + const target = modelTarget({ tool: 'upload_file', selector }); + if (target instanceof HTMLInputElement && target.type === 'file') pairs.push([target, selector]); + } + return pairs.length && recordFileInventory(params.token, pairs) ? pairs.map(([, selector]) => selector) : []; + } // Whether an upload's target is the very node that inventory named with // this selector. A node the page renders later under the same selector, as // a reopened composer does, was never observed. @@ -4023,6 +4039,9 @@ ...(result.inventoried ? { inventoried: true } : {}), ...(result.reason ? { reason: result.reason } : {}), runToken, documentToken, snapshotToken: params.snapshotToken }); } + else if (msg.action === 'page_monitor_record_upload_targets') { + respond({ recorded: recordUploadTargets(msg.params || {}), runToken, documentToken }); + } else if (msg.action === 'page_monitor_traversal') { // Value-free: the background names the URL it already observed; only // whether page script started this document's latest traversal there diff --git a/test/page-monitor-upload-release.mjs b/test/page-monitor-upload-release.mjs index 40a53ea19..883e9e63d 100644 --- a/test/page-monitor-upload-release.mjs +++ b/test/page-monitor-upload-release.mjs @@ -313,6 +313,31 @@ for (const [build, engine] of [['chrome', chromium], ['firefox', firefox]]) { assert.equal((await check(current, fresh.selector, 'inventory-2')).inventoried, true, 'the fresh inventory lists the new node'); assert.equal((await check(current, fresh.selector, 'inventory-1')).inventoried, undefined, 'the older inventory is spent'); }, instagramDirectThreadHtml())); + // A vision run's upload handoff (vision-runtime.js): the blocked + // click_at's result named the input by selector, and the monitor keeps + // the node that selector names when the handoff starts, under the + // agent's token, as an inventory would. + await t.test('a handoff\'s blocked input is known by the node its selector named when the handoff started', () => withFixture(async page => { + const selector = 'input[type="file"][accept="audio/*,.mp4,.mov,.png,.jpg,.jpeg"][multiple]'; + const record = params => page.evaluate(params => deliver('page_monitor_record_upload_targets', params), params); + const check = (snapshot, selector, fileInventoryToken) => page.evaluate(params => deliver('page_monitor_validate_model', + { tool: 'upload_file', ...params }), { selector, snapshotToken: snapshot.snapshotToken, fileInventoryToken }); + // Only a selector naming one file input is kept. + const recorded = await record({ token: 'handoff-1', selectors: [selector, 'input[type="file"].missing', '[role="textbox"]'] }); + assert.deepEqual(recorded.recorded, [selector], JSON.stringify(recorded)); + assert.equal(recorded.runToken, 'upload-release-run'); + const snapshot = await capture(page, { uploadTargets: [selector] }); + const listed = await check(snapshot, selector, 'handoff-1'); + assert.equal(listed.reason, 'target_uncovered', JSON.stringify(listed)); + assert.equal(listed.inventoried, true, JSON.stringify(listed)); + assert.equal((await check(snapshot, 'input[type=file]', 'handoff-1')).inventoried, undefined, 'a guessed selector for the same node'); + assert.deepEqual((await record({ token: 'handoff-2', selectors: [selector], runToken: 'another-run' })).recorded, [], 'another run records nothing'); + assert.equal((await check(snapshot, selector, 'handoff-1')).inventoried, true, 'and leaves the handoff\'s record'); + await page.evaluate(() => window.rerenderComposerInput()); + const remounted = await check(await capture(page, { uploadTargets: [selector] }), selector, 'handoff-1'); + assert.equal(remounted.reason, 'target_uncovered', JSON.stringify(remounted)); + assert.equal(remounted.inventoried, undefined, `a re-mounted input was never named: ${JSON.stringify(remounted)}`); + }, instagramDirectThreadHtml())); for (const authority of ['user', 'Stop', 'run', 'document']) await t.test(`${authority} revokes upload authority`, () => withFixture(async page => { const snapshot = await capture(page); const attached = await attach(page, snapshot, async page => { diff --git a/test/quality/vision-mode-upload-mv3.mjs b/test/quality/vision-mode-upload-mv3.mjs index 6bc066040..d39f49ec8 100644 --- a/test/quality/vision-mode-upload-mv3.mjs +++ b/test/quality/vision-mode-upload-mv3.mjs @@ -6,6 +6,7 @@ import os from 'node:os'; import path from 'node:path'; import { chromium } from 'playwright'; import { fixtureFixedChoiceReply } from '../semantic-decision-stub.mjs'; +import { uploadReleaseHtml } from '../lib/upload-release-fixture.mjs'; // A file upload in a vision run, end to end in the installed MV3 extension: // background, Agent, CDP capture, trusted input and the file-chooser guard are @@ -13,6 +14,11 @@ import { fixtureFixedChoiceReply } from '../semantic-decision-stub.mjs'; // file input, the guard stops the chooser, the next step gets the element // tools and attaches the user's video with upload_file, and the step after // that is back on screenshots, on a fresh full frame. +// The second page is a release form as a three-asset GitHub release met it: +// the binary input is hidden behind an "Attach binaries" button, on a page too +// deep for the page observation to certify. The first upload_file, with the +// selector the blocked click's result named, must attach the file; it used to +// be refused until the model read an inventory and sent the same call again. // node test/quality/vision-mode-upload-mv3.mjs const out = path.resolve(process.env.WEBBRAIN_QUALITY_ARTIFACTS || 'ci/artifacts/vision-mode-upload-mv3'); await fs.mkdir(out, { recursive: true }); @@ -34,9 +40,15 @@ document.getElementById('upload').addEventListener('change', async event => { status.textContent = 'Attached ' + file.name; status.style.background = 'rgb(0,160,0)'; });`; +// As upload-release-binding-mv3.mjs builds it: the general observation runs +// out of budget before it reaches the binary widget. +const releaseBody = uploadReleaseHtml.replace('', + ``); const providerCalls = []; let actionCalls = 0; +// Where the scripted model clicks, in CSS px of the page (the scenario sets it). +let clickTarget = LAYOUT.upload; const labelOf = message => (Array.isArray(message.content) ? message.content.filter(block => block.type === 'text').map(block => block.text).join('\n') : ''); const centre = (rect, frame) => ({ x: Math.round((rect.x + rect.w / 2) / frame.ratio), y: Math.round((rect.y + rect.h / 2) / frame.ratio) }); @@ -74,9 +86,9 @@ const server = http.createServer(async (request, response) => { let call; if (offered.includes('upload_file') && blocked && !uploaded) { call = { name: 'upload_file', args: { selector: /"selector":"((?:[^"\\]|\\.)+)"/.exec(blocked.text)?.[1]?.replace(/\\"/g, '"') || '#upload', attachmentId } }; - } else if (offered.includes('click_at') && !blocked && frame) call = { name: 'click_at', args: centre(LAYOUT.upload, frame) }; + } else if (offered.includes('click_at') && !blocked && frame) call = { name: 'click_at', args: centre(clickTarget, frame) }; else if (offered.includes('take_screenshot') && !frame) call = { name: 'take_screenshot', args: {} }; - else call = { name: 'done', args: { summary: 'The clip was attached to the QA video input.', outcome: 'success' } }; + else call = { name: 'done', args: { summary: 'The clip was attached to the upload input.', outcome: 'success' } }; const step = actionCalls++; completion = { role: 'assistant', content: '', tool_calls: [{ id: `vision-upload-${step}`, type: 'function', function: { name: call.name, arguments: JSON.stringify(call.args) } }] }; } @@ -85,11 +97,10 @@ const server = http.createServer(async (request, response) => { return; } response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' }); - response.end(body); + response.end(request.url === '/release' ? releaseBody : body); }); await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); const base = `http://127.0.0.1:${server.address().port}`; -const url = `${base}/upload`; const profile = await fs.mkdtemp(path.join(os.tmpdir(), 'webbrain-vision-upload-mv3-')); let context; try { @@ -110,51 +121,81 @@ try { }); assert.equal((await message('update_provider', { providerId: 'openrouter', config: { apiKey: 'owned-fixture-fake-key', baseUrl: `${base}/v1`, model: 'anthropic/claude-sonnet-4.5', contextWindow: 200000, enabled: true, supportsAskStreaming: false, supportsVision: true } })).ok, true); assert.equal((await message('set_active_provider', { providerId: 'openrouter' })).ok, true); - const page = await context.newPage(); await page.goto(url); - const tabId = await harness.evaluate(async url => (await chrome.tabs.query({})).find(tab => tab.url === url)?.id, url); - assert.ok(Number.isInteger(tabId), 'the fixture tab is known'); - const result = await message('chat', { - tabId, text: 'Attach the clip I uploaded to the QA video input.', mode: 'act', requestId: 'vision-upload-mv3', locale: 'en', - attachments: [{ kind: 'video', name: 'clip.mp4', dataUrl: `data:video/mp4;base64,${mp4.toString('base64')}`, mimeType: 'video/mp4', size: mp4.length }], - }); - const events = await harness.evaluate(() => events); - const nativeAttachments = await page.evaluate(() => window.nativeAttachments); - await fs.writeFile(path.join(out, 'evidence.json'), JSON.stringify({ result, events, nativeAttachments, providerCalls }, - (_key, item) => typeof item === 'string' && item.startsWith('data:image/') ? `[image ${item.length} chars]` : item, 2)); - const actionRequests = providerCalls.filter(call => Array.isArray(call.tools) && call.tools.length); - const offered = actionRequests.map(call => call.tools.map(tool => tool.function.name)); - const toolTexts = request => request.messages.filter(item => item.role === 'tool').map(item => String(item.content)); - const summary = JSON.stringify({ offered: offered.map(names => (names.includes('click_at') ? 'vision' : names.includes('upload_file') ? 'tree' : names.join(','))), tools: (result.updates || []).filter(event => event.type === 'tool_result').map(event => ({ name: event.data?.name, result: JSON.stringify(event.data?.result).slice(0, 300) })) }, null, 1); - assert.ok(offered[0]?.includes('click_at') && !offered[0].includes('upload_file'), `the run starts on screenshots: ${summary}`); - const handoffAt = offered.findIndex(names => names.includes('upload_file')); - assert.ok(handoffAt > 0, `a later step offers upload_file: ${summary}`); - assert.ok(!offered[handoffAt].includes('click_at'), 'the handoff step is on the element tools'); - // The real CDP click reached the file input and the guard stopped its chooser. - const blocked = toolTexts(actionRequests[handoffAt]).find(text => /"filePickerBlocked":true/.test(text)) || ''; - assert.match(blocked, /"uploadHandoff":true/, blocked); - assert.match(blocked, /"dispatched":true/, blocked); - assert.match(blocked, /"selector":"[^"]+"/, `the guard named the input: ${blocked}`); - const uploads = (result.updates || []).filter(event => event.type === 'tool_result' && event.data?.name === 'upload_file'); - assert.equal(uploads.length, 1, summary); - assert.equal(uploads[0].data.result.success, true, JSON.stringify(uploads[0].data.result)); - assert.deepEqual(nativeAttachments, [{ name: 'clip.mp4', size: mp4.length, mime: 'video/mp4', sha256: sha(mp4) }], 'the page input received the exact bytes once'); - // The step after the upload is back on screenshots, on a fresh full frame. - const backAt = offered.findIndex((names, index) => index > handoffAt && names.includes('click_at')); - assert.ok(backAt > handoffAt, `vision resumed after the upload: ${summary}`); - assert.ok(!offered[backAt].includes('upload_file')); - const back = actionRequests[backAt]; - const frameAt = back.messages.findLastIndex(item => /UNTRUSTED SCREENSHOT f\d+/.test(labelOf(item)) && item.content.some(block => block.type === 'image_url')); - assert.ok(frameAt >= 0, 'the resumed step has a full frame'); - const frameLabel = labelOf(back.messages[frameAt]); - assert.match(frameLabel, /\(vision resumed\)/, frameLabel); - assert.notEqual(/SCREENSHOT (f\d+)/.exec(frameLabel)[1], 'f1'); - assert.ok(back.messages.slice(frameAt + 1).every(item => !/UNTRUSTED SCREEN/.test(labelOf(item))), 'the fresh frame is the latest screen'); - const olderImages = back.messages.slice(0, frameAt).flatMap(item => (Array.isArray(item.content) ? item.content : [])).filter(block => block.type === 'image_url').length; - assert.equal(olderImages, 0, 'screenshots from before the handoff left the request'); - assert.equal(events.findLast(event => event.type === 'run_complete')?.data?.status, 'completed', summary); - assert.equal(JSON.stringify(providerCalls).includes(mp4.toString('base64')), false, 'the video bytes never reach the model'); - console.log(JSON.stringify({ passed: true, offered: offered.map(names => (names.includes('click_at') ? 'vision' : 'tree')), frame: /SCREENSHOT (f\d+)/.exec(frameLabel)[1], attachment: nativeAttachments[0] })); + // One vision run on `pathname`: the scripted model clicks `target` (a rect, + // or a function of the page that measures one) and uploads the clip. + const runScenario = async (pathname, { requestId, text, target, evidence }) => { + providerCalls.length = 0; + const url = `${base}${pathname}`; + const page = await context.newPage(); await page.goto(url); + clickTarget = typeof target === 'function' ? await target(page) : target; + const tabId = await harness.evaluate(async url => (await chrome.tabs.query({})).find(tab => tab.url === url)?.id, url); + assert.ok(Number.isInteger(tabId), 'the fixture tab is known'); + await harness.evaluate(() => { window.events = []; }); + const result = await message('chat', { + tabId, text, mode: 'act', requestId, locale: 'en', + attachments: [{ kind: 'video', name: 'clip.mp4', dataUrl: `data:video/mp4;base64,${mp4.toString('base64')}`, mimeType: 'video/mp4', size: mp4.length }], + }); + const events = await harness.evaluate(() => events); + const calls = [...providerCalls]; + const pageState = await page.evaluate(() => ({ nativeAttachments: window.nativeAttachments, attachmentEvents: window.attachmentEvents, chooserClicks: window.chooserClicks })); + await fs.writeFile(path.join(out, evidence), JSON.stringify({ result, events, pageState, providerCalls: calls }, + (_key, item) => typeof item === 'string' && item.startsWith('data:image/') ? `[image ${item.length} chars]` : item, 2)); + + const actionRequests = calls.filter(call => Array.isArray(call.tools) && call.tools.length); + const offered = actionRequests.map(call => call.tools.map(tool => tool.function.name)); + const toolTexts = request => request.messages.filter(item => item.role === 'tool').map(item => String(item.content)); + const toolResults = (result.updates || []).filter(event => event.type === 'tool_result'); + const summary = JSON.stringify({ offered: offered.map(names => (names.includes('click_at') ? 'vision' : names.includes('upload_file') ? 'tree' : names.join(','))), tools: toolResults.map(event => ({ name: event.data?.name, result: JSON.stringify(event.data?.result).slice(0, 300) })) }, null, 1); + assert.ok(offered[0]?.includes('click_at') && !offered[0].includes('upload_file'), `the run starts on screenshots: ${summary}`); + const handoffAt = offered.findIndex(names => names.includes('upload_file')); + assert.ok(handoffAt > 0, `a later step offers upload_file: ${summary}`); + assert.ok(!offered[handoffAt].includes('click_at'), 'the handoff step is on the element tools'); + // The real CDP click reached the file input and the guard stopped its chooser. + const blocked = toolTexts(actionRequests[handoffAt]).find(text => /"filePickerBlocked":true/.test(text)) || ''; + assert.match(blocked, /"uploadHandoff":true/, blocked); + assert.match(blocked, /"dispatched":true/, blocked); + assert.match(blocked, /"selector":"[^"]+"/, `the guard named the input: ${blocked}`); + // The first upload_file attaches the file: no binding refusal, no inventory. + const uploads = toolResults.filter(event => event.data?.name === 'upload_file'); + assert.equal(uploads.length, 1, summary); + assert.equal(uploads[0].data.result.success, true, JSON.stringify(uploads[0].data.result)); + assert.equal(toolResults.some(event => event.data?.name === 'get_interactive_elements'), false, summary); + assert.equal(JSON.stringify(toolResults).includes('action_binding_unavailable'), false, summary); + // The step after the upload is back on screenshots, on a fresh full frame. + const backAt = offered.findIndex((names, index) => index > handoffAt && names.includes('click_at')); + assert.ok(backAt > handoffAt, `vision resumed after the upload: ${summary}`); + assert.ok(!offered[backAt].includes('upload_file')); + const back = actionRequests[backAt]; + const frameAt = back.messages.findLastIndex(item => /UNTRUSTED SCREENSHOT f\d+/.test(labelOf(item)) && item.content.some(block => block.type === 'image_url')); + assert.ok(frameAt >= 0, 'the resumed step has a full frame'); + const frameLabel = labelOf(back.messages[frameAt]); + assert.match(frameLabel, /\(vision resumed\)/, frameLabel); + assert.notEqual(/SCREENSHOT (f\d+)/.exec(frameLabel)[1], 'f1'); + assert.ok(back.messages.slice(frameAt + 1).every(item => !/UNTRUSTED SCREEN/.test(labelOf(item))), 'the fresh frame is the latest screen'); + const olderImages = back.messages.slice(0, frameAt).flatMap(item => (Array.isArray(item.content) ? item.content : [])).filter(block => block.type === 'image_url').length; + assert.equal(olderImages, 0, 'screenshots from before the handoff left the request'); + assert.equal(events.findLast(event => event.type === 'run_complete')?.data?.status, 'completed', summary); + assert.equal(JSON.stringify(calls).includes(mp4.toString('base64')), false, 'the video bytes never reach the model'); + return { offered: offered.map(names => (names.includes('click_at') ? 'vision' : 'tree')), frame: /SCREENSHOT (f\d+)/.exec(frameLabel)[1], blocked, pageState }; + }; + + const video = await runScenario('/upload', { requestId: 'vision-upload-mv3', text: 'Attach the clip I uploaded to the QA video input.', + target: LAYOUT.upload, evidence: 'evidence.json' }); + assert.deepEqual(video.pageState.nativeAttachments, [{ name: 'clip.mp4', size: mp4.length, mime: 'video/mp4', sha256: sha(mp4) }], 'the page input received the exact bytes once'); + + const release = await runScenario('/release', { requestId: 'vision-upload-release-mv3', text: 'Attach the clip I uploaded to the release binaries.', + target: async page => { + const box = await page.locator('#binary-button').boundingBox(); + return { x: box.x, y: box.y, w: box.width, h: box.height }; + }, evidence: 'evidence-release.json' }); + // The button's own handler called click() on the hidden input; the guard + // named that input, and the clip reached it, not the Markdown slot. + assert.match(release.blocked, /"selector":"#releases-upload"/, release.blocked); + assert.equal(release.pageState.chooserClicks, 1); + assert.deepEqual(release.pageState.attachmentEvents, { markdown: [], binary: [[{ name: 'clip.mp4', size: mp4.length }]] }); + console.log(JSON.stringify({ passed: true, video: { offered: video.offered, frame: video.frame, attachment: video.pageState.nativeAttachments[0] }, + release: { offered: release.offered, frame: release.frame, attachment: release.pageState.attachmentEvents.binary[0][0] } })); } finally { await context?.close(); await new Promise(resolve => server.close(resolve)); const resolved = path.resolve(profile); diff --git a/test/vision-mode-agent.mjs b/test/vision-mode-agent.mjs index bac3ff027..dc6c86d49 100644 --- a/test/vision-mode-agent.mjs +++ b/test/vision-mode-agent.mjs @@ -1275,6 +1275,114 @@ for (const build of (process.env.VISION_BUILDS || 'chrome,firefox').split(',')) assert.doesNotMatch(refused, /uploadHandoff/); }); + // The page-action binding check (page-feedback.js) in the real loop. The + // page monitor is faked as GitHub's release form reads to it: the binary + // input is hidden on a page too deep for the observation to certify, so an + // upload_file is target_uncovered, and the monitor says `inventoried` only + // for the very node an inventory with this token listed under this exact + // selector. The first upload_file of every handoff used to be refused here, + // and each file cost a get_interactive_elements and the same call again. + function fakeReleaseMonitor(ctx, tab) { + const dom = new Map([['#upload', 'binary'], ['input[type="file"]', 'binary']]); + const monitor = { inventory: null, log: [], validations: [] }; + const saved = api.tabs.sendMessage; + let snapshots = 0; + api.tabs.sendMessage = async (_tab, message) => { + const params = message.params || {}; + if (message.action === 'page_monitor_capture_model') { + monitor.log.push({ capture: params.uploadTargets || [] }); + return { ready: true, runToken: params.runToken, documentToken: 'content-1', snapshotToken: `owned-${++snapshots}`, + targetCount: 2, focusedTargetAvailable: false, + page: { success: true, url: PAGE, pageContent: 'button "Attach binaries by dropping them here or selecting them." [ref_attach]' } }; + } + if (message.action === 'page_monitor_record_upload_targets') { + const pairs = (params.selectors || []).filter(selector => dom.has(selector)).map(selector => [dom.get(selector), selector]); + monitor.log.push({ record: params.selectors, token: params.token }); + if (pairs.length) monitor.inventory = { token: params.token, entries: new Map(pairs) }; + return { recorded: pairs.map(([, selector]) => selector), runToken: params.runToken, documentToken: 'content-1' }; + } + if (message.action === 'page_monitor_validate_model') { + monitor.validations.push(params); + const node = dom.get(params.selector); + const inventoried = !!node && typeof params.fileInventoryToken === 'string' && !!params.fileInventoryToken + && monitor.inventory?.token === params.fileInventoryToken && monitor.inventory.entries.get(node) === params.selector; + return { ready: false, reason: node ? 'target_uncovered' : 'target_unresolved', ...(inventoried ? { inventoried: true } : {}), + runToken: params.runToken, documentToken: 'content-1', snapshotToken: params.snapshotToken }; + } + if (message.action === 'page_monitor_prepare') return { ready: true, modelBindingValid: true }; + return { ready: true }; + }; + // The page's monitor joins the run as its content script does. + ctx.agent._enrichUserMessageWithCurrentPage = async (_tab, _messages, content) => { + assert.equal(ctx.agent.pageMonitorState({ tab: { id: tab }, frameId: 0, documentId: 'document-1', url: PAGE }, 'content-1').active, true); + return { role: 'user', content }; + }; + return { monitor, restore: () => { api.tabs.sendMessage = saved; } }; + } + const uploadResults = request => toolTexts(request).filter(text => /"attachmentState"|action_binding_unavailable/.test(text)); + + for (const streaming of [false, true]) { + const label = `${build}${streaming ? ' (streaming)' : ''}`; + + test(`${label}: the first upload_file of a handoff, with the blocked click's selector, attaches past the page-action binding check`, async () => { + const tab = nextTab++; + const reasons = []; + const ctx = runAgent(step => { + reasons.push(ctx.agent._visionRunState(tab, 'act').reason); + if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] }; + if (step === 2) return { toolCalls: [call('upload_file', { selector: '#upload', attachmentId: 'attachment_t_1' })] }; + if (step === 3) return { toolCalls: [call('done', { summary: 'Attached.', outcome: 'success' })] }; + return { content: 'done' }; + }); + ctx.upload = attached; + const restoreChooser = blockChooser(ctx); + const { monitor, restore } = fakeReleaseMonitor(ctx, tab); + try { await runLoop(ctx.agent, tab, streaming); } finally { restore(); restoreChooser(); } + assert.deepEqual(reasons, ['vision_capable_model', 'file_upload_needs_element_inventory', 'vision_capable_model']); + assert.deepEqual(ctx.dispatched.filter(entry => entry.name === 'upload_file').map(entry => entry.args.selector), ['#upload']); + const [result, ...others] = uploadResults(ctx.requests[2]); + assert.equal(others.length, 0, 'one upload_file'); + assert.match(result, /"success":true/, result); + assert.doesNotMatch(result, /action_binding_unavailable/); + // The handoff observed the input the click's result named before the + // capture that preceded its first decision. + const record = monitor.log.findIndex(entry => entry.record); + assert.deepEqual(monitor.log[record].record, ['#upload']); + assert.equal(monitor.log.filter(entry => entry.record).length, 1, 'only the start of the handoff observes its input'); + assert.deepEqual(monitor.log[record + 1], { capture: ['#upload'] }, JSON.stringify(monitor.log)); + const checked = monitor.validations.filter(params => params.tool === 'upload_file'); + assert.ok(checked.length >= 1 && checked.every(params => params.fileInventoryToken === monitor.log[record].token), JSON.stringify(checked)); + }); + } + + test(`${build}: during a handoff a guessed upload selector is still refused by the binding check`, async () => { + for (const observe of [true, false]) { + const tab = nextTab++; + const ctx = runAgent((step, _messages, options) => { + const vision = names(options.tools).includes('click_at'); + if (step === 1) return { toolCalls: [call('click_at', { x: 300, y: 200 })] }; + if (step === 2) return { toolCalls: [call('upload_file', { selector: 'input[type="file"]', attachmentId: 'attachment_t_1' })] }; + if (step === 3 && !vision) return { toolCalls: [call('upload_file', { selector: '#upload', attachmentId: 'attachment_t_1' })] }; + if (step <= 4) return { toolCalls: [call('done', { summary: 'Upload step finished.', outcome: 'partial' })] }; + return { content: 'done' }; + }); + ctx.upload = attached; + const restoreChooser = blockChooser(ctx); + const { restore } = fakeReleaseMonitor(ctx, tab); + // Without the handoff's observation, the result's own selector meets + // the same refusal: what every file's first upload_file met before. + if (!observe) ctx.agent._pageFeedbackObserveUploadTargets = async () => false; + try { await runLoop(ctx.agent, tab, false); } finally { restore(); restoreChooser(); } + const label = observe ? 'observed handoff' : 'unobserved handoff'; + const [guessed] = uploadResults(ctx.requests[2]); + assert.match(guessed, /"errorCode":"action_binding_unavailable"/, `${label}: ${guessed}`); + assert.match(guessed, /"bindingFailure":"target_uncovered"/, label); + const [, named] = uploadResults(ctx.requests[3]); + assert.match(named, observe ? /"success":true/ : /"bindingFailure":"target_uncovered"/, `${label}: ${named}`); + assert.deepEqual(ctx.dispatched.filter(entry => entry.name === 'upload_file').map(entry => entry.args.selector), observe ? ['#upload'] : [], label); + } + }); + if (build === 'firefox') { test('firefox: a vision click settles the file-chooser guard and reports a blocked chooser in vision terms', async () => { const agent = new Agent({}); diff --git a/test/vision-mode-guards.mjs b/test/vision-mode-guards.mjs index d3d972900..5d3a9b190 100644 --- a/test/vision-mode-guards.mjs +++ b/test/vision-mode-guards.mjs @@ -503,6 +503,169 @@ for (const build of ['chrome', 'firefox']) { agent._endVisionRun(tab, null); }); + // The page-action binding check (page-feedback.js) during an upload + // handoff. The page monitor is faked as GitHub's release form reads to it: + // the binary input is hidden behind its widget on a page too deep for the + // observation to certify, so every upload_file is target_uncovered, and the + // monitor says `inventoried` only for the very node an inventory with this + // token listed under this exact selector. A three-asset release used to + // spend a refusal, a get_interactive_elements and the same upload_file + // again on every file (trace steps 15-18 and 24-27). + const RELEASE_PAGE = 'https://owned.example/owned/repository/releases/new'; + const GUESSED_SAME_INPUT = 'input[type="file"][multiple]:not([accept])'; + function uploadBindingFixture({ strict = false, documentToken = 'content-1' } = {}) { + const agent = agentFor(); + agent.visionMode = 'auto'; + agent.strictGuards = strict; + const tab = 37; + agent._currentUrl = async () => RELEASE_PAGE; + agent._lastTabUrls = new Map([[tab, RELEASE_PAGE]]); + agent._uncertainTextMutationBlock = async () => null; + agent._finalizeToolResultOnce = async (_tab, _name, _args, result) => result; + const sent = []; + agent._executeToolImpl = async (_tab, name, args) => { + sent.push({ name, selector: args.selector }); + return { success: true, dispatched: true, file: 'chrome.zip', attached: { name: 'chrome.zip', size: 4 }, attachmentState: 'input_attached' }; + }; + // Which node each selector names on the page right now. + const dom = new Map([['#releases-upload', 'binary'], [GUESSED_SAME_INPUT, 'binary'], ['#fc-release-body', 'markdown']]); + const monitor = { inventory: null, records: [], validations: [] }; + const answer = message => { + const params = message.params || {}; + if (message.action === 'page_monitor_record_upload_targets') { + monitor.records.push(params); + const pairs = (params.selectors || []).filter(selector => dom.has(selector)).map(selector => [dom.get(selector), selector]); + if (params.runToken !== 'pf-run' || !pairs.length) return { recorded: [], runToken: 'pf-run', documentToken }; + monitor.inventory = { token: params.token, entries: new Map(pairs) }; + return { recorded: pairs.map(([, selector]) => selector), runToken: 'pf-run', documentToken }; + } + if (message.action === 'page_monitor_validate_model') { + monitor.validations.push(params); + const node = dom.get(params.selector); + const inventoried = !!node && params.tool === 'upload_file' && typeof params.fileInventoryToken === 'string' + && !!params.fileInventoryToken && monitor.inventory?.token === params.fileInventoryToken + && monitor.inventory.entries.get(node) === params.selector; + return { ready: false, reason: node ? 'target_uncovered' : 'target_unresolved', ...(node ? { certificationFailure: 'identity_unobserved' } : {}), + ...(inventoried ? { inventoried: true } : {}), runToken: params.runToken, documentToken: 'content-1', snapshotToken: params.snapshotToken }; + } + return { ready: true }; + }; + const frame = { token: 'content-1', id: 'document-1' }; + const state = { token: 'pf-run', url: RELEASE_PAGE, frames: new Map([[0, frame]]), steeringRevision: 0, + actionBinding: { snapshotToken: 'snapshot-1', runToken: 'pf-run', documentToken: 'content-1' } }; + const run = { token: 'pf-run', url: RELEASE_PAGE, documents: new Map(), frames: new Map([[0, { ...frame }]]), revision: 0, + events: new Map(), modelState: state, gestures: new Set(), gestureLeases: new Map() }; + agent._pageFeedbackRuns = new Map([[tab, run]]); + const blocked = (selector = '#releases-upload') => ({ success: false, dispatched: true, filePickerBlocked: true, selector, error: 'Blocked a native file chooser.' }); + const upload = selector => agent.executeTool(tab, 'upload_file', { selector, downloadId: 2157 }, null, { guardAdvisories: [] }); + return { agent, tab, run, dom, monitor, sent, blocked, upload, answer }; + } + async function withFakeMonitor(answer, work) { + const saved = { chrome: globalThis.chrome, browser: globalThis.browser }; + const area = { get: async () => ({}), set: async () => {}, remove: async () => {} }; + const fake = { + storage: { local: area, session: area }, + runtime: { getURL: value => `extension://test/${value}`, sendMessage: async () => ({}) }, + tabs: { get: async id => ({ id, url: RELEASE_PAGE }), query: async () => [], sendMessage: async (_tab, message) => answer(message) }, + webNavigation: { getAllFrames: async () => [{ frameId: 0, parentFrameId: -1, documentId: 'document-1', url: RELEASE_PAGE }] }, + }; + globalThis.chrome = globalThis.browser = fake; + try { return await work(); } finally { + for (const name of ['chrome', 'browser']) { + if (saved[name] === undefined) delete globalThis[name]; else globalThis[name] = saved[name]; + } + } + } + const refusedAsUncovered = (result, sent, label) => { + assert.deepEqual(sent, [], `${label}: nothing was attached`); + assert.equal(result.errorCode, 'action_binding_unavailable', `${label}: ${JSON.stringify(result)}`); + assert.equal(result.bindingFailure, 'target_uncovered', label); + assert.equal(result.recoveryRequired, 'get_interactive_elements', label); + }; + + for (const [mode, strict] of [['default', false], ['strict', true]]) { + test(`${build}: ${mode} checks: the first upload_file of a handoff ${strict ? 'is refused like an inventory\'s' : 'attaches to the input the blocked click named'}; a guessed selector is refused`, async () => { + const f = uploadBindingFixture({ strict }); + await withFakeMonitor(f.answer, async () => { + f.agent._beginVisionRun(f.tab); + assert.equal(f.agent._visionNoteUploadHandoff(f.tab, 'click', f.blocked(), { vision: { tool: 'click_at' } }).uploadHandoff, true); + // What the first upload_file met before: the input is not covered and + // no inventory named it. + refusedAsUncovered(await f.upload('#releases-upload'), f.sent, 'before the handoff starts'); + await f.agent._visionUploadHandoffTick(f.tab, 1); + assert.equal(f.agent._visionRunState(f.tab, 'act').reason, 'file_upload_needs_element_inventory'); + assert.equal(f.monitor.records.length, 1); + assert.deepEqual(f.monitor.records[0].selectors, ['#releases-upload']); + assert.deepEqual(f.run.uploadObservationScope, { documentToken: 'content-1', url: RELEASE_PAGE, + token: f.monitor.records[0].token, selectors: ['#releases-upload'], source: 'upload_handoff' }); + // A guessed selector, for the same input or another one, keeps its refusal. + for (const guessed of [GUESSED_SAME_INPUT, '#fc-release-body']) refusedAsUncovered(await f.upload(guessed), f.sent, guessed); + const first = await f.upload('#releases-upload'); + assert.equal(f.monitor.validations.at(-1).fileInventoryToken, f.monitor.records[0].token, 'the monitor is asked about the recorded node'); + if (strict) refusedAsUncovered(first, f.sent, 'Strict checks'); + else { + assert.deepEqual(f.sent, [{ name: 'upload_file', selector: '#releases-upload' }], JSON.stringify(first)); + assert.equal(first.success, true); + assert.deepEqual(first.guardAdvisories.map(item => item.guard), ['page-feedback-target-binding']); + assert.match(first.guardAdvisories[0].note, /the exact input the blocked file-chooser click named/); + assert.doesNotMatch(first.guardAdvisories[0].note, /inventory/); + // The upload ends the handoff; the run is back on screenshots. + assert.equal(f.agent._visionNoteUploadHandoff(f.tab, 'upload_file', first, {}), first); + await f.agent._visionUploadHandoffTick(f.tab, 2); + assert.equal(f.agent._visionRunState(f.tab, 'act').active, true); + f.sent.length = 0; + } + // A node the page renders later under the same selector was never observed. + f.dom.set('#releases-upload', 'binary-remounted'); + refusedAsUncovered(await f.upload('#releases-upload'), f.sent, 'a re-mounted input'); + assert.equal(f.monitor.records.length, 1, 'only the start of a handoff observes its inputs'); + }); + f.agent._endVisionRun(f.tab, null); + }); + } + + test(`${build}: a handoff observes every input its batch blocked, on the page and run it starts in, never for the compact tier`, async () => { + // Another chooser in the batch that armed the handoff is covered by it. + const batch = uploadBindingFixture(); + await withFakeMonitor(batch.answer, async () => { + batch.agent._beginVisionRun(batch.tab); + batch.agent._visionNoteUploadHandoff(batch.tab, 'click', batch.blocked(), { vision: { tool: 'click_at' } }); + batch.agent._visionNoteUploadHandoff(batch.tab, 'click', batch.blocked('#fc-release-body'), { vision: { tool: 'click_at' } }); + await batch.agent._visionUploadHandoffTick(batch.tab, 1); + assert.deepEqual(batch.monitor.records.map(record => record.selectors), [['#releases-upload', '#fc-release-body']]); + assert.deepEqual(batch.run.uploadObservationScope.selectors, ['#releases-upload', '#fc-release-body']); + await batch.agent._visionUploadHandoffTick(batch.tab, 2); + assert.equal(batch.monitor.records.length, 1, 'an active handoff does not observe again'); + }); + batch.agent._endVisionRun(batch.tab, null); + // The monitor answered for another document (the page was replaced while + // the click's update settled): nothing is observed. + const replaced = uploadBindingFixture({ documentToken: 'content-2' }); + await withFakeMonitor(replaced.answer, async () => { + replaced.agent._beginVisionRun(replaced.tab); + replaced.agent._visionNoteUploadHandoff(replaced.tab, 'click', replaced.blocked(), { vision: { tool: 'click_at' } }); + await replaced.agent._visionUploadHandoffTick(replaced.tab, 1); + assert.equal(replaced.monitor.records.length, 1); + assert.equal(replaced.run.uploadObservationScope, undefined); + refusedAsUncovered(await replaced.upload('#releases-upload'), replaced.sent, 'another document'); + }); + replaced.agent._endVisionRun(replaced.tab, null); + // The compact tier's upload_file finds its own target by discovery, and a + // chooser the guard could not name gives no selector: neither is observed. + for (const [label, result, promptTier] of [['compact', null, 'compact'], ['unnamed', { selector: undefined }, '']]) { + const f = uploadBindingFixture(); + await withFakeMonitor(f.answer, async () => { + f.agent._userAttachmentHandles.set(f.tab, new Map([['attachment_t_1', { name: 'chrome.zip' }]])); + f.agent._beginVisionRun(f.tab); + assert.equal(f.agent._visionNoteUploadHandoff(f.tab, 'click', { ...f.blocked(), ...result }, { vision: { tool: 'click_at' }, promptTier }).uploadHandoff, true, label); + await f.agent._visionUploadHandoffTick(f.tab, 1); + assert.deepEqual(f.monitor.records, [], label); + assert.equal(f.run.uploadObservationScope, undefined, label); + }); + f.agent._endVisionRun(f.tab, null); + } + }); + test(`${build}: back on vision, a failed capture leaves no frame to aim at and retires the old one in the history`, async () => { const agent = agentFor(); agent.visionMode = 'auto';