> SpecStream is the live web data engine for teams and agents. This is the Markdown copy of https://specstream.ai/docs/integrations/openai-computer-use. Every page, with what SpecStream does and costs: https://specstream.ai/llms.txt # OpenAI computer use Run OpenAI's `computer` tool on a SpecStream browser session. The Responses API returns clicks and keystrokes, Playwright runs them in a real Chromium over the session's `connect_url`, and you send back a screenshot. A session costs $0.10 an hour, charged by the second, so a 5 minute task is under 1 cent of browser time. [SpecStream for OpenAI computer useAgent frameworks & builders](https://specstream.ai/integrations/openai-computer-use.md) ## What you need - Python 3.10 or newer. - An OpenAI API key and a model that supports the `computer` tool. The script uses `gpt-6.1-sol`, the model in OpenAI's own example. - A SpecStream account. New accounts start with $10 of credit, no card. [Create one](https://specstream.ai/signup?utm_source=openai-computer-use). 1. 1 ## 1. Install the packages The OpenAI SDK calls the Responses API, Playwright drives the browser, and httpx calls SpecStream. Playwright connects to the session's browser, so there is no local browser to download and you can skip `playwright install`. Terminal ```bash pip install openai playwright httpx ``` 2. 2 ## 2. Create a SpecStream key and set both keys In the SpecStream workspace, open [Agents & API](https://specstream.ai/workspace/agents?tab=keys&utm_source=openai-computer-use), choose **API keys**, then **Create API key**. A key that opens a browser needs the `finds:spend` scope and a spending limit above 0. The limit is the most the key can spend in total, so a small one is enough for a test. Set it next to your model key in the shell that runs the script: Terminal ```bash export SPECSTREAM_API_KEY="YOUR_SPECSTREAM_API_KEY" export OPENAI_API_KEY="YOUR_OPENAI_API_KEY" ``` 3. 3 ## 3. Save the agent Save this file as `agent.py`. It opens a session, connects Playwright to it, opens the start page, and runs OpenAI's three steps in a loop: send the task, run the returned actions, return a screenshot. The screenshots show the page only, with no address bar, so the script opens `START_URL` itself before the first request. agent.py ```python import base64 import os import time import httpx from openai import OpenAI from playwright.sync_api import sync_playwright API = "https://api.specstream.ai/v1" HEADERS = {"Authorization": f"Bearer {os.environ['SPECSTREAM_API_KEY']}"} MODEL = "gpt-6.1-sol" TOOLS = [{"type": "computer"}] START_URL = "https://news.ycombinator.com" TASK = "Tell me the title of the top story on this page. Use the computer tool for UI interaction." KEYS = { "ENTER": "Enter", "RETURN": "Enter", "ESC": "Escape", "ESCAPE": "Escape", "TAB": "Tab", "SPACE": "Space", "BACKSPACE": "Backspace", "DELETE": "Delete", "DEL": "Delete", "HOME": "Home", "END": "End", "PAGEUP": "PageUp", "PAGEDOWN": "PageDown", "UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight", "ARROWUP": "ArrowUp", "ARROWDOWN": "ArrowDown", "ARROWLEFT": "ArrowLeft", "ARROWRIGHT": "ArrowRight", "CTRL": "Control", "CONTROL": "Control", "SHIFT": "Shift", "OPTION": "Alt", "ALT": "Alt", "META": "Meta", "CMD": "Meta", "COMMAND": "Meta", } BUTTONS = {"left": "left", "right": "right", "wheel": "middle"} def key(name): return KEYS.get(name.upper(), name) def handle(page, actions): """Run the actions of one computer_call on the page, in order.""" for action in actions: kind = action.type if kind == "screenshot": continue # one is taken after every batch if kind == "keypress": page.keyboard.press("+".join(key(k) for k in action.keys)) continue if kind == "type": page.keyboard.type(action.text) continue if kind == "wait": time.sleep(2) continue # The mouse actions: keys holds the modifiers for this one action. held = [key(k) for k in (getattr(action, "keys", None) or [])] for k in held: page.keyboard.down(k) try: if kind == "click": button = BUTTONS[getattr(action, "button", "left")] page.mouse.click(action.x, action.y, button=button) elif kind == "double_click": page.mouse.dblclick(action.x, action.y) elif kind == "move": page.mouse.move(action.x, action.y) elif kind == "scroll": page.mouse.move(action.x, action.y) page.mouse.wheel(action.scroll_x, action.scroll_y) elif kind == "drag": path = [(point.x, point.y) for point in action.path] page.mouse.move(*path[0]) page.mouse.down() for x, y in path[1:]: page.mouse.move(x, y) page.mouse.up() else: raise ValueError(f"Unsupported action: {kind}") finally: for k in reversed(held): page.keyboard.up(k) def main(): session = httpx.post( f"{API}/browser/sessions", headers=HEADERS, json={"timeout_s": 600}, timeout=60 ) session.raise_for_status() session = session.json()["data"] print("Watch it:", session["live_url"]) client = OpenAI() try: with sync_playwright() as p: browser = p.chromium.connect_over_cdp(session["connect_url"]) context = browser.contexts[0] page = context.pages[0] if context.pages else context.new_page() page.goto(START_URL) response = client.responses.create(model=MODEL, tools=TOOLS, input=TASK) for _ in range(30): call = next((item for item in response.output if item.type == "computer_call"), None) if call is None: print(response.output_text) break handle(page, call.actions) image = base64.b64encode(page.screenshot(type="png")).decode() response = client.responses.create( model=MODEL, tools=TOOLS, previous_response_id=response.id, input=[ { "type": "computer_call_output", "call_id": call.call_id, "output": { "type": "computer_screenshot", "image_url": f"data:image/png;base64,{image}", "detail": "original", }, } ], ) finally: ended = httpx.post( f"{API}/browser/sessions/{session['id']}/release", headers=HEADERS, timeout=60 ).json()["data"] print(f"{ended['seconds']} seconds, ${ended['charged_usd']:.4f}") if __name__ == "__main__": main() ``` 4. 4 ## 4. Run it The script prints the address where you can watch the browser, then the model's answer, then the seconds the session ran and what it cost. Change `START_URL` and `TASK` to your own. Terminal ```bash python agent.py ``` ## What the script does Each part of the loop follows OpenAI's computer use guide. - `POST /v1/browser/sessions` opens the browser. The answer's `data` holds the session's `id`, its `connect_url` and its `live_url`. - `connect_url` is a WebSocket that speaks the Chrome DevTools Protocol, so Playwright's `connect_over_cdp` takes it as it is. Its token belongs to that one session, so your SpecStream key never reaches the browser side. - The first request sends the task with `tools=[{"type": "computer"}]`. The first `computer_call` may hold only a `screenshot` action, and the script answers it with the current screen. - Each `computer_call` holds an ordered `actions` array. The script runs the actions in order, then takes one screenshot. - The screenshot goes back as a `computer_call_output` with the same `call_id`, as a `computer_screenshot` with `detail` set to `original`, and `previous_response_id` continues the conversation, so the script keeps no message history. - The loop ends when a response holds no `computer_call`, and the script prints `output_text`. - `POST /v1/browser/sessions/{id}/release` ends the session, also when the script fails, and returns the `seconds` it ran and `charged_usd`. ## How each action reaches the browser The `computer` tool can request nine actions. A mouse action may carry a `keys` array of modifiers, such as Ctrl for a click that opens a new tab. The script holds them down for that one action. | The model returns | The script runs | | --- | --- | | `click` | `page.mouse.click(x, y)` with `button`, where `wheel` is Playwright's `middle` | | `double_click` | `page.mouse.dblclick(x, y)` | | `move` | `page.mouse.move(x, y)` | | `scroll` | `page.mouse.move(x, y)`, then `page.mouse.wheel(scroll_x, scroll_y)` | | `drag` | `page.mouse.move` to the first point of `path`, `down`, `move` through the other points, `up` | | `keypress` | `page.keyboard.press` with names such as `CTRL` and `ENTER` changed to Playwright's `Control` and `Enter` | | `type` | `page.keyboard.type(text)` | | `wait` | `time.sleep(2)` | | `screenshot` | Nothing: the script takes one after every batch | ## If your code still uses computer-use-preview Many guides still show the preview integration. OpenAI's migration table lists four changes, and the script above is the newer form of each. | Part | Preview | Now | | --- | --- | --- | | Model | `computer-use-preview` | `gpt-6.1-sol` | | Tool | `{"type": "computer_use_preview"}` with `display_width` and `display_height` | `{"type": "computer"}` | | Actions | One `action` on each `computer_call` | An `actions` array on each `computer_call` | | Truncation | `truncation: "auto"` required | Not needed | ## Screenshots and coordinates The model gives `x` and `y` in the pixels of the screenshot you return. The session's screen is 1280 by 720 unless you send another `viewport`, the same size as the browser in OpenAI's local example, and Playwright's screenshot is that screen. The script passes the coordinates to the mouse unchanged. OpenAI recommends `detail: "original"` on screenshots to keep the resolution and improve click accuracy. A larger `viewport` gives the model more to read and uses more input tokens for each screenshot. ## What a session costs Browser time is $0.10 an hour, charged by the second. The script drives the session over `connect_url`, so the clicks and keystrokes add nothing: you pay for the time alone. The $1 per 1,000 actions applies only to steps you send to `POST /v1/browser/sessions/{id}/actions`. The model's tokens are billed by OpenAI: each Responses request is charged for its input and output tokens, and screenshots count as image input. | Session length | Browser time | | --- | --- | | 1 minute | $0.0017 | | 5 minutes | $0.0083 | | 10 minutes | $0.0167 | | 1 hour | $0.10 | ## Keep a run inside limits The model acts on real pages, so bound the run in the session and in the script. - `timeout_s` is the most seconds the session runs, from 60 to 3,600. Leave it out and the session ends after 300. The script sends 600. - `allowed_domains` limits the hosts the browser may open, such as `["example.com", "*.example.com"]`. A page outside them is refused. - `viewport` sets the screen, 1280 by 720 unless you send another, for example `{"width": 1280, "height": 800}`. - The loop stops after 30 turns. Raise or lower the number in `range(30)`. - The key's spending limit bounds what all its sessions cost together. - `live_url` is a page that shows the browser's screen as it changes. The script prints it at the start, so you can watch the run. - Text on a page is data. OpenAI's guide asks you to treat screen content as untrusted and to confirm purchases, sent data and changes that are hard to undo with a person. ## If a run fails The status of the call tells you which side refused. | What you see | What to do | | --- | --- | | `401` from SpecStream | The key is missing or wrong. Check `SPECSTREAM_API_KEY`. | | `402` from SpecStream | The balance or the key's spending limit does not hold the first minute. Add credit or raise the limit. | | `403` from SpecStream | The key lacks the `finds:spend` scope. Create a key that has it. | | `429` from SpecStream | The account holds as many open sessions as it may. Release one, or create again after the seconds in `Retry-After`. | | `503` from SpecStream | No browser is free at the moment. Create again after the seconds in `Retry-After`. | | An error from OpenAI that names the model or the tool | The project has no access to a model with the `computer` tool. Check the model's Tools section in OpenAI's models catalog. | | `Unsupported action` from the script | The model returned an action the handler does not know. Add a branch for it in `handle`. | ## More [SpecStream for OpenAI computer use](https://specstream.ai/integrations/openai-computer-use.md) covers what it does in OpenAI computer use on one page. Every other tool has its guide in [Connect your tools](https://specstream.ai/docs/integrations.md). You see the most a request can cost before it starts, and the matches are paid from your balance. [Billing](https://specstream.ai/docs/billing.md) has the details. For help, email [support@specstream.ai](mailto:support@specstream.ai).