> SpecStream is the live web data engine for teams and agents. This is the Markdown copy of https://specstream.ai/docs/integrations/gemini-computer-use. Every page, with what SpecStream does and costs: https://specstream.ai/llms.txt # Gemini Computer Use Run Gemini computer use on a browser that SpecStream hosts. One call opens a session, Playwright connects to its `connect_url`, and Gemini drives the page: it reads a screenshot, answers with an action, and your loop carries it out. The browser costs $0.10 an hour, charged by the second, so a ten minute task costs under 2 cents of browser time. The script below is the whole loop in one Python file. [SpecStream for Gemini Computer UseAgent frameworks & builders](https://specstream.ai/integrations/gemini-computer-use.md) ## What you need - A Gemini API key from [Google AI Studio (opens in a new tab)](https://aistudio.google.com/apikey), set as `GEMINI_API_KEY`, and Python with `google-genai` 2.7.0 or higher, Playwright and requests. - A SpecStream key. [Get your key](https://specstream.ai/workspace/search?utm_source=gemini-computer-use&utm_medium=howto) on the Search API screen of your SpecStream workspace. New accounts start with $10 of credit, no card. 1. 1 ## 1. Install the packages and set both keys Google's guide also runs `playwright install chromium`. You can leave that out here: the browser runs at SpecStream, and Playwright on your machine only connects to it. Terminal ```bash pip install google-genai playwright requests export GEMINI_API_KEY="your Gemini key" export SPECSTREAM_API_KEY="your SpecStream key" ``` 2. 2 ## 2. Open a browser session `POST /v1/browser/sessions` opens the browser. Every field is optional. `timeout_s` sets the most seconds the session runs, from 60 to 3,600, and `viewport` sets the screen Gemini sees. Google's examples use 1440 by 900, so the script asks for the same. The reply holds the session under `data`: its `id`, a `connect_url` that speaks the Chrome DevTools Protocol, and a `live_url` that shows the browser's screen while the agent works. The token in `connect_url` belongs to this session alone, so you can hand it to a sandbox without your API key. Terminal ```bash curl -X POST https://api.specstream.ai/v1/browser/sessions \ -H "Authorization: Bearer $SPECSTREAM_API_KEY" \ -H "Content-Type: application/json" \ -d '{"timeout_s": 600, "viewport": {"width": 1440, "height": 900}}' ``` 3. 3 ## 3. Save the agent loop Save this as `gemini_agent.py`. It opens the session, connects Playwright with `connect_over_cdp`, and sends Gemini the goal with a first screenshot through `client.interactions.create` and the `computer_use` tool in the `browser` environment. Each turn, the script reads the `function_call` steps, carries out each action, and sends back one `function_result` per call with the page's URL and a new screenshot. `previous_interaction_id` continues the same task. The loop ends when Gemini answers with text and no action, or after `MAX_TURNS` turns. Gemini gives coordinates on a 1000 by 1000 grid. `px` scales each one to the viewport before the click. `act` handles every action Google lists for the browser environment, including `scroll`, `drag_and_drop`, `press_key` and `hotkey`. The `finally` block releases the session, so the charge stops that second even when the script fails, and prints what the session cost. gemini\_agent.py ```python import base64 import json import os import time import requests from google import genai from playwright.sync_api import sync_playwright SESSIONS = "https://api.specstream.ai/v1/browser/sessions" HEADERS = {"Authorization": "Bearer " + os.environ["SPECSTREAM_API_KEY"]} MODEL = "gemini-4-hydrogen" WIDTH, HEIGHT = 1440, 900 TOOLS = [{"type": "computer_use", "environment": "browser"}] GOAL = "Find the title of the top story on Hacker News and the site it links to." START = "https://news.ycombinator.com" MAX_TURNS = 20 KEYS = { "control": "Control", "ctrl": "Control", "cmd": "Meta", "command": "Meta", "shift": "Shift", "alt": "Alt", "enter": "Enter", "return": "Enter", "tab": "Tab", "escape": "Escape", "backspace": "Backspace", "delete": "Delete", "space": "Space", "pageup": "PageUp", "pagedown": "PageDown", "up": "ArrowUp", "down": "ArrowDown", "left": "ArrowLeft", "right": "ArrowRight", } def key(name): return KEYS.get(name.lower(), name) def px(value, size): # Gemini answers on a 1000 by 1000 grid. Scale each coordinate to the viewport. return int(value / 1000 * size) def act(page, name, args): x = px(args["x"], WIDTH) if "x" in args else None y = px(args["y"], HEIGHT) if "y" in args else None if name == "click": page.mouse.click(x, y) elif name == "double_click": page.mouse.dblclick(x, y) elif name == "triple_click": page.mouse.click(x, y, click_count=3) elif name == "right_click": page.mouse.click(x, y, button="right") elif name == "middle_click": page.mouse.click(x, y, button="middle") elif name == "move": page.mouse.move(x, y) elif name == "mouse_down": page.mouse.move(x, y) page.mouse.down() elif name == "mouse_up": page.mouse.move(x, y) page.mouse.up() elif name == "drag_and_drop": page.mouse.move(px(args["start_x"], WIDTH), px(args["start_y"], HEIGHT)) page.mouse.down() page.mouse.move(px(args["end_x"], WIDTH), px(args["end_y"], HEIGHT), steps=10) page.mouse.up() elif name == "scroll": amount = int(args.get("magnitude_in_pixels", 300)) moves = {"up": (0, -amount), "down": (0, amount), "left": (-amount, 0), "right": (amount, 0)} page.mouse.move(x, y) page.mouse.wheel(*moves[args["direction"]]) elif name == "type": if x is not None and y is not None: page.mouse.click(x, y) page.keyboard.type(args["text"]) if args.get("press_enter"): page.keyboard.press("Enter") elif name == "press_key": page.keyboard.press(key(args["key"])) elif name == "key_down": page.keyboard.down(key(args["key"])) elif name == "key_up": page.keyboard.up(key(args["key"])) elif name == "hotkey": page.keyboard.press("+".join(key(item) for item in args["keys"])) elif name == "navigate": page.goto(args["url"]) elif name == "go_back": page.go_back() elif name == "go_forward": page.go_forward() elif name == "wait": time.sleep(args.get("seconds", 1)) elif name != "take_screenshot": # every reply carries a new screenshot raise ValueError("No handler for " + name) def screenshot(page): return base64.b64encode(page.screenshot(type="png")).decode("utf-8") def allowed(args): decision = args.get("safety_decision") or {} if decision.get("decision") != "require_confirmation": return False print("Gemini asks before this step:", decision.get("explanation", "")) if input("Allow it? (y/n) ").strip().lower() != "y": raise SystemExit("Stopped before the step.") return True created = requests.post( SESSIONS, headers=HEADERS, json={"timeout_s": 600, "viewport": {"width": WIDTH, "height": HEIGHT}}, timeout=60, ) created.raise_for_status() session = created.json()["data"] print("Watch the browser:", session["live_url"]) client = genai.Client() try: with sync_playwright() as playwright: browser = playwright.chromium.connect_over_cdp(session["connect_url"]) page = browser.contexts[0].new_page() page.set_viewport_size({"width": WIDTH, "height": HEIGHT}) page.goto(START) interaction = client.interactions.create( model=MODEL, input=[ {"type": "text", "text": GOAL}, {"type": "image", "data": screenshot(page), "mime_type": "image/png"}, ], tools=TOOLS, ) for turn in range(MAX_TURNS): calls = [step for step in interaction.steps if step.type == "function_call"] if not calls: print(" ".join( block.text for step in interaction.steps if step.type == "model_output" for block in step.content if block.type == "text" )) break done = [] for call in calls: args = dict(call.arguments) print("Step:", call.name, args.get("intent", "")) outcome = {"safety_acknowledgement": True} if allowed(args) else {} try: act(page, call.name, args) page.wait_for_load_state(timeout=5000) except Exception as error: outcome["error"] = str(error) done.append((call.name, call.id, outcome)) time.sleep(1) shot = screenshot(page) interaction = client.interactions.create( model=MODEL, previous_interaction_id=interaction.id, input=[ { "type": "function_result", "name": name, "call_id": call_id, "result": [ {"type": "text", "text": json.dumps({"url": page.url, **outcome})}, {"type": "image", "data": shot, "mime_type": "image/png"}, ], } for name, call_id, outcome in done ], tools=TOOLS, ) finally: ended = requests.post(SESSIONS + "/" + session["id"] + "/release", headers=HEADERS, timeout=60) print("The session cost $" + str(ended.json()["data"]["charged_usd"])) ``` 4. 4 ## 4. Run it Change `GOAL` and `START` to your task, then run the script. It prints the `live_url` first: open it to watch the browser. Each step prints the action and Gemini's `intent`, the reason it gives for that action. The last lines are Gemini's answer and the session's cost. Terminal ```text python gemini_agent.py Watch the browser: https://... Step: click Open the top story to see where it links. Step: go_back Return to the front page. The top story is ... The session cost $0.0021 ``` ## When Gemini asks before a step Gemini can attach a `safety_decision` to an action, for example before a payment, a sent message or a cookie consent. When the decision is `require_confirmation`, Google's rule is that the end user is asked. The script prints Gemini's explanation and waits for `y`. After a yes it carries out the step and sets `safety_acknowledgement` in the result, as Google's guide shows. After a no it stops, and the session is released. In a product, replace the `input` call in `allowed` with your own approval screen. Keep the stop: an agent that skips the question can buy, send or accept something for your user. ## Keep the agent on the sites you chose - `allowed_domains` on the session lists the hosts the browser may open, such as `example.com` or `*.example.com`. A page outside them is refused, whatever the model asks for. - `timeout_s` ends the session after that many seconds, connected or not, so a loop that never finishes cannot run all day. The most is 3,600. - `MAX_TURNS` ends the loop on your side. Google's sample stops at 5 turns. Most real tasks need more, so the script uses 20. - `block_media` set to true skips images, video, audio and fonts. Leave it off for computer use: Gemini reads the screenshot, and a page without images is harder to read. - To add Google's own check for hidden instructions in a page, add `"enable_prompt_injection_detection": True` to the tool in `TOOLS`. ## Choose the model The script uses `gemini-4-hydrogen`, the model in Google's examples. Google's guide also lists `gemini-4-carbon`, `gemini-4-argon`, `gemini-3.8-flash`, `gemini-3.5-flash-lite` and `gemini-3-flash-preview` for computer use. Change `MODEL` and nothing else. Code written for the older `gemini-2.5-computer-use-preview-10-2025` used `generate_content` and other action names, so it does not run on this loop as it is. ## If something doesn't work | What you see | What to do | | --- | --- | | Clicks land in the wrong place | The coordinates were not scaled, or the viewport differs from `WIDTH` and `HEIGHT`. Keep `px` on every coordinate and the same two numbers in the session's `viewport`. | | `401` when the session opens | The SpecStream key is unknown, expired or revoked. Create a new key in your workspace and set `SPECSTREAM_API_KEY` again. | | `402` when the session opens | The balance does not hold the first minute of the session. Add balance in your SpecStream workspace. | | `422` when the session opens | A field is wrong, and the reply names it. `timeout_s` goes from 60 to 3,600. | | Playwright cannot connect | The session has ended, so its `connect_url` no longer works. A session ends at its `timeout_s`, and at once when the client disconnects. Open a new session. | | A step returns an `error` that names a blocked page | The host is outside `allowed_domains`, is a private address, or the site opted out. Gemini reads the error in the result and can choose another step. | | `No handler for ...` in a result | Gemini called an action the script does not know, such as a custom function you added. Add a branch for it in `act`. | | The loop ends without an answer | The task needed more than `MAX_TURNS` turns. Raise it, and raise `timeout_s` with it. | ## Cost Browser time costs $0.10 an hour, charged by the second. Driven over `connect_url` with Playwright, as on this page, steps are not charged: a ten minute session costs $0.017 and the longest session, one hour, costs $0.10. The session's `charged_usd` shows the amount, and the script prints it. Sending one step at a time to the session's `/actions` route costs $1 per 1,000 steps on top of the time. Google bills the Gemini tokens, and each turn sends one screenshot. The rates are on [pricing](https://specstream.ai/pricing.md). ## When the task only needs to read Computer use is for pages that need clicks: a form, a filter, a flow behind a button. For the text of a page, the [Extract API](https://specstream.ai/docs/extract-api.md) is $0.80 per 1,000 pages, with a real browser where the page needs one. For an answer from the web, the [Search API](https://specstream.ai/docs/search-api.md) is $5 per 1,000 searches of up to 10 results. Both run on the same key, and the [REST API reference](https://specstream.ai/docs/api.md) lists every call. See also [headless browsers for AI agents](https://specstream.ai/headless-browser-for-ai-agents.md) and the [Browser Use guide](https://specstream.ai/docs/integrations/browser-use.md). ## Remove Delete `gemini_agent.py` and revoke the key in your SpecStream workspace. A session that is still open ends at its `timeout_s`. To end it now, call `POST /v1/browser/sessions/{id}/release`. Questions: [support@specstream.ai](mailto:support@specstream.ai). ## More [SpecStream for Gemini Computer Use](https://specstream.ai/integrations/gemini-computer-use.md) covers what it does in Gemini Computer Use on one page. Every other tool has its guide in [Connect your tools](https://specstream.ai/docs/integrations.md). You see the most a request can cost before it starts, and the matches are paid from your balance. [Billing](https://specstream.ai/docs/billing.md) has the details. For help, email [support@specstream.ai](mailto:support@specstream.ai).