# Computer use (/docs/use-cases/computer-use)

<!-- agent-signals: reading_time_min: 6 · est_tokens: 3345 · updated: 2026-07-30 -->
Related: [Coding Agents](/docs/use-cases/coding-agents.md), [GitHub Actions CI/CD](/docs/use-cases/ci-cd.md), [Run Kubernetes with k3s](/docs/use-cases/k3s.md), [Cloud browser](/docs/use-cases/remote-browser.md)

Computer use agents interact with graphical desktops the same way a human would — viewing the screen, clicking, typing, and scrolling. E2B provides the sandboxed desktop environment where these agents operate safely, with [VNC](https://en.wikipedia.org/wiki/Virtual_Network_Computing) streaming for real-time visual feedback.

For a complete working implementation, see [E2B Surf](https://github.com/e2b-dev/surf) — an open-source computer use agent you can try via the [live demo](https://surf.e2b.dev).

## How it works [#how-it-works]

The computer use agent loop follows this pattern:

1. **User sends a command** — e.g., "Open Firefox and search for AI news"
2. **Agent creates a desktop sandbox** — an Ubuntu 22.04 environment with [XFCE](https://xfce.org/) desktop and pre-installed applications
3. **Agent takes a screenshot** — captures the current desktop state via E2B Desktop SDK
4. **LLM analyzes the screenshot** — a vision model (e.g., [OpenAI Computer Use API](https://developers.openai.com/api/docs/guides/tools-computer-use)) decides what action to take
5. **Action is executed** — click, type, scroll, or keypress via E2B Desktop SDK
6. **Repeat** — new screenshot is taken and sent back to the LLM until the task is complete

## Install the E2B Desktop SDK [#install-the-e2b-desktop-sdk]

The [E2B Desktop](https://github.com/e2b-dev/desktop) SDK gives your agent a full Linux desktop with mouse, keyboard, and screen capture APIs.

<CodeGroup>
  <CodeBlockTabs defaultValue="JavaScript & TypeScript" groupId="javascript-typescript+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="JavaScript & TypeScript">
        JavaScript & TypeScript
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="JavaScript & TypeScript">
      ```bash  
      npm i @e2b/desktop
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Python">
      ```bash  
      pip install e2b-desktop
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

## Core implementation [#core-implementation]

The following snippets are adapted from [E2B Surf](https://github.com/e2b-dev/surf).

### Setting up the sandbox [#setting-up-the-sandbox]

Create a desktop sandbox and start VNC streaming so you can view the desktop in a browser.

<CodeGroup>
  <CodeBlockTabs defaultValue="JavaScript & TypeScript" groupId="javascript-typescript+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="JavaScript & TypeScript">
        JavaScript & TypeScript
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="JavaScript & TypeScript">
      ```typescript  
      import { Sandbox } from '@e2b/desktop'

      // Create a desktop sandbox with a 5-minute timeout
      const sandbox = await Sandbox.create({
        resolution: [1024, 720],
        dpi: 96,
        timeoutMs: 300_000,
      })

      // Start VNC streaming for browser-based viewing
      await sandbox.stream.start()
      const streamUrl = sandbox.stream.getUrl()
      console.log('View desktop at:', streamUrl)
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Python">
      ```python  
      from e2b_desktop import Sandbox

      # Create a desktop sandbox with a 5-minute timeout
      sandbox = Sandbox.create(
          resolution=(1024, 720),
          dpi=96,
          timeout=300,
      )

      # Start VNC streaming for browser-based viewing
      sandbox.stream.start()
      stream_url = sandbox.stream.get_url()
      print("View desktop at:", stream_url)
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

### Executing desktop actions [#executing-desktop-actions]

The E2B Desktop SDK maps directly to mouse and keyboard actions. Here's how Surf translates LLM-returned actions into desktop interactions.

<CodeGroup>
  <CodeBlockTabs defaultValue="JavaScript & TypeScript" groupId="javascript-typescript+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="JavaScript & TypeScript">
        JavaScript & TypeScript
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="JavaScript & TypeScript">
      ```typescript  
      import { Sandbox } from '@e2b/desktop'

      const sandbox = await Sandbox.create({ timeoutMs: 300_000 })

      // Mouse actions
      await sandbox.leftClick(500, 300)
      await sandbox.rightClick(500, 300)
      await sandbox.doubleClick(500, 300)
      await sandbox.middleClick(500, 300)
      await sandbox.moveMouse(500, 300)
      await sandbox.drag([100, 200], [400, 500])

      // Keyboard actions
      await sandbox.write('Hello, world!')  // Type text
      await sandbox.press('Enter')          // Press a key

      // Scrolling
      await sandbox.scroll('down', 3)  // Scroll down 3 ticks
      await sandbox.scroll('up', 3)    // Scroll up 3 ticks

      // Screenshots
      const screenshot = await sandbox.screenshot()  // Returns Buffer

      // Run terminal commands
      await sandbox.commands.run('ls -la /home')
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Python">
      ```python  
      from e2b_desktop import Sandbox

      sandbox = Sandbox.create(timeout=300)

      # Mouse actions
      sandbox.left_click(500, 300)
      sandbox.right_click(500, 300)
      sandbox.double_click(500, 300)
      sandbox.middle_click(500, 300)
      sandbox.move_mouse(500, 300)
      sandbox.drag([100, 200], [400, 500])

      # Keyboard actions
      sandbox.write("Hello, world!")  # Type text
      sandbox.press("Enter")          # Press a key

      # Scrolling
      sandbox.scroll("down", 3)  # Scroll down 3 ticks
      sandbox.scroll("up", 3)    # Scroll up 3 ticks

      # Screenshots
      screenshot = sandbox.screenshot()  # Returns bytes

      # Run terminal commands
      sandbox.commands.run("ls -la /home")
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

### Agent loop [#agent-loop]

The core loop takes screenshots, sends them to an LLM, and executes the returned actions on the desktop. This is a simplified version of how [Surf](https://github.com/e2b-dev/surf) drives the computer use cycle.

<CodeGroup>
  <CodeBlockTabs defaultValue="JavaScript & TypeScript" groupId="javascript-typescript+python">
    <CodeBlockTabsList>
      <CodeBlockTabsTrigger value="JavaScript & TypeScript">
        JavaScript & TypeScript
      </CodeBlockTabsTrigger>

      <CodeBlockTabsTrigger value="Python">
        Python
      </CodeBlockTabsTrigger>
    </CodeBlockTabsList>

    <CodeBlockTab value="JavaScript & TypeScript">
      ```typescript  
      import { Sandbox } from '@e2b/desktop'

      const sandbox = await Sandbox.create({
        resolution: [1024, 720],
        timeoutMs: 300_000,
      })
      await sandbox.stream.start()

      while (true) {
        // 1. Capture the current desktop state
        const screenshot = await sandbox.screenshot()

        // 2. Send screenshot to your LLM and get the next action
        //    (use OpenAI Computer Use, Anthropic Claude, etc.)
        const action = await getNextActionFromLLM(screenshot)

        if (!action) break // LLM signals task is complete

        // 3. Execute the action on the desktop
        switch (action.type) {
          case 'click':
            await sandbox.leftClick(action.x, action.y)
            break
          case 'type':
            await sandbox.write(action.text)
            break
          case 'keypress':
            await sandbox.press(action.keys)
            break
          case 'scroll':
            await sandbox.scroll(
              action.scrollY < 0 ? 'up' : 'down',
              Math.abs(action.scrollY)
            )
            break
          case 'drag':
            await sandbox.drag(
              [action.startX, action.startY],
              [action.endX, action.endY]
            )
            break
        }
      }

      await sandbox.kill()
      ```
    </CodeBlockTab>

    <CodeBlockTab value="Python">
      ```python  
      from e2b_desktop import Sandbox

      sandbox = Sandbox.create(
          resolution=(1024, 720),
          timeout=300,
      )
      sandbox.stream.start()

      while True:
          # 1. Capture the current desktop state
          screenshot = sandbox.screenshot()

          # 2. Send screenshot to your LLM and get the next action
          #    (use OpenAI Computer Use, Anthropic Claude, etc.)
          action = get_next_action_from_llm(screenshot)

          if not action:
              break  # LLM signals task is complete

          # 3. Execute the action on the desktop
          if action.type == "click":
              sandbox.left_click(action.x, action.y)
          elif action.type == "type":
              sandbox.write(action.text)
          elif action.type == "keypress":
              sandbox.press(action.keys)
          elif action.type == "scroll":
              direction = "up" if action.scroll_y < 0 else "down"
              sandbox.scroll(direction, abs(action.scroll_y))
          elif action.type == "drag":
              sandbox.drag(
                  [action.start_x, action.start_y],
                  [action.end_x, action.end_y],
              )

      sandbox.kill()
      ```
    </CodeBlockTab>
  </CodeBlockTabs>
</CodeGroup>

The `getNextActionFromLLM` / `get_next_action_from_llm` function is where you integrate your chosen LLM. See [Connect LLMs to E2B](/docs/quickstart/connect-llms) for integration patterns with OpenAI, Anthropic, and other providers.

## Related guides [#related-guides]

<CardGroup cols="3">
  <Card title="Desktop template" icon="<svg xmlns=&#x22;http://www.w3.org/2000/svg&#x22; viewBox=&#x22;0 0 24 24&#x22; fill=&#x22;none&#x22;><path d=&#x22;M14 21H16M14 21C13.1716 21 12.5 20.3284 12.5 19.5V17L12 17M14 21H10M10 21H8M10 21C10.8284 21 11.5 20.3284 11.5 19.5V17L12 17M12 17V21&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/><path d=&#x22;M16 3H8C5.17157 3 3.75736 3 2.87868 3.87868C2 4.75736 2 6.17157 2 9V11C2 13.8284 2 15.2426 2.87868 16.1213C3.75736 17 5.17157 17 8 17H16C18.8284 17 20.2426 17 21.1213 16.1213C22 15.2426 22 13.8284 22 11V9C22 6.17157 22 4.75736 21.1213 3.87868C20.2426 3 18.8284 3 16 3Z&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/></svg>" href="/docs/template/examples/desktop">
    Build desktop sandboxes with Ubuntu, XFCE, and VNC streaming
  </Card>

  <Card title="Connect LLMs" icon="<svg xmlns=&#x22;http://www.w3.org/2000/svg&#x22; viewBox=&#x22;0 0 24 24&#x22; fill=&#x22;none&#x22;><path d=&#x22;M16.998 7.12652C17.3182 7.04393 17.654 7 18 7C20.2091 7 22 8.79086 22 11C22 13.2091 20.2091 15 18 15C17.6451 15 17.3009 14.9538 16.9733 14.867M16.998 7.12652C16.9993 7.08451 17 7.04233 17 7C17 4.79086 15.2091 3 13 3C11.0824 3 9.47994 4.34939 9.09041 6.15043M16.998 7.12652C16.9769 7.80763 16.7854 8.44584 16.4649 9M16.9733 14.867C16.9909 14.7472 17 14.6247 17 14.5C17 13.2905 16.1411 12.2816 15 12.05M16.9733 14.867C16.7957 16.0737 15.756 17 14.5 17H14C11.7909 17 10 18.7909 10 21M9.09041 6.15043C8.74377 6.05243 8.37801 6 8 6C5.79086 6 4 7.79086 4 10C4 10.3886 4.05542 10.7643 4.15878 11.1195M9.09041 6.15043C10.1015 6.43625 10.9498 7.10965 11.4649 8M4.15878 11.1195C2.9114 11.4832 2 12.6352 2 14C2 15.6569 3.34315 17 5 17C6.30622 17 7.41746 16.1652 7.82929 15M4.15878 11.1195C4.24921 11.4303 4.37632 11.7255 4.53513 12&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/><path d=&#x22;M11.8361 11.7435C11.3257 12.2353 10.453 12.3202 9.70713 11.9008C8.9612 11.4814 8.58031 10.6917 8.73535 10&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/></svg>" href="/docs/quickstart/connect-llms">
    Integrate AI models with sandboxes using tool calling
  </Card>

  <Card title="Sandbox lifecycle" icon="<svg xmlns=&#x22;http://www.w3.org/2000/svg&#x22; viewBox=&#x22;0 0 24 24&#x22; fill=&#x22;none&#x22;><path d=&#x22;M4 2V5.13219C4 5.42605 4.36724 5.55908 4.55527 5.33333C6.3854 3.2875 9.04499 2 12.0051 2C17.5251 2 22 6.47715 22 12C22 15.9582 19.7015 19.3793 16.367 21&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/><path d=&#x22;M11.7347 22.0001C12.2016 22.0001 12.6611 21.9688 13.1111 21.9084M2.26537 8.66675C2.15297 9.06394 2.06477 9.46536 2 9.86901M2.03457 13.5381C2.10487 13.9381 2.19644 14.3343 2.30852 14.7245M3.83292 17.9963C4.07124 18.3497 4.3296 18.69 4.6071 19.0147M7.42857 21.3607C7.78228 21.5632 8.15042 21.7464 8.53228 21.9084&#x22; stroke=&#x22;currentColor&#x22; stroke-linecap=&#x22;round&#x22; stroke-linejoin=&#x22;round&#x22; stroke-width=&#x22;1.5&#x22;/></svg>" href="/docs/sandbox">
    Create, manage, and control sandbox lifecycle
  </Card>
</CardGroup>