Skip to content

BrowserOperator/web-agent

BranchesTags

Repository files navigation

Web Agent - Browser Automation & Evaluation Platform

Extended kernel-images Chromium environment with Browser Operator DevTools and eval server for browser automation, testing, and AI agent evaluation.

🏗️ Architecture

This platform provides:

  • Browser Operator DevTools - Custom DevTools frontend with AI chat panel
  • Eval Server API - HTTP/WebSocket API for browser automation and evaluation
  • Headful Chrome with GUI access via WebRTC
  • Chrome DevTools Protocol for automation (Playwright, Puppeteer)
  • Screen Recording API for session capture
  • Local Docker Compose for development
  • Google Cloud Run deployment option

📋 Prerequisites

For Local Development

  1. Docker and Docker Compose installed
  2. Make utility
  3. Git with submodule access
  4. Python 3 (for running evals)

For Cloud Run Deployment

  1. Google Cloud Account with billing enabled
  2. gcloud CLI installed and authenticated
  3. All of the above

🚀 Local Development - Two Deployment Options

Option 1: Docker Compose (Recommended for Development)

Best for: Background services, docker-compose workflows, persistent containers

# 1. Initialize submodules
make init

# 2. Build Docker images (takes ~30 minutes first time)
make build

# 3. Start all services in background
make compose-up

# 4. Verify everything works
make test

Option 2: Direct Docker Run (Interactive Mode)

Best for: Interactive debugging, seeing live logs, quick testing

# 1. Initialize submodules
make init

# 2. Build Docker images (takes ~30 minutes first time)
make build

# 3. Start in interactive mode (logs to terminal)
make run

# In another terminal, verify
make test

Access Points

After starting with either make compose-up or make run, access:

Service URL Purpose
WebRTC Client http://localhost:8000 Live browser view with control
DevTools UI http://localhost:8001 Enhanced DevTools with AI chat
Eval Server API http://localhost:8080 HTTP REST API for automation
WebRTC Neko http://localhost:8081 WebRTC control interface
Eval Server WS ws://localhost:8082 WebSocket JSON-RPC API
CDP Endpoint http://localhost:9222/json Chrome DevTools Protocol
Recording API http://localhost:444/api Screen recording controls

Available Make Commands

make help              # Show all available commands
make init              # Initialize git submodules
make build             # Build images (smart caching)
make rebuild           # Force complete rebuild
make build-devtools    # Build DevTools base (~30 min)
make rebuild-devtools  # Fast rebuild with local changes
make compose-up        # Start in background
make run               # Start in interactive mode
make stop              # Stop all containers
make restart           # Restart containers
make logs              # View container logs
make test              # Run API verification test
make clean             # Clean up everything

Comparison: make run vs make compose-up

Feature make run make compose-up
Log visibility Live logs in terminal Background, use make logs
Stopping Ctrl+C or docker stop make stop or docker-compose down
Restarting Stop and run again docker-compose restart
Use case Interactive debugging Background development
Startup script run-local.sh docker-compose.yml
Lock cleanup Script cleans before start Container cleans on start
Volume mounts Defined in script Defined in compose file

Development Workflow

With Docker Compose (make compose-up):

Editing Eval Server Code:

# 1. Make changes in eval-server/nodejs/
vim eval-server/nodejs/src/api-server.js

# 2. Restart container (no rebuild needed, volume-mounted)
docker-compose restart

# 3. Test changes
make test

Editing DevTools:

# 1. Make changes in browser-operator-core/front_end/
vim browser-operator-core/front_end/panels/ai_chat/...

# 2. Rebuild DevTools only
make rebuild-devtools

# 3. Restart containers
docker-compose down && docker-compose up -d

Full Rebuild:

make rebuild        # Rebuild everything from scratch
make compose-up     # Start containers

With Direct Docker Run (make run):

Editing Eval Server Code:

# 1. Make changes in eval-server/nodejs/
vim eval-server/nodejs/src/api-server.js

# 2. Since eval-server is NOT volume-mounted in run mode, rebuild
make rebuild

# 3. Stop and restart
# Press Ctrl+C in the terminal running 'make run'
make run

Editing DevTools:

# 1. Make changes in browser-operator-core/front_end/
vim browser-operator-core/front_end/panels/ai_chat/...

# 2. Rebuild DevTools only
make rebuild-devtools

# 3. Stop and restart
# Press Ctrl+C in the terminal running 'make run'
make run

Full Rebuild:

make rebuild        # Rebuild everything from scratch
# Press Ctrl+C in the terminal running 'make run'
make run           # Start in interactive mode

Customizing Browser Data Location

With make run:

# Default: ./chromium-data
make run

# Custom location
CHROMIUM_DATA_HOST=/path/to/data make run

# Ephemeral (no persistence)
CHROMIUM_DATA_HOST="" make run

With make compose-up:

# Edit docker-compose.yml to change CHROMIUM_DATA_HOST
# Or set environment variable:
CHROMIUM_DATA_HOST=/path/to/data make compose-up

Opening URLs on Startup

With make run:

# Open specific URLs when browser starts
URLS="https://google.com https://github.com" make run

With make compose-up:

# Add URLS to docker-compose.yml environment section

Running Evaluations

# Simple test
make test

# Specific evaluation
cd evals
python3 run.py --path data/web-task-agent/flight-001.yaml --verbose

# All evaluations in a directory
python3 run.py --path data/web-task-agent/ --verbose

Troubleshooting

Container won't start (docker-compose):

# Check logs
docker logs kernel-browser-extended

# Clean restart
make stop
make clean
make build
make compose-up

Container won't start (make run):

# Stop existing container
docker stop kernel-browser-extended
docker rm kernel-browser-extended

# Clean rebuild
make clean
make rebuild
make run

Port conflicts:

# Remove existing container
docker rm -f kernel-browser-extended

# Then start with your preferred method
make compose-up  # OR make run

Lock file errors (should be automatic now): The system now automatically cleans lock files on startup. If you still see errors:

With docker-compose:

docker-compose down
rm -f ./chromium-data/user-data/Singleton*
make compose-up

With make run:

# Press Ctrl+C to stop
rm -f ./chromium-data/user-data/Singleton*
make run

Seeing stale code after changes (make run):

# Eval server code is NOT volume-mounted in run mode
# You must rebuild after code changes
make rebuild
# Press Ctrl+C in terminal running 'make run'
make run

Want to see live logs (docker-compose):

# Option 1: Follow logs
make logs

# Option 2: Switch to interactive mode
make stop
make run

🚀 Google Cloud Run Deployment

Configure Google Cloud

# Set your project ID
export PROJECT_ID="your-gcp-project-id"
gcloud config set project $PROJECT_ID

# Authenticate (if not already done)
gcloud auth login
gcloud auth application-default login

Deploy to Cloud Run

# Automated deployment (recommended)
./deployment/cloudrun/deploy.sh

# Or with custom settings
./deployment/cloudrun/deploy.sh --project your-project-id --region us-central1

Access Cloud Run Service

After deployment, you'll get URLs like:

🌐 Service Endpoints:
   Main Interface:    https://kernel-browser-xxx-uc.a.run.app
   WebRTC Client:     https://kernel-browser-xxx-uc.a.run.app/
   Chrome DevTools:   https://kernel-browser-xxx-uc.a.run.app/ws  
   Recording API:     https://kernel-browser-xxx-uc.a.run.app/api
   Health Check:      https://kernel-browser-xxx-uc.a.run.app/health

📖 Detailed Usage

WebRTC Live View

Access the main URL in your browser to get real-time Chrome access:

  • Full mouse/keyboard control
  • Copy/paste support
  • Window resizing
  • Audio streaming (experimental)

Chrome DevTools Protocol

Connect automation tools to the /ws endpoint:

// Playwright
const browser = await chromium.connectOverCDP('wss://your-service-url/ws');

// Puppeteer  
const browser = await puppeteer.connect({
  browserWSEndpoint: 'wss://your-service-url/ws',
});

Recording API

Capture screen recordings via REST API:

# Start recording
curl -X POST https://your-service-url/api/recording/start -d '{}'

# Stop recording  
curl -X POST https://your-service-url/api/recording/stop -d '{}'

# Download recording
curl https://your-service-url/api/recording/download --output recording.mp4

⚙️ Configuration

Environment Variables

Key configuration options in service.yaml:

env:
- name: ENABLE_WEBRTC
  value: "true"               # Enable WebRTC streaming
- name: WIDTH  
  value: "1024"              # Browser width
- name: HEIGHT
  value: "768"               # Browser height
- name: CHROMIUM_FLAGS
  value: "--no-sandbox..."   # Chrome launch flags
- name: NEKO_ICESERVERS
  value: '[{"urls": [...]}]' # TURN/STUN servers

Resource Limits

Default Cloud Run settings:

  • CPU: 4 cores
  • Memory: 8GB
  • Timeout: 1 hour
  • Concurrency: 1 (one browser per container)

Scaling

  • Min instances: 0 (scales to zero when unused)
  • Max instances: 10 (adjustable)
  • Cold start: ~30-60 seconds

🔧 Advanced Configuration

Custom Chrome Flags

Edit service.yaml to modify Chrome behavior:

- name: CHROMIUM_FLAGS
  value: "--user-data-dir=/home/kernel/user-data --disable-dev-shm-usage --custom-flag"

TURN Server for WebRTC

For production WebRTC, configure a TURN server:

- name: NEKO_ICESERVERS
  value: '[{"urls": ["turn:turn.example.com:3478"], "username": "user", "credential": "pass"}]'

WebArena Configuration (Optional)

The platform supports running WebArena benchmark evaluations against self-hosted test websites. This is completely optional and only needed if you're running WebArena tasks.

What is WebArena?

WebArena is a research benchmark with 812 tasks across 7 self-hosted websites (e-commerce, forums, GitLab, Wikipedia, etc.). To run these evaluations, you need to route specific domains to a custom IP address.

Quick Setup

1. Configure environment variables in evals/.env:

cd evals
cp .env.example .env
vim .env

Add:

# WebArena Infrastructure Configuration
WEBARENA_HOST_IP=172.16.55.59        # IP where WebArena sites are hosted
WEBARENA_NETWORK=172.16.55.0/24      # Network CIDR for routing

# WebArena Site URLs (optional - customize if needed)
SHOPPING=http://onestopmarket.com
SHOPPING_ADMIN=http://onestopmarket.com/admin
REDDIT=http://reddit.com
GITLAB=http://gitlab.com
WIKIPEDIA=http://wikipedia.org

2. Start container (configuration is auto-loaded):

make compose-up  # OR make run

3. Verify WebArena routing is enabled:

docker logs kernel-browser-extended | grep -i webarena

You should see:

🌐 [init] Configuring WebArena DNS mapping to 172.16.55.59...
🌐 [init] Adding route to 172.16.55.0/24 via 172.17.0.1...

4. Run WebArena evaluations:

cd evals
python3 run_webarena.py --task-id 1 --verbose

How It Works

When WEBARENA_HOST_IP is set:

  • DNS Mapping: Chromium routes WebArena domains (gitlab.com, reddit.com, etc.) to your specified IP
  • Network Routing: Container adds route to reach the WebArena network
  • Automatic: Configuration happens on container startup via scripts/init-container.sh

Without configuration (default):

  • System works normally with standard DNS resolution
  • WebArena routing is completely disabled
  • No impact on regular browser automation

Deployment-Specific IPs

You can use different IP addresses for different environments:

# Local development
WEBARENA_HOST_IP=172.16.55.59
WEBARENA_NETWORK=172.16.55.0/24

# Cloud deployment
WEBARENA_HOST_IP=34.123.45.67
WEBARENA_NETWORK=34.123.45.0/24

# Disable WebArena (default)
WEBARENA_HOST_IP=
WEBARENA_NETWORK=

See CLAUDE.md for detailed WebArena configuration documentation.

📁 Project Structure

web-agent/
├── browser-operator-core/      # Submodule: DevTools frontend source
├── kernel-images/              # Submodule: Base browser environment
├── deployment/                 # Deployment configurations
│   ├── cloudrun/               # Google Cloud Run deployment
│   │   ├── deploy.sh           # Cloud deployment script
│   │   ├── cloudbuild.yaml     # CI/CD pipeline config
│   │   ├── service.yaml        # Cloud Run service definition
│   │   ├── service-secrets.yaml # Service with Secret Manager
│   │   ├── cloudrun-wrapper.sh # Cloud Run entrypoint
│   │   ├── cloudrun-kernel-wrapper.sh # Alternative wrapper
│   │   ├── supervisord-cloudrun.conf # Supervisor for Cloud Run
│   │   └── nginx.conf          # Reverse proxy config
│   └── local/                  # Local deployment
│       └── run-local.sh        # Interactive Docker run script
├── nginx/                      # Nginx configurations
│   └── nginx-devtools.conf     # DevTools nginx config
├── scripts/                    # Utility scripts
│   ├── init-container.sh       # Auto-cleanup of lock files
│   └── test-eval-server.sh     # Eval server build test
├── supervisor/services/        # Service configs (overrides)
├── eval-server/
│   └── nodejs/                 # Eval server (use this, NOT submodule)
│       ├── src/                # API server, evaluation server, lib
│       ├── start.js            # Server entrypoint
│       └── package.json
├── evals/
│   ├── run.py                  # Python evaluation runner
│   ├── lib/judge.py            # Judge implementations
│   └── data/                   # Evaluation YAML files
├── Dockerfile.local            # Main Docker build (local dev)
├── Dockerfile.devtools         # DevTools frontend build
├── Dockerfile.cloudrun         # Cloud Run build
├── docker-compose.yml          # Local deployment config
├── Makefile                    # Build commands
├── CLAUDE.md                   # Technical documentation
└── README.md                   # This file

🐛 Troubleshooting

Local Development Issues

See the detailed troubleshooting section under Local Docker Compose Deployment above.

Common quick fixes:

# Clean restart
make stop && make clean && make build && make compose-up

# Check logs
docker logs kernel-browser-extended

# Verify services
docker exec kernel-browser-extended supervisorctl status

Cloud Run Issues

  1. Build Timeout

    # Use local build for testing
./deploy.sh --local

  2. Port Binding Errors

    • Cloud Run requires port 8080
    • nginx proxies internal services
    • Check nginx.conf for port mappings

  3. Chrome Crashes

    • Ensure --no-sandbox flag is set
    • Check memory limits (8GB minimum)
    • Verify non-root user execution

Cloud Run Debug Commands

# View service logs
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=kernel-browser" --project=$PROJECT_ID --limit=50

# Check service status
gcloud run services describe kernel-browser --region=us-central1

# Test endpoints
curl https://your-service-url/health
curl https://your-service-url/json/version

🔒 Security Considerations

  • Service runs as non-root user
  • Chrome uses --no-sandbox (required for containers)
  • WebRTC streams are not encrypted by default
  • Consider VPC/firewall rules for production
  • Use Cloud IAM for API access control

💰 Cost Estimation

Approximate Cloud Run costs:

  • CPU: $0.00002400 per vCPU-second
  • Memory: $0.00000250 per GiB-second
  • Requests: $0.40 per million requests

Example: 1 hour session ≈ $0.50-1.00

🔄 CI/CD Pipeline

The cloudbuild.yaml provides:

  1. Submodule initialization
  2. Docker image build with caching
  3. Container Registry push
  4. Cloud Run deployment
  5. Traffic routing

Build Commands

# Normal build (with cache) - recommended for development
gcloud builds submit --config deployment/cloudrun/cloudbuild.yaml

# Force rebuild without cache - use when dependencies change
gcloud builds submit --config deployment/cloudrun/cloudbuild.yaml --substitutions=_NO_CACHE=true

# Automated deployment with Twilio TURN server setup
./deployment/cloudrun/deploy.sh

Cache Control

The build system uses Docker layer caching by default to reduce build times and costs:

  • With cache: ~5-10 minutes, lower cost
  • Without cache: 30+ minutes, higher cost ($3-5 per build)

Use _NO_CACHE=true only when:

  • Dependencies have changed significantly
  • Base images need updating
  • Debugging build issues

📚 Additional Resources

🎯 API Examples

Eval Server HTTP API

# Execute browser task
curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Navigate to google.com and search for puppies",
    "url": "about:blank",
    "wait_timeout": 5000,
    "model": {
      "main_model": {
        "provider": "openai",
        "model": "gpt-4",
        "api_key": "your-api-key"
      }
    }
  }'

# Get page content
curl -X POST http://localhost:8080/page/content \
  -H "Content-Type: application/json" \
  -d '{"clientId": "test", "tabId": "tab-001", "format": "html"}'

# Capture screenshot
curl -X POST http://localhost:8080/page/screenshot \
  -H "Content-Type: application/json" \
  -d '{"clientId": "test", "tabId": "tab-001", "fullPage": false}'

WebSocket JSON-RPC API

const WebSocket = require('ws');
const ws = new WebSocket('ws://localhost:8082');

ws.on('open', () => {
  // Subscribe to evaluations
  ws.send(JSON.stringify({
    jsonrpc: '2.0',
    method: 'subscribe',
    params: { clientId: 'my-client' },
    id: 1
  }));
});

ws.on('message', (data) => {
  const response = JSON.parse(data);
  console.log('Received:', response);
});

Need help? Check CLAUDE.md for detailed technical docs or open an issue.

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

2 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Contributors 3

Languages