How to Automate Web Tasks Using Browser Automation

To automate web tasks using browser automation, you need a browser control layer (such as Playwright or Selenium), an LLM to interpret goals and decide what to click or type, and a secure execution environment to run the browser process safely. The combination of these three components — orchestrator, browser engine, and runtime — is the pattern behind modern tools like browser-use and Skyvern, and it works for a wide range of tasks: form filling, data scraping, monitoring, and multi-step data entry.

This guide walks through each layer in concrete steps, with code and workflow patterns you can adapt to your own use case.

Key Takeaways

• LLM-guided browser automation works by converting natural language goals into DOM actions (click, type, scroll, submit) using an LLM as the decision layer and Playwright or a similar engine as the executor.
• Retries and screenshot-based verification are essential for reliable automation; web UIs change and steps fail silently without them.
• Running browser agents in a sandboxed cloud environment (like Novita Agent Sandbox) isolates execution, handles resource management, and avoids polluting your local environment.
• Novita’s LLM API provides the model backend for goal interpretation and action planning; the sandbox provides the runtime where the browser actually runs.
• Different task types need different levels of LLM involvement: simple repetitive scraping rarely needs an LLM; goal-directed multi-step workflows do.

What Browser Automation Actually Does

Traditional browser automation (Selenium, Playwright) works by targeting specific DOM elements using CSS selectors, XPaths, or element IDs. You write code that says: find the button with class .submit-btn, click it. This is fast and deterministic, but it breaks the moment the page structure changes.

LLM-guided browser automation replaces hardcoded selectors with a reasoning loop:

1. Take a screenshot or extract the page’s accessibility tree.
2. Send it to an LLM with the current goal: “Find the login form and submit with these credentials.”
3. The LLM outputs an action: click(element="Sign in button").
4. Execute the action in the browser.
5. Verify the result (screenshot, DOM state) and loop until the goal is complete.

Tools like browser-use and Skyvern implement this loop. They differ in details — how they serialize page state, how they manage multi-step memory — but the core structure is the same.

The result: automation that is far more robust to page changes, can generalize across sites, and can handle tasks you’d struggle to script deterministically.

---

When to Add an LLM to the Pipeline

Not every web task needs an LLM. The choice affects cost, latency, and complexity.

Task type	LLM needed?	Reason
Scrape a fixed data table	No	Static selector + Playwright is faster and cheaper
Fill a known form with fixed fields	No	Script the selectors directly
Navigate an unknown site to find a specific data point	Yes	Page structure varies; LLM generalizes
Multi-step workflow (search → filter → extract → submit)	Yes	Requires goal tracking and dynamic decision-making
Monitor for a specific condition and react	Depends	Use LLM if the condition is semantic (e.g., “if pricing changes”)
Handle CAPTCHAs and popups automatically	Yes	Requires contextual judgment

When you do need an LLM, you want a model with strong instruction following and vision capability (to interpret screenshots). Models accessible via Novita LLM API cover both text-based and multimodal workflows.

---

Step 1: Set Up Your Execution Environment

Browser automation needs a real browser, system dependencies (Chromium binaries, display server on Linux), and process isolation. Running this on your laptop works for development, but for reliable production runs you want an isolated environment that:

• Starts fresh every run
• Has the right binaries installed
• Can run headless Chrome without a physical display
• Won’t interfere with your local system state

Novita Agent Sandbox provides this as a cloud-hosted Linux environment. Install the Python SDK:

pip install novita-sandbox

Create a sandbox and install your browser automation dependencies inside it:

import os
from novita_sandbox.code_interpreter import Sandbox

sandbox = Sandbox.create()

# Install browser-use and Playwright inside the sandbox
result = sandbox.commands.run(
    "pip install browser-use playwright && playwright install chromium"
)
print(result.stdout)

This installs the required packages once per session. The sandbox persists the installation across subsequent code blocks in the same session — no need to reinstall on each step.

If you prefer to run locally during development:

pip install browser-use playwright
playwright install chromium

Set your Novita API key (used in Step 2):

export NOVITA_API_KEY=your_api_key_here

---

Step 2: Connect an LLM API for Goal Interpretation

browser-use is compatible with any OpenAI-compatible API endpoint. Novita’s LLM API follows this interface, so you can point browser-use at Novita by setting the base URL and model name.

from langchain_openai import ChatOpenAI

# Novita AI LLM API — OpenAI-compatible endpoint
llm = ChatOpenAI(
    model="meta-llama/llama-3.3-70b-instruct",  # or any model on novita.ai/llm-api
    openai_api_base="https://api.novita.ai/v3/openai",
    openai_api_key=os.environ["NOVITA_API_KEY"],
)

For multimodal tasks where the agent interprets screenshots, use a vision-capable model:

llm = ChatOpenAI(
    model="qwen/qwen2.5-vl-72b-instruct",
    openai_api_base="https://api.novita.ai/v3/openai",
    openai_api_key=os.environ["NOVITA_API_KEY"],
)

The model you choose affects how well the agent handles ambiguous pages. Vision models can read page content directly from screenshots rather than relying on the DOM structure alone, which helps on JavaScript-heavy sites where the accessibility tree is incomplete.

---

Step 3: Write Your First Browser Automation Task

With the environment and LLM configured, define a task as a plain-language goal:

import asyncio
from browser_use import Agent

async def run_task(task_description: str):
    agent = Agent(
        task=task_description,
        llm=llm,  # from Step 2
    )
    result = await agent.run()
    return result

# Example: extract the current price of a product
task = "Go to https://example-store.com/product/laptop-x1 and find the current price. Return only the price value."

result = asyncio.run(run_task(task))
print(result)

This is the minimal working loop. The agent opens a browser, navigates to the URL, interprets the page, finds the price field, and returns it — without you writing a single CSS selector.

For multi-step tasks, describe the full workflow:

task = """
1. Go to https://example-crm.com and log in with email 'user@example.com' and password from env var CRM_PASSWORD.
2. Navigate to Contacts > Import.
3. Upload the file at /tmp/contacts.csv.
4. Confirm the import when prompted.
5. Return the number of contacts imported as shown on the confirmation screen.
"""

The agent maintains state across steps, tracks which steps have completed, and retries individual steps if they fail.

---

Step 4: Handle Retries and Failures

Web automation fails for predictable reasons: network latency, overlapping elements, dynamic content that loads after the DOM is ready, and popups that appear unexpectedly. Build retry logic around each task run.

import asyncio
from browser_use import Agent

async def run_with_retry(task: str, max_attempts: int = 3, delay: float = 2.0):
    last_error = None
    for attempt in range(1, max_attempts + 1):
        try:
            agent = Agent(task=task, llm=llm)
            result = await agent.run()
            return result
        except Exception as e:
            last_error = e
            print(f"Attempt {attempt} failed: {e}")
            if attempt < max_attempts:
                await asyncio.sleep(delay)
    raise RuntimeError(f"Task failed after {max_attempts} attempts. Last error: {last_error}")

For tasks where partial progress matters (the agent completed steps 1–3 but failed on step 4), browser-use supports checkpointing via its max_steps and step-level callbacks. Use these to log completed steps so a retry can resume from a known-good state rather than starting over.

One pattern that helps: break complex workflows into discrete sub-tasks, each with its own retry loop, rather than treating a ten-step workflow as a single retryable unit. A form submission failure on step 9 shouldn’t require re-running steps 1–8.

---

Step 5: Verify Progress with Screenshots

Screenshots are the most reliable way to verify that a task completed correctly, because DOM state and return values can diverge from what the user would actually see.

Playwright (which browser-use uses internally) provides screenshot capture:

from playwright.async_api import async_playwright

async def capture_screenshot(url: str, output_path: str = "/tmp/screenshot.png"):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        await page.wait_for_load_state("networkidle")
        await page.screenshot(path=output_path, full_page=True)
        await browser.close()
    return output_path

Integrate verification into your task loop:

async def run_with_verification(task: str):
    agent = Agent(task=task, llm=llm)

    # Capture before state
    before_screenshot = await capture_screenshot(target_url, "/tmp/before.png")

    result = await agent.run()

    # Capture after state and optionally pass to LLM for semantic check
    after_screenshot = await capture_screenshot(target_url, "/tmp/after.png")

    return result, after_screenshot

For automated pipelines, pass the after-screenshot to a vision LLM with a simple verification prompt:

# Pseudo-code for vision-based verification
verification_prompt = f"""
Look at this screenshot. The task was: "{task}"
Did the task complete successfully? Answer yes or no, and explain what you see.
"""
# Pass after_screenshot + prompt to your vision-capable LLM

This catches cases where the browser returned a success state but the actual outcome was wrong — a form appeared to submit but an error message appeared below the fold, for example.

---

Workflows for Common Task Types

Form Filling

Goal-directed form filling is where LLM-guided automation wins most clearly. Instead of mapping field names to selectors, describe what to fill:

task = """
Fill out the contact form at https://example.com/contact:
- Name: Jane Developer
- Email: jane@example.com  
- Message: Requesting a demo for enterprise pricing
Click Submit and confirm the success message appears.
"""

The agent handles field discovery, tab ordering, dropdowns, and required-field validation automatically.

Web Scraping

For scraping across multiple pages or sites with inconsistent structure, an LLM agent handles navigation that rule-based scrapers break on:

task = """
Go to https://example-jobs.com/search?q=software+engineer&location=remote
Extract the job title, company, and salary range for the first 10 results.
If pagination exists, click Next and continue until 10 results are collected.
Return results as a JSON list.
"""

For high-volume scraping of a single consistent site, direct Playwright with static selectors is faster and cheaper. Use the agent approach when the target site changes frequently or when you’re scraping across multiple sites.

Monitoring

Use a browser agent as a polling monitor that checks a semantic condition:

task = """
Go to https://example-product.com/pricing
Check if the Pro plan price has changed from $49/month.
Return the current price and whether it matches $49/month.
"""

Run this on a schedule and alert when the condition is no longer true. The LLM handles layout changes that would break a selector-based monitor silently.

Data Entry

Multi-step data entry into complex forms (CRM records, invoice systems, admin panels) is one of the most valuable uses for browser agents:

task = f"""
Log into https://crm.example.com using credentials from environment variables.
Create a new contact with:
- First name: {first_name}
- Last name: {last_name}
- Email: {email}
- Company: {company}
Save the record and return the new contact's ID from the URL.
"""

---

FAQ

What is the difference between browser-use and Skyvern for browser automation?

Both use an LLM to guide browser actions, but they differ in how they serialize page state. browser-use extracts the DOM accessibility tree and sends it as text to the LLM, which works well for text-heavy pages. Skyvern takes a screenshot-first approach, relying more heavily on vision models. In practice, choose based on your target sites: accessibility-tree-based approaches are faster and cheaper on well-structured pages; screenshot-based approaches are more robust on heavily visual or JavaScript-rendered pages. Date checked: June 2026.

Do I need a cloud sandbox to run browser automation, or can I run it locally?

You can run Playwright and browser-use locally. A cloud sandbox like Novita Agent Sandbox becomes useful when you need to run multiple agents in parallel, when you want to isolate browser processes from your local environment, or when you’re running automation in a CI/CD pipeline or production backend that doesn’t have a display server. The sandbox also gives you a consistent, reproducible environment without managing Chromium binaries yourself.

Which LLM models work best for browser automation?

Strong instruction-following and tool-use capability matters more than raw benchmark scores for browser automation. For text-DOM tasks, a fast instruction-following model (like Llama 3.3 70B via Novita LLM API) works well. For screenshot-based verification or agents that need to interpret visual page layouts, use a vision-capable model (such as Qwen2.5-VL). The right choice depends on your task type and latency budget.

How do I handle authentication for sites with 2FA or CAPTCHA?

For 2FA: provide a TOTP secret to your agent so it can generate the one-time code at runtime (using a library like pyotp). For CAPTCHAs: browser-use and Skyvern have integrations with CAPTCHA-solving services. Be aware that automated CAPTCHA solving may violate the target site’s terms of service — check before building production workflows.

What happens when a browser agent takes a wrong action and can’t recover?

browser-use supports a configurable max_steps limit and will stop and report failure if it exceeds that limit without completing the task. For production workflows, wrap each run in the retry pattern from Step 4, use screenshots to detect unexpected states, and log intermediate steps so you can diagnose failures. Dead-end states (the agent clicks something that navigates away from the target) are the most common failure mode.

Is browser automation suitable for production use at scale?

Yes, with the right architecture. Each browser instance needs isolated resources (CPU, memory, a display server on Linux). Running 50 parallel browser agents on a single machine is not realistic; distributing them across cloud sandboxes is. Novita Agent Sandbox supports concurrent sandbox instances, which is the standard pattern for scaling browser automation beyond single-machine limits.

---

Recommended Articles

• How to Build an Agent with Novita AI Sandbox, LLM Products, and Browser Use
• How to Build a Chrome Plugin Using Novita AI Sandbox
• How Agent Sandboxes Power Secure, Scalable AI Innovation