Web Scraping

The library provides enterprise web scraping with automatic fallback chains and bot protection bypass.

Quick Start

typescript

import { Connector, ScrapeProvider, Services } from 'oneringai';

// Create ZenRows connector
Connector.create({
  name: 'zenrows',
  serviceType: Services.Zenrows,
  auth: { type: 'api_key', apiKey: process.env.ZENROWS_API_KEY! },
  baseURL: 'https://api.zenrows.com/v1',
});

// Create scrape provider
const scraper = ScrapeProvider.create({ connector: 'zenrows' });

// Scrape a URL
const result = await scraper.scrape('https://example.com', {
  includeMarkdown: true,
  includeLinks: true,
});

if (result.success) {
  console.log(result.result?.title);
  console.log(result.result?.content);
  console.log(result.finalUrl);
}

ZenRows Provider

ZenRows provides enterprise-grade scraping with:

JavaScript rendering for SPAs
Premium proxies (residential IPs)
Anti-bot and CAPTCHA bypass
Markdown conversion
Screenshot capture

typescript

import { ScrapeProvider } from 'oneringai';

const scraper = ScrapeProvider.create({ connector: 'zenrows' });

// Full control with ZenRows options
const result = await scraper.scrape('https://protected-site.com', {
  includeMarkdown: true,
  includeScreenshot: true,
  vendorOptions: {
    jsRender: true,           // Enable JS rendering (default: true)
    premiumProxy: true,       // Use residential IPs (default: true)
    wait: 5000,               // Wait 5s before scraping
    waitFor: '.content',      // Wait for CSS selector
    device: 'mobile',         // Mobile user agent
    proxyCountry: 'us',       // Use US proxies
    autoparse: true,          // Auto-structure data
  },
});

Using web_scrape Tool with Agent (ConnectorTools)

The web_scrape tool is available via ConnectorTools. It tries native fetch first, then falls back to the bound scrape provider:

typescript

import { Agent, Connector, ConnectorTools, tools } from 'oneringai';

// Create scrape connector
Connector.create({
  name: 'zenrows',
  serviceType: 'zenrows',
  auth: { type: 'api_key', apiKey: process.env.ZENROWS_API_KEY! },
  baseURL: 'https://api.zenrows.com/v1',
});

const scrapeTools = ConnectorTools.for('zenrows');

const agent = Agent.create({
  connector: 'openai',
  model: 'gpt-4.1',
  tools: [tools.webFetch, ...scrapeTools],
});

// Agent uses automatic fallback: native → API
await agent.run('Scrape https://example.com and summarize');

Tool Parameters

The web_scrape tool accepts:

url (required) — URL to scrape
timeout — Timeout in milliseconds (default: 30000)
includeHtml — Include raw HTML (default: false)
includeMarkdown — Convert to markdown (recommended for LLMs)
includeLinks — Extract links
waitForSelector — Wait for CSS selector (for JS-heavy sites)

Note: The tool automatically detects available scrape connectors by serviceType. Scraping strategy is handled internally - the tool will use the best available method.

Best Practices

Configure a connector - Set up ZenRows or similar for protected sites
Request markdown - Cleaner output for LLM processing
Handle errors - Check result.success and result.error
Use waitForSelector - For JavaScript-heavy sites that need time to render