Advanced / Advanced Features

Advanced Features

Power user features for complex scraping scenarios.

API Discovery & Replay

Automatically detect and replay API calls for faster, more reliable data extraction.

Many websites load data through internal APIs. ScraperQ can detect these APIs and use them directly instead of parsing HTML.

How API Discovery Works

  1. Monitor network requests — ScraperQ watches XHR/Fetch calls as you browse
  2. Identify data APIs — AI analyzes responses to find endpoints returning structured data
  3. Analyze patterns — Detect pagination parameters, items arrays, and response structures
  4. Build template — Create an API-based template that calls the endpoint directly

Credit Cost

API Discovery consumes 8 credits and only charges if a valid API endpoint is found.

API discovery panel

Replace with actual extension screenshot

ScraperQ analyzing API endpoints

API Replay Mode

Replay discovered API calls with proper authentication.

When an API requires authentication (tokens, cookies), ScraperQ can replay the request with the correct headers:

1

Capture Auth Headers

When you're logged into a website, ScraperQ captures the authentication headers from the original API call.

2

Configure Replay

In the template settings, enable "API Replay Mode". ScraperQ will use the captured auth headers for each request.

3

Run with Replay

When running the template, ScraperQ:

  • Refreshes authentication if expired
  • Replays API calls with valid tokens
  • Handles rate limiting automatically

Session Dependency

API Replay works best when your browser session is active. For long-running jobs, ScraperQ will refresh tokens automatically when possible.

Custom CSS Selectors

Write custom selectors for precise data extraction.

For complex scraping scenarios, you can write custom CSS selectors:

Selector Examples

SelectorDescription
.product-titleAll elements with class "product-title"
#product-123Element with ID "product-123"
div[data-testid="product"]Div with data-testid attribute
ul > li:nth-child(2)Second list item in a ul
a[href*="product"]Links containing "product" in href

Deduplication

Avoid duplicate entries in your scraped data.

When scraping multiple pages, duplicates can occur. ScraperQ's deduplication feature prevents this:

How to Enable

1

Select Deduplication Field

Choose a unique field to identify duplicates (e.g., URL, product ID, title).

2

Configure Behavior

ScraperQ will automatically skip items with duplicate values in the selected field.

Best Practices

  • Use URL or unique ID as the deduplication key
  • Enable deduplication for multi-page scraping jobs
  • Check deduplication stats in the job summary

Humanized Crawl Navigation

Scrape websites while mimicking human behavior.

ScraperQ includes built-in anti-detection features:

Random Delays

Configurable wait times between requests to avoid triggering rate limits.

User Agent Rotation

Rotate between browser user agents to appear as different visitors.

Request Throttling

Automatic rate limiting to stay within acceptable request limits.

Responsible Scraping

Always respect website terms of service and robots.txt. ScraperQ is designed for authorized data extraction only.

Test Case: API-Based Scraping

Example of using API Discovery for faster data extraction.

Scenario: Scrape E-commerce Product Data via API

  1. Navigate to an e-commerce product listing page
  2. Open ScraperQ and click "Create Template"
  3. Enable "API Discovery" option
  4. Let ScraperQ analyze network requests
  5. ScraperQ finds the product listing API endpoint
  6. Review and approve the API template
  7. Run the template — data extracted via API (faster than HTML parsing)

Expected Result: Faster data extraction (API calls vs HTML parsing), complete structured data, reliable pagination.

← Export Options Back to Installation →