Name	Name	Last commit message	Last commit date
parent directory ..
Cargo.toml	Cargo.toml
README.md	README.md
advanced_configuration.rs	advanced_configuration.rs
anti_bots.rs	anti_bots.rs
auto_encoding.rs	auto_encoding.rs
blacklist.rs	blacklist.rs
budget.rs	budget.rs
cache.rs	cache.rs
cache_chrome_hybrid.rs	cache_chrome_hybrid.rs
cache_remote_skip_browser.rs	cache_remote_skip_browser.rs
callback.rs	callback.rs
chrome.rs	chrome.rs
chrome_remote.rs	chrome_remote.rs
chrome_remote_tls.rs	chrome_remote_tls.rs
chrome_screenshot.rs	chrome_screenshot.rs
chrome_screenshot_with_config.rs	chrome_screenshot_with_config.rs
chrome_sendable.rs	chrome_sendable.rs
chrome_single_page.rs	chrome_single_page.rs
chrome_viewport.rs	chrome_viewport.rs
chrome_web_automation.rs	chrome_web_automation.rs
concurrent_profiles.rs	concurrent_profiles.rs
configuration.rs	configuration.rs
cron.rs	cron.rs
css_scrape.rs	css_scrape.rs
debug.rs	debug.rs
depth.rs	depth.rs
download.rs	download.rs
download_to_react.rs	download_to_react.rs
encoding.rs	encoding.rs
example.rs	example.rs
gemini.rs	gemini.rs
loop.rs	loop.rs
not_a_robot.rs	not_a_robot.rs
not_a_robot_chrome_ai.rs	not_a_robot_chrome_ai.rs
not_a_robot_haiku.rs	not_a_robot_haiku.rs
openai.rs	openai.rs
openai_cache.rs	openai_cache.rs
openai_extra.rs	openai_extra.rs
openai_multi.rs	openai_multi.rs
parallel_backends.rs	parallel_backends.rs
parallel_backends_chrome.rs	parallel_backends_chrome.rs
queue.rs	queue.rs
real_world.rs	real_world.rs
remote_multimodal.rs	remote_multimodal.rs
remote_multimodal_benchmark.rs	remote_multimodal_benchmark.rs
remote_multimodal_dual.rs	remote_multimodal_dual.rs
remote_multimodal_dual_automation.rs	remote_multimodal_dual_automation.rs
remote_multimodal_listing.rs	remote_multimodal_listing.rs
remote_multimodal_multi.rs	remote_multimodal_multi.rs
remote_multimodal_quotes.rs	remote_multimodal_quotes.rs
remote_multimodal_scrape.rs	remote_multimodal_scrape.rs
rss.rs	rss.rs
scrape.rs	scrape.rs
serde.rs	serde.rs
sitemap.rs	sitemap.rs
sitemap_only.rs	sitemap_only.rs
smart.rs	smart.rs
spider_browser_cloud.rs	spider_browser_cloud.rs
spider_cloud_markdown.rs	spider_cloud_markdown.rs
subscribe.rs	subscribe.rs
subscribe_download.rs	subscribe_download.rs
subscribe_multiple.rs	subscribe_multiple.rs
transform_markdown.rs	transform_markdown.rs
url_glob.rs	url_glob.rs
url_glob_subdomains.rs	url_glob_subdomains.rs
warc.rs	warc.rs
warc_chrome.rs	warc_chrome.rs
webdriver.rs	webdriver.rs
webdriver_remote.rs	webdriver_remote.rs
webdriver_screenshot.rs	webdriver_screenshot.rs
whitelist.rs	whitelist.rs

Examples

First git clone https://github.com/spider-rs/spider.git and cd spider. Use the release flag for the best performance --release when running the examples below. It is recommended to use the headless-browser project for web crawling and scraping via a headless Docker container or by launching your own local chrome-headless-shell when using the chrome examples.

Basic

Simple concurrent crawl Simple.

cargo run --example example

Subscribe to realtime changes Subscribe.

cargo run --example subscribe

Subscribe to realtime changes Subscribe.

cargo run --example subscribe_multiple

Live handle index mutation example Callback.

cargo run --example callback

Enable log output Debug.

cargo run --example debug

Scrape the webpage with and gather html Scrape.

cargo run --example scrape

Scrape and download the html file to fs Download HTML. *Note: Enable feature flag [full_resources] to gather all files like css, jss, and etc.

cargo run --example download

Scrape and download html to react components and store to fs Download to React Component.

cargo run --example download_to_react

Crawl the page and output the links via Serde.

cargo run --example serde --features serde

Crawl links with a budget of amount of pages allowed Budget.

cargo run --example budget

Crawl links at a given cron time Cron.

cargo run --example cron

Crawl links with chrome headless rendering Chrome.

cargo run --example chrome --features chrome

Crawl links with chrome headed rendering Chrome.

cargo run --example chrome --features chrome_headed

Crawl links with chrome headless rendering remote connections Chrome.

cargo run --example chrome_remote --features chrome

Crawl links with view port configuration Chrome Viewport.

cargo run --example chrome_viewport --features chrome

Take a screenshot of a page during crawl Chrome Screenshot.

cargo run --example chrome_screenshot --features="spider/sync spider/chrome spider/chrome_store_page"

Crawl links with smart mode detection. Runs HTTP by default until Chrome Rendering is needed. Smart.

cargo run --example smart --features smart

Use different encodings for the page. Encoding.

cargo run --example encoding --features encoding

Use advanced configuration re-use. Advanced Configuration.

cargo run --example cache_chrome_hybrid --features="spider/sync spider/chrome spider/cache_chrome_hybrid"

Use chrome hybrid caching. Chrome Cache Hybrid.

cargo run --example advanced_configuration

End-to-end remote cache warm + skip-browser return path using HYBRID_CACHE_ENDPOINT (index_cache_server / hybrid_cache_server). Remote Cache Skip Browser.

HYBRID_CACHE_ENDPOINT=http://127.0.0.1:8080 cargo run --example cache_remote_skip_browser --features="spider/sync spider/chrome spider/chrome_remote_cache" -- https://example.com/

Use URL globbing for a domain. URL Globbing.

cargo run --example glob --features glob

Use URL globbing for a domain and subdomains. URL Globbing Subdomains.

cargo run --example url_glob_subdomains --features glob

Downloading files in a subscription. Subscribe Download.

cargo run --example subscribe_download

Add links to gather mid crawl. Queue.

cargo run --example queue

Use OpenAI to get custom Javascript to run in a browser. OpenAI. Make sure to set OPENAI_API_KEY=$MY_KEY as an env variable or pass it in before the script.

cargo run --example openai

OPENAI_API_KEY=replace_me_with_key cargo run --example openai

or setting multiple actions to drive the browser

OPENAI_API_KEY=replace_me_with_key cargo run --example openai_multi

or to get custom data from the GPT with JS scripts if needed.

OPENAI_API_KEY=replace_me_with_key cargo run --example openai_extra

Remote Multimodal (OpenRouter / Vision+Text)

Single page extraction from a book details page Remote Multimodal Scrape.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_scrape --features "spider/sync spider/chrome spider/agent_chrome"

Multi-page extraction by crawling from a category page Remote Multimodal Multi.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_multi --features "spider/sync spider/chrome spider/agent_chrome"

Dual-model routing (vision + text model split) Remote Multimodal Dual.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual --features "spider/sync spider/chrome spider/agent_chrome"

Dual-model multi-round automation Remote Multimodal Dual Automation.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual_automation --features "spider/sync spider/chrome spider/agent_chrome"

Quote extraction with structured JSON output Remote Multimodal Quotes.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_quotes --features "spider/sync spider/chrome spider/agent_chrome"

Listing page product extraction Remote Multimodal Listing.

OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_listing --features "spider/sync spider/chrome spider/agent_chrome"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

Examples

Basic

Remote Multimodal (OpenRouter / Vision+Text)

FilesExpand file tree

examples

Directory actions

More options

Directory actions

More options

Latest commit

History

examples

Folders and files

parent directory

README.md

Examples

Basic

Remote Multimodal (OpenRouter / Vision+Text)