Product
Web Harvester
Web Harvester is an open-source, provider-agnostic web collection engine built in Go. It combines health-aware proxy rotation, authenticated proxies, adaptive rate limiting, cooldowns, runtime configuration reloads, Prom…
About
Web Harvester is an open-source, provider-agnostic web collection engine built in Go. It combines health-aware proxy rotation, authenticated proxies, adaptive rate limiting, cooldowns, runtime configuration reloads, Prometheus metrics, and Redis Streams. Built for resilient, high-throughput pipelines, it includes reproducible benchmarks for measuring performance and reliability. Clone the repo, run the benchmarks, inspect the code, and help build better web data infrastructure.
Discovery history
- Product Hunt 新品
Discuss this product