Powering Real-Time Insurance Price Comparison Across Scandinavia

Pricerobot set out to build a real-time insurance comparison platform that enables users across Scandinavian countries to find the best car and home insurance deals on the market. The core challenge was extracting accurate pricing and product data from multiple insurance providers, each with unique business rules, technical implementations, and security protections.

The system had to operate under strict performance constraints, returning normalized and comparable insurance offers within approximately 2 minutes and 30 seconds per request. At the same time, provider websites frequently changed flows, introduced new security mechanisms, or applied aggressive anti-bot protections. The challenge was to design a flexible, resilient system that could adapt to these variations while maintaining speed, reliability, and data accuracy at scale.

Lexis Solutions designed and implemented a modular, adaptive web scraping and automation platform tailored specifically for large-scale insurance comparison. Each insurance provider was isolated into its own dedicated actor, ensuring changes or failures in one provider would not impact others. A centralized API layer orchestrates requests, validates inputs, applies provider-specific configurations, and aggregates results into a unified response.

The architecture was optimized iteratively. Initial browser automation ensured stability and fast delivery, while performance-critical providers were later migrated to direct HTTP-based implementations for speed. Advanced handling of security mechanisms, captcha-solving integrations, and dynamic proxy strategies ensured consistent execution within strict time limits. This approach resulted in a scalable, maintainable system capable of evolving alongside constantly changing insurance provider platforms.

Each insurance provider is handled by a separate, isolated actor with a consistent processing flow. This modular design simplifies maintenance, enables independent optimization per provider, and prevents cascading failures when individual websites change their structure or protection mechanisms.

Depending on provider complexity, the system dynamically selects the most effective data retrieval method – ranging from direct HTTP requests and Cheerio-based parsing to full browser automation. This hybrid approach balances performance, stability, and development effort while meeting strict execution time requirements.

A dedicated monitoring pipeline collects execution results from all actors, tracks failures, timeouts, and empty results, and sends real-time notifications to Slack. Historical execution data is stored and analyzed to identify recurring issues, website changes, and optimization opportunities, providing full transparency for both the development team and the client.