Adaptive Automation in InsurTech: Comparing Insurance Prices in Real Time

Comparing insurance prices sounds simple on the surface. In practice, it’s a complex engineering problem that combines web scraping, automation, performance optimization, and constant adaptation to third-party systems. In this article, we share how we helped Pricerobot build a real-time insurance comparison platform that aggregates car and home insurance offers from leading providers across Scandinavian markets – delivering accurate, comparable results under strict time constraints.

The Problem with Insurance Data

Unlike modern SaaS products, insurance companies rarely expose clean, public APIs for price comparison. Instead, pricing logic is buried behind:

• Multi-step forms with dynamic validation
• Session-based pricing flows
• Anti-bot and fraud prevention mechanisms
• Country-specific rules and product variations

For Pricerobot, the challenge was not just extracting data, but doing so reliably, quickly, and consistently across multiple providers and countries.

Each comparison request had to:

• Trigger parallel data collection from multiple insurers
• Normalize vastly different pricing models
• Return a comparable result set in under 2.5 minutes

Why a Naive Scraping Approach Would Fail

A single, monolithic scraper would quickly become unmaintainable. Any change by one provider could break the entire system, slow down responses, or silently return incorrect results.

The platform needed to be:

• Modular, so providers could evolve independently
• Adaptive, choosing the fastest and most reliable extraction method per provider
• Observable, with full visibility into failures and performance bottlenecks
• This led to a deliberately modular and adaptive architecture.

Modular Provider Architecture

Each insurance provider was implemented as an isolated actor, responsible for handling its own data flow, validations, and execution logic.

This approach delivered several advantages:

• Failures in one provider never impacted others
• Providers could be optimized or refactored independently
• New insurers could be onboarded without touching core logic

A centralized orchestration API handled:

• Input validation
• Provider configuration
• Parallel execution
• Aggregation of normalized results

This separation of concerns proved critical for long-term maintainability.

Adaptive Scraping and Automation Strategy

Not all providers require the same level of automation. Some insurers could be handled efficiently using:

• Direct HTTP requests
• Lightweight HTML parsing

Others required:

• Full browser automation
• Session persistence
• Captcha handling and security workarounds

The system dynamically selects the most appropriate strategy per provider. Over time, browser-based flows were selectively replaced with faster HTTP-based implementations where possible, reducing execution time without sacrificing reliability. This hybrid approach balanced speed, stability, and development effort.

Performance as a First-Class Constraint

Unlike batch scraping systems, Pricerobot operates in near real time. Performance was not a “nice to have” – it was a core product requirement. Optimizations included:

• Parallel execution across providers
• Fine-tuned timeout handling
• Provider-specific performance profiling
• Continuous refactoring of slow execution paths

The result was a system that consistently delivers full comparison results within strict time limits, even as providers change their internal flows.

Centralized Monitoring and Transparency

At scale, silent failures are more dangerous than visible ones. We built a centralized monitoring layer that:

• Collects execution results from every provider run
• Tracks timeouts, empty results, and partial failures
• Sends real-time alerts to Slack when anomalies occur

Historical execution data is stored and analyzed to:

• Detect recurring provider issues
• Identify performance regressions
• Prioritize optimization work

This transparency allows both the engineering team and the client to understand system behavior at all times.

What We Learned

A few lessons stood out clearly: 1. Isolation beats cleverness Modular provider isolation prevents cascading failures. 2. Performance must be designed, not tuned later Real-time systems require performance-first thinking from day one. 3. Adaptability is non-negotiable Third-party systems will change – your architecture must absorb that change. 4. Monitoring is part of the product Observability is essential for trust and long-term scalability.

Final Thoughts

Building a real-time insurance comparison platform is not about scraping alone. It’s about designing systems that can survive constant external change while delivering fast, accurate results to end users. By combining modular architecture, adaptive automation strategies, and deep monitoring, Pricerobot was able to scale its platform confidently across multiple markets and providers – without sacrificing performance or reliability. If you’re building data-driven products that rely on third-party systems, these principles apply far beyond insurance alone.