Case Study
Real-time Patient Flow Aggregation from 26 Hospitals
A centralized data engineering solution that scrapes and normalizes live patient queue data from 26 different hospital emergency departments. Provides real-time visibility into patient flow for the Regional Command Center, enabling load balancing and transfer coordination.
The Regional Command Center needed real-time visibility into patient queue data across 26 emergency departments. Each hospital used different legacy queue management software with no standardized API access, inconsistent data formats, and varying field naming conventions. The government required centralized monitoring to coordinate patient transfers and balance loads across the healthcare network.
I designed and built the ED-Q System end-to-end - a centralized patient flow aggregation platform using real-time web scraping. The system uses Puppeteer to extract live queue data from 26 different hospital interfaces, normalizes the heterogeneous data into a unified schema, and pushes updates via WebSocket to the RCC dashboard for operator visibility.
26
Hospitals Scraped
Real-time data extraction
99.9%
Data Accuracy
With validation pipelines
~1 min
Update Frequency
Near real-time
100%
System Coverage
All ED queues monitored
The ED-Q System uses a distributed scraping architecture with Node.js worker pools for parallel data extraction. Each hospital has a dedicated schema mapping configuration to normalize its unique data format. Puppeteer with stealth plugins handles JavaScript-rendered queue interfaces. Data flows through validation pipelines before being pushed via WebSocket to the RCC dashboard. The system includes automatic retry logic, session management, and comprehensive error alerting.
Challenge
Scraping 26 different hospital queue systems with unique interfaces
Solution
Created a flexible schema mapping system allowing per-hospital configuration of field extractors, selectors, and data transformations. Each hospital's unique interface is handled by its configuration file while sharing the core scraping engine.
Challenge
Maintaining data accuracy without standardized API access
Solution
Implemented multi-stage validation pipelines checking data types, value ranges, and consistency. Added automated health checks comparing current vs. historical patterns to detect anomalies and scraping failures.
Challenge
Handling rate limiting and avoiding detection
Solution
Used Puppeteer stealth plugins, randomized delays, and request throttling to avoid triggering anti-bot measures. Implemented intelligent scheduling to distribute load and maintain sustainable scraping rates.
Real-time healthcare coordination platform for 26 hospitals with live dashboards, STEMI/Stroke/Trauma portals, intelligent triage ticketing, and 11-KPI compliance analytics.
UNFCCC - Automation Engineer. Chrome extension with Node.js and Puppeteer that replaced manual stream monitoring with automated quality checks and fast alerts.