Common Eventstream Patterns in Fabric

Eventstreams are Microsoft Fabric's native streaming ingestion engine, capturing real-time data from event sources and routing it to analytics destinations with in-flight transformations. Unlike batch ETL where data arrives in periodic loads, Eventstreams process events as they occur—enabling dashboards that update in seconds, alerts that fire instantly when thresholds are breached, and analytics that reflect the current state of your business rather than yesterday's snapshot. Understanding common Eventstream patterns is essential for designing real-time analytics architectures that are reliable, cost-efficient, and maintainable at scale.

Eventstream Architecture Overview

An Eventstream consists of three components that form a processing pipeline:

| Component | Role | Examples | |---|---|---| | Sources | Ingest events into the stream | Azure Event Hubs, IoT Hub, Custom App, Sample Data | | Operators | Transform events in-flight | Filter, Manage Fields, Aggregate, Group By, Union | | Destinations | Route processed events to storage/analytics | KQL Database, Lakehouse, Eventhouse, Custom App, Reflex |

Events flow left to right through the stream: sources produce events, operators transform them, and destinations consume the results. A single Eventstream can have multiple sources and multiple destinations, enabling complex fan-in and fan-out topologies from a single configuration.

Source Integration Patterns

Azure Event Hubs (High-Throughput Enterprise)

Event Hubs is the primary source for enterprise streaming scenarios. It handles millions of events per second with partitioned, ordered delivery:

• Partition strategy: Match Event Hub partition count to your throughput requirements. Start with 4-8 partitions for moderate workloads, scale to 32+ for high-throughput scenarios. Each partition delivers events in order.
• Consumer groups: Create a dedicated consumer group for each Eventstream connection. Sharing consumer groups between Eventstreams causes checkpoint conflicts and missed events.
• Serialization: Use Avro or JSON serialization. Avro provides schema evolution support and compact binary encoding. JSON is simpler but larger and schema-less.
• Connection setup: Provide the Event Hub namespace, hub name, shared access policy name, and key. Use a policy with Listen permission only—the Eventstream does not need Send or Manage permissions.

IoT Hub (Device Telemetry)

IoT Hub extends Event Hub with device management capabilities:

• Device-to-cloud messages: Telemetry data flows through the built-in Event Hub endpoint. Connect the Eventstream to IoT Hub's Event Hub-compatible endpoint.
• Device twin changes: Route device twin change notifications to capture device state updates (firmware versions, configuration changes, health status).
• Message routing: Use IoT Hub message routing to pre-filter messages before they reach the Eventstream, reducing processing volume and cost.

Custom Application Sources

For applications that generate events outside Azure messaging services, use the Custom App source:

• REST API ingestion: The Eventstream exposes an HTTPS endpoint that accepts JSON payloads. Applications POST events directly—no SDK required.
• SDK integration: Use the Azure Event Hubs SDK (available in Python, .NET, Java, JavaScript) for higher-throughput programmatic ingestion with batching and retry logic.
• Webhook patterns: Configure third-party SaaS applications (Salesforce, Stripe, GitHub) to send webhooks to the Eventstream's REST endpoint for real-time integration.

In-Flight Processing Patterns

Pattern 1: Filter and Route

The most common pattern filters a high-volume stream into targeted substreams, each routed to a different destination:

• Source: All IoT device telemetry (temperature, humidity, vibration, power consumption)
• Filter 1: Temperature > threshold → KQL Database for real-time alerting
• Filter 2: All events → Lakehouse for historical analysis
• Filter 3: Power consumption events only → Aggregate by device per hour → Separate KQL table for energy dashboards

This pattern reduces storage costs by routing only relevant events to expensive hot-path analytics while archiving everything to cost-effective Lakehouse storage.

Pattern 2: Windowed Aggregation

Time-window aggregations convert raw event streams into summarized metrics:

| Window Type | Behavior | Use Case | |---|---|---| | Tumbling | Fixed-size, non-overlapping intervals (e.g., every 5 minutes) | Periodic summaries: average temperature per 5-minute window | | Hopping | Fixed-size, overlapping intervals (e.g., 10-minute window every 5 minutes) | Smoothed metrics: rolling average that updates more frequently than the window size | | Session | Dynamic windows based on activity gaps (e.g., close window after 30 seconds of inactivity) | User session analysis: group events into logical sessions |

Configure window aggregations in the Eventstream operator panel by selecting the window type, duration, and aggregation function (COUNT, SUM, AVG, MIN, MAX). The output is a new event per window containing the aggregated result, group key, and window start/end timestamps.

Pattern 3: Enrichment with Reference Data

Add context to streaming events by joining with reference data:

• Device metadata: Enrich device telemetry with device location, owner, and maintenance schedule from a reference table
• Customer information: Add customer name, segment, and account tier to transaction events
• Geolocation: Convert IP addresses or coordinates to city/state/country names

Reference data is loaded from Lakehouse tables or KQL databases. The Eventstream performs a lookup join for each incoming event, appending the reference columns to the output. Keep reference data small (under 100MB) and relatively static for optimal performance.

Pattern 4: Fan-In (Multi-Source Merge)

Combine events from multiple sources into a single unified stream:

• Source 1: Web application clickstream from Event Hub A
• Source 2: Mobile app events from Event Hub B
• Source 3: Backend API events from Custom App source
• Union operator: Merge all three into a single stream with a common schema
• Destination: Unified KQL database for cross-platform analytics

Use the Manage Fields operator after the Union to normalize field names across sources (e.g., rename "userId" from source 1 and "user_id" from source 2 to a common "user_identifier" field).

Destination Architecture Patterns

Hot Path: KQL Database / Eventhouse

Route events that need immediate querying to a KQL database (or Eventhouse for larger workloads):

• Query latency: Sub-second for KQL queries on ingested data
• Retention: Configure retention policies (default 365 days) to automatically purge old data
• Dashboards: Connect Real-Time Dashboards directly to KQL tables for auto-refreshing visualizations
• Alerts: Use Data Activator (Reflex) connected to KQL queries to trigger alerts when conditions are met

Warm Path: Lakehouse

Route events for historical analysis and batch processing to a Lakehouse:

• Delta format: Events land as Delta tables, enabling time travel, schema evolution, and ACID transactions
• Direct Lake: Build Power BI semantic models using Direct Lake mode for near-real-time reporting without data duplication
• Data science: Use Spark notebooks to train ML models on historical event data and score new events in real time
• Partitioning: Configure the Lakehouse destination to partition by date for optimal query performance on time-range filters

Cold Path: OneLake Archive

For compliance and long-term retention, events flow to OneLake storage with lifecycle policies:

• Cost optimization: Move data older than 90 days to cool storage tier
• Compliance: Immutable storage policies prevent deletion during retention periods
• Deep analysis: Access archived data with Spark for yearly trend analysis or forensic investigation

Monitoring and Troubleshooting

Monitor Eventstream health through the Fabric workspace:

• Throughput metrics: Events per second ingested, processed, and delivered to each destination
• Latency metrics: End-to-end time from source event generation to destination delivery
• Error counts: Failed events with error details (serialization failures, destination write errors, schema mismatches)
• Backlog: Number of events waiting to be processed—a growing backlog indicates insufficient capacity

Common issues and resolutions:

| Symptom | Cause | Resolution | |---|---|---| | Events arriving late | Source-side batching or network latency | Reduce batch size on Event Hub producer | | Missing events | Consumer group conflict | Ensure dedicated consumer group per Eventstream | | Schema errors | Source schema changed without updating Eventstream | Update Eventstream schema mapping to match new source format | | Destination write failures | KQL database ingestion throttling | Scale Eventhouse capacity or reduce event volume |

Related Resources

• Real-Time Analytics in Fabric
• Fabric Real-Time Intelligence
• Data Activator Alerts
• Microsoft Fabric Services