Skip to main content

Architecture Overview

This section explains WHY NPipeline is designed the way it is and WHAT happens under the hood. It covers design decisions, performance optimizations, and engineering principles that make NPipeline fast and reliable.

Ready to build pipelines? Start with Core Concepts for practical HOW TO guides.

Understanding NPipeline Architecture​

NPipeline's architecture is designed around several core principles: graph-based data flow, lazy evaluation, streaming-first design, and composability. This section breaks down each major architectural component to help you understand how system works internally.

WHY: Design Philosophy & Principles​

These sections explain the design decisions behind NPipeline:

Design Principles​

Learn the core design philosophy:

  • Separation of Concerns - How NPipeline divides responsibilities
  • Lazy Evaluation - Why data is only processed when consumed
  • Streaming First - Why NPipeline treats all data as streams
  • Composability - How components work together
  • Testability - How design enables effective testing
  • Observability - How to understand what's happening internally

Optimization Principles​

Understand how NPipeline achieves exceptional performance:

  • Plan-based execution eliminating per-item decisions
  • Zero reflection during steady state
  • ICountable for right-sized allocations
  • ValueTask optimization (up to 90% GC reduction)
  • Streaming-first design with lazy evaluation
  • Memory layout and cache efficiency

WHAT: Internal Architecture & Mechanisms​

These sections explain how NPipeline works internally:

Architectural Foundations​

Learn the fundamental architectural building blocks:

  • Graph-Based Architecture (DAGs)
  • Node Types (Source, Transform, Sink)
  • Streaming Data Model with IAsyncEnumerable<T>

Component Architecture​

Explore the major system components and their roles:

  • Pipeline Definition (IPipelineDefinition)
  • Pipeline Builder
  • Pipeline Context
  • Pipeline Runner
  • Node Execution Model

Execution Flow​

Understand how pipelines execute data:

  • Sequential Execution (default)
  • Parallel Execution patterns
  • Data flow through pipeline graph

Data Flow Details​

Deep dive into how data moves through the system:

  • How Data Pipes Work
  • Lazy Evaluation principles
  • Memory efficiency patterns

Execution Plan Caching​

Learn how NPipeline caches compiled execution plans:

  • Why Plan Caching Matters (300-500μs per run overhead)
  • How the Caching System Works
  • When Caching is Applied & When It's Disabled
  • Performance Impact (75% reduction for subsequent runs)
  • Custom Cache Implementations (Redis, Distributed, etc.)

Error Handling Architecture​

Understand error propagation and handling:

  • Error Propagation mechanisms
  • Error Containment strategies
  • Dead-Letter Handling for failed items
  • Supporting Components (Lineage, Dead Letter Sinks, State Registry)

Cancellation Model​

Learn about graceful shutdown and cancellation:

  • Token Propagation through pipeline
  • Node cancellation responsibilities
  • Graceful termination patterns

Dependency Injection Integration​

Learn how DI is integrated into NPipeline:

  • Automatic Node Resolution
  • Injecting Dependencies into Nodes
  • Service Provider Integration

Node Instantiation​

Understand how nodes are created and optimized:

  • Compiled Factory Pattern for 3-5x faster instantiation
  • Node Constructor Patterns (Parameterless, Mixed, DI-only)
  • Factory Selection and Trade-offs
  • Analyzer Support (NP9505) with automatic code fixes
  • Performance Characteristics

Performance Characteristics​

Understand performance implications:

  • Memory Usage patterns
  • Throughput characteristics
  • Scalability strategies (vertical and horizontal)

Extension Points​

Discover how to extend NPipeline:

  • Custom Nodes
  • Custom Execution Strategies
  • Custom Context Data

High-Level Architecture Diagram​

Figure: NPipeline architecture showing relationship between core components, data flow through graph-based pipeline, and supporting components

See Also​

Next Steps​