Skip to main content

Architecture Overview

Prerequisites​

Before understanding NPipeline architecture, you should be familiar with:

This section explains how NPipeline works internally - design decisions, performance optimizations, and engineering principles that make NPipeline fast and reliable.

For implementation guides, see Core Concepts.

Understanding NPipeline Architecture​

NPipeline's architecture is designed around several core principles: graph-based data flow, lazy evaluation, streaming-first design, and composability. This section breaks down each major architectural component to help you understand how system works internally.

Main Architecture Sections​

Core Concepts​

Learn to fundamental architectural building blocks:

  • Graph-Based Architecture (DAGs)
  • Node Types (Source, Transform, Sink)
  • Streaming Data Model with IAsyncEnumerable<T>

Optimization Principles​

Understand how NPipeline achieves exceptional performance:

  • Plan-based execution eliminating per-item decisions
  • Zero reflection during steady state
  • ICountable for right-sized allocations
  • ValueTask optimization (up to 90% GC reduction)
  • Streaming-first design with lazy evaluation
  • Memory layout and cache efficiency

Component Architecture​

Explore of major system components and their roles:

  • Pipeline Definition (IPipelineDefinition)
  • Pipeline Builder
  • Pipeline Context
  • Pipeline Runner
  • Node Execution Model

Execution Flow​

Understand how pipelines execute data:

  • Sequential Execution (default)
  • Parallel Execution patterns
  • Data flow through pipeline graph

Data Flow Details​

Deep dive into how data moves through system:

  • How Data Pipes Work
  • Lazy Evaluation principles
  • Memory efficiency patterns

Dependency Injection Integration​

Learn how DI is integrated into NPipeline:

  • Automatic Node Resolution
  • Injecting Dependencies into Nodes
  • Service Provider Integration

Error Handling Architecture​

Understand error propagation and handling:

  • Error Propagation mechanisms
  • Error Containment strategies
  • Dead-Letter Handling for failed items
  • Supporting Components (Lineage, Dead Letter Sinks, State Registry)

Cancellation Model​

Learn about graceful shutdown and cancellation:

  • Token Propagation through pipeline
  • Node cancellation responsibilities
  • Graceful termination patterns

Performance Characteristics​

Understand performance implications:

  • Memory Usage patterns
  • Throughput characteristics
  • Scalability strategies (vertical and horizontal)

Extension Points​

Discover how to extend NPipeline:

  • Custom Nodes
  • Custom Execution Strategies
  • Custom Context Data

Design Principles​

Learn to core design philosophy:

  • Separation of Concerns
  • Lazy Evaluation
  • Streaming First
  • Composability
  • Testability
  • Observability

High-Level Architecture Diagram​

Figure: NPipeline architecture showing relationship between core components, data flow through graph-based pipeline, and supporting components

See Also​

Next Steps​