Nodes: Source, Transform, Sink, and Specialized
Nodes are the fundamental building blocks of NPipeline pipelines. Each node type serves a specific purpose in the data processing workflow, from generating data sources to consuming final results.
Node Types
- Source Nodes - Generate or fetch data from external systems
- Transform Nodes - Process data item by item using
ITransformNode<TIn, TOut> - Stream Transform Nodes - Process entire data streams using
IStreamTransformNode<TIn, TOut> - Sink Nodes - Consume and finalize data at the end of your pipeline
Specialized Node Types
- Aggregation - Combine multiple items into aggregated results
- Batching - Group items into batches for efficient processing
- Branch - Split data flows into multiple paths
- Join - Merge data from multiple input streams
- Time-Windowed Join - Join data with temporal constraints
- Lookup - Enrich data by querying external sources
- Tap - Monitor data without modifying it
Choosing the Right Node Type
Selecting the appropriate node type is crucial for building efficient and maintainable pipelines:
| Use Case | Recommended Node Type | Key Benefit |
|---|---|---|
| Simple data transformation | TransformNode<TIn, TOut> | One-to-one mapping with minimal overhead |
| Stream-based operations | IStreamTransformNode<TIn, TOut> | Batching, unbatching, windowing, or stream cardinality changes |
| Data enrichment | TransformNode<TIn, TOut> | Per-item lookups with async service calls |
| High-throughput scenarios | ValueTaskTransform<TIn, TOut> | Synchronous operations with zero allocation |
| Complex workflows | Combination of node types | Each node serves its specific purpose |
Table 1: Node Type Selection Guide
Next Steps
- Node Definition Structure - Understanding the nested configuration structure of NodeDefinition
- Transform Nodes - Learn implementation details and patterns
- Source Nodes - Discover how to create data sources
- Sink Nodes - Understand data consumption patterns