Skip to main content

Connectors Overview

Connectors are pre-built nodes that make it easy to read data from and write data to external systems. They are specialized ISourceNode and ISinkNode implementations that handle the specifics of communicating with systems like databases, file formats, message queues, and cloud services.

Using connectors, you can quickly assemble pipelines that integrate with your existing infrastructure without having to write boilerplate code for file I/O or network communication.

Core Concepts​

Storage Abstraction​

All connectors work through the IStorageProvider abstraction, which enables them to work with multiple backend systems:

  • Storage Provider Interface - Learn about the abstraction layer that powers connectors
    • Works with filesystems, cloud storage (S3, Azure), databases, and custom backends
    • Unified API for read, write, delete, list, and metadata operations
    • Built-in support for filesystem with resilient directory traversal

Available Connectors​

The following connectors are available:

  • CSV: Read from and write to Comma-Separated Values (CSV) files.
    • Works with any storage backend via the IStorageProvider abstraction

General Usage Pattern​

Most source connectors are added to a pipeline using AddSource(), and sink connectors are added using AddSink(). They often require some configuration, such as a file path or a connection string, which is passed to their constructor.

// Example of using a source and sink connector
var pipeline = new PipelineBuilder()
    // Read data from a source connector
    .AddSource("user_source", new CsvSourceNode<User>(StorageUri.FromFilePath("users.csv")))

    // ... add transforms ...

    // Write data to a sink connector
    .AddSink("summary_sink", new CsvSinkNode<UserSummary>(StorageUri.FromFilePath("summaries.csv")), "summarizer")
    .Build();

Note: NPipeline uses a storage abstraction layer that requires StorageUri objects instead of plain file paths. Use StorageUri.FromFilePath() for local files or StorageUri.Parse() for absolute URIs (e.g., "s3://bucket/key").

Explore the documentation for each specific connector to learn about its installation, configuration options, and usage examples.

Next Steps​