Fluvio - Generate

Generate

Generate a Connector

CDK generate helps developers build a sample Connector project by answering a few simple questions.

Use cdk generate to create a new connector project:

$ cdk generate
🤷   Project Name: my-connector
🔧   Destination: ~/my-connector ...
🔧   project-name: my-connector ...
🔧   Generating template ...
✔ 🤷   Will your Connector be public? · false
🤷   Please set a group name: acme
✔ 🤷   Which type of Connector would you like [source/sink]? · source
Ignoring: /var/folders/r8/4x6_d2rn283946frzd1gc1pr0000gn/T/.tmptToFV3/cargo-generate.toml
[1/6]   Done: Cargo.toml             
[2/6]   Done: Connector.toml
[3/6]   Done: sample-config.yaml
[4/6]   Done: src/config.rs
[5/6]   Done: src/main.rs
[6/6]   Done: src
🔧   Moving generated files into: `~/my-connector`...
💡   Initializing a fresh Git repository
✨   Done! New project created ~/my-connector

The generator created Rust project ready to compile:

$ tree 
.
├── Cargo.toml
├── Connector.toml
├── README.md
├── sample-config.yaml
└── src
    ├── config.rs
    └── main.rs

This a simple connector with the code in src/main.rs:

mod config;
use config::CustomConfig;

use fluvio::{RecordKey, TopicProducer};
use fluvio_connector_common::{
    connector,
    Result
};

#[connector(source)]
async fn start(config: CustomConfig, producer: TopicProducer) -> Result<()> {
    println!("Starting my-connector source connector with {config:?}");
    for i in 1..1000 {
        let value = format!("Hello, Fluvio - {i}");
        producer.send(RecordKey::NULL, value).await?;
        producer.flush().await?;
        std::thread::sleep(std::time::Duration::from_millis(1000));
    }
    Ok(())
}

Connectors may also have can have configuration parameters as defined in src/config.rs:

use fluvio_connector_common::connector;

#[connector(config)]
#[derive(Debug)]
pub(crate) struct CustomConfig {
    #[allow(dead_code)]
    foo: String,
}

The Connector.toml file contains the definition of the Connector parameters required to load the file in the Cluster and publish it to Connector Hub.

$ cat Connector.toml
[package]
name = "my-connector"
group = "acme"
version = "0.1.0"
apiVersion = "0.1.0"
fluvio = "0.21.3"
description = ""
license = "Apache-2.0"
visibility = "private"

[direction]
source = true

[deployment]
binary = "my-connector"

[custom.properties.foo]
title = "Foo"
description = "Foo"
type = "string"

Sections

package is used to build the connector FQDN acme/my-connector@0.1.0, and the description to publish to Hub. The group name is equivalent to the package owner in the Hub.
direction is used to declare the direction data flows through the connector, with respect to the Fluvio cluster. An inbound connector uses source = true, and and outbound connectdor uses sink = true
custom.properties.foo defines a user configuration key foo that can be used in the logic of the connector.

The project is ready to build and test. Checkout the next section for instructions.

Steps

Edit this page

Generate a Connector

Steps