Skip to content

Multi-cloud Setup

Senren supports deploying infrastructure across AWS and GCP from a single control plane.

Supported Cloud Providers

AWS

  • All regions where you have Kubernetes clusters
  • Example identifiers: aws:us-east-1, aws:eu-west-1, aws:ap-southeast-1

GCP

  • All regions where you have Kubernetes clusters
  • Example identifiers: gcp:us-central1, gcp:europe-west1, gcp:asia-east1

Coming Soon

  • Azure: azure:eastus, azure:westeurope, etc.
  • On-premise: Custom identifiers for self-hosted Kubernetes

Architecture

Each cloud provider + region combination gets a regional plane deployment:

┌──────────────────┐
│ Control Plane    │ (Single, cloud-agnostic)
└────────┬─────────┘
         │ Kafka
    ┌────┴─────┬──────────┬────────────┐
    ▼          ▼          ▼            ▼
┌─────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐
│AWS      │ │AWS      │ │GCP       │ │GCP       │
│us-east-1│ │eu-west-1│ │us-cent-1 │ │eu-west-1 │
│         │ │         │ │          │ │          │
│Regional │ │Regional │ │Regional  │ │Regional  │
│Plane    │ │Plane    │ │Plane     │ │Plane     │
└─────────┘ └─────────┘ └──────────┘ └──────────┘

Regional Plane Setup

Each regional plane deployment needs:

  1. Kubernetes cluster in that cloud/region
  2. Kafka connectivity to the control plane Kafka
  3. Regional plane deployment configured for that region

Example: AWS us-east-1

apiVersion: apps/v1
kind: Deployment
metadata:
  name: regional-plane
  namespace: senren-system
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: regional-plane
        image: senren-api:latest
        env:
        - name: MODE
          value: "regional"
        - name: REGION_ID
          value: "aws:us-east-1"
        - name: KAFKA_BOOTSTRAP_SERVERS
          value: "kafka.control-plane.svc:9092"
        - name: KAFKA_TOPIC_PREFIX
          value: "prod"

Example: GCP us-central1

apiVersion: apps/v1
kind: Deployment
metadata:
  name: regional-plane
  namespace: senren-system
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: regional-plane
        image: senren-api:latest
        env:
        - name: MODE
          value: "regional"
        - name: REGION_ID
          value: "gcp:us-central1"
        - name: KAFKA_BOOTSTRAP_SERVERS
          value: "kafka.control-plane.svc:9092"  # Via VPN/VPC peering
        - name: KAFKA_TOPIC_PREFIX
          value: "prod"

Multi-cloud Kafka Connectivity

Regional planes need to connect to the control plane's Kafka cluster. Options:

Option 1: Public Kafka Endpoint

Simplest but requires proper authentication:

- name: KAFKA_BOOTSTRAP_SERVERS
  value: "kafka.example.com:9092"
- name: KAFKA_SECURITY_PROTOCOL
  value: "SASL_SSL"
- name: KAFKA_SASL_MECHANISM
  value: "PLAIN"

Option 2: VPN/VPC Peering

More secure, requires networking setup:

  • AWS VPC ↔ GCP VPC peering
  • Site-to-site VPN
  • Use private Kafka endpoints

Option 3: Kafka on Managed Service

Use a multi-cloud Kafka provider:

  • Confluent Cloud (AWS, GCP, Azure)
  • AWS MSK with multi-region replication
  • Aiven for Apache Kafka

Multi-cluster Setup

Multiple Kubernetes clusters in the same region can share infrastructure or be isolated.

Example: Production + Shadow Traffic

from senren import Database

# Production cluster
prod_db = Database(
    name="user-features",
    type="redis-cluster",
    memory_mb=8192,
    regions=["aws:us-east-1:prod"],  # :prod suffix for cluster ID
)

# Shadow traffic cluster (same region, different cluster)
shadow_db = Database(
    name="user-features",
    type="redis-cluster",
    memory_mb=8192,
    regions=["aws:us-east-1:shadow"],
)

Each cluster gets its own regional plane with a unique REGION_ID.

Example: Multi-cloud Deployment

Deploy feature store across AWS and GCP:

from senren import SenrenClient, Database

client = SenrenClient(endpoint="control.senren.dev")

client.apply_state(
    databases=[
        Database(
            name="global-features",
            type="redis-cluster",
            memory_mb=16384,
            regions=[
                # AWS regions
                "aws:us-east-1",
                "aws:eu-west-1",
                "aws:ap-southeast-1",

                # GCP regions
                "gcp:us-central1",
                "gcp:europe-west1",
                "gcp:asia-east1",
            ],
        )
    ]
)

Senren will provision identical Redis clusters in all 6 regions automatically.

Data Consistency

Important: Regional deployments are independent. Senren does not replicate data between regions.

If you need cross-region data consistency, you must: - Use application-level replication - Use CDC (Change Data Capture) tools - Use databases with built-in replication (future roadmap)

Network Latency Considerations

Control plane → Regional plane: - Uses Kafka (async messaging) - Latency: typically < 100ms - Not latency-sensitive

Application → Regional database: - Direct connection to regional infrastructure - Latency: depends on proximity - Deploy regional planes close to users for best performance

Cost Optimization

Run control plane in cheapest region: - Control plane is cloud-agnostic - Can run in AWS us-east-1 (typically cheapest)

Regional planes are lightweight: - Single replica per region - Low CPU/memory usage - Use small instance types

Infrastructure costs dominate: - Actual databases (Redis, etc.) are the main cost - Regional plane overhead is minimal

Next Steps