Sign In
Chapters · #34

Then vs Now — From OpenTelemetry, ClickHouse, and Redpanda to FORJD

Reading Progress70%

Chapter 34: Then vs Now — From OpenTelemetry, ClickHouse, and Redpanda to FORJD

This chapter is the explicit Then vs Now record for DEML's data plane. Earlier chapters describe the steady-state architecture; this chapter documents what we initially operated, why that topology failed our tenancy and security bar, and what we run instead. Operators and agents must not reintroduce the retired stack under a new name.

Why a first data plane existed

DEML began as a full-stack learning and observability product. Identity, billing, Angular dashboards, and telemetry processing lived in one repository. To keep Django off the hot path, the first architecture borrowed a familiar industry pattern:

  1. Emit — services and the browser produced OpenTelemetry (OTLP) spans and metrics.
  2. Route — an OpenTelemetry Collector batched and exported high-volume telemetry.
  3. Buffer — a Kafka-compatible broker (Redpanda) absorbed bursts between producers and consumers.
  4. Store / analyze — a columnar OLAP store (ClickHouse) powered dashboards, CES-style rollups, and long-window analytics.
  5. Project locally — DEML workers (and later a Railway-hosted Rust data plane) materialized product read models beside Django.

That stack was coherent for a single-owner SaaS that treated telemetry as operator observability. It was the wrong spine for a multi-tenant product that must forward ciphertext-only events, bind every call to a tenant, and keep the learning/control plane free of stream-broker operations.

Then: what we initially did

Concern Initial DEML-owned plane
Signal shape OTLP spans/metrics; often plaintext or lightly redacted
Ingest edge OTel Collector (infrastructure/otel-collector, OTLP :4318)
Buffering Redpanda (or equivalent Kafka-protocol bus) on Railway/Fly
Analytics warehouse ClickHouse (columnar OLAP) queried for CES and dashboards
Durable processing DEML/Railway workers, local outbox patterns, dual ownership
Product reads Mix of Django models + local projections + CH queries
Auth to data plane Shared service credentials, cluster DSNs, collector env vars
Failure mode Partial empties, dual writes, "is CH lag or is the app down?"

Concrete friction showed up in operations and security review:

  • Credential and surface sprawl. Collector hosts, ClickHouse users/passwords, Redpanda ACLs, and worker DSNs multiplied. A partner integration could not be expressed as one tenant-bound token.
  • Wrong trust boundary. Firebase end-user identity and OTLP exporter credentials are not substitutes for sealed envelopes. Labeling plaintext or OTLP product telemetry as E2EE was dishonest.
  • Split brain. Django, ClickHouse, and Redpanda each held partial truth. Replay meant replaying a broker topic and reconciling a columnar table and healing local projections.
  • Control-plane weight. Running OLAP + broker + collector beside identity and Stripe billing forced DEML into stream-platform ops it did not want to own.
  • Symmetry failure. Not every account could run the same pipeline without provisioning CH/Redpanda capacity per tenant—antithetical to symmetrical multi-account control.

Appendix C still documents an alternate Cloud Run topology that once assumed an OTel Collector writing into an analytics warehouse. That section is historical residency math—not the product path. Appendix T records the retirement of the Railway Rust data-plane roles that briefly replaced parts of this stack before FORJD became exclusive.

Now: what we do

Concern Steady-state FORJD plane
Signal shape Client-sealed AES-256-GCM envelopes (key_id + ciphertext + routing metadata)
Ingest edge Angular/API → Django BFF (Firebase or deml_… key) → FORJD /api/v1/ingest
Buffering FORJD Postgres outbox + Dragonfly streams (inside FORJD)
Analytics / CES FORJD durable stream_results + analytics APIs via Django adapters
Durable processing FORJD workflows, engine roles, Prefect soft-fail where configured
Product reads Stable DEML paths; Django adapts only allowlisted FORJD routes
Auth to data plane Tenant-bound opaque fjsvc_ service token; secret ref in DEML Postgres
Failure mode Typed 503 / forjd_degraded (or explicit degraded: true fallback envelopes)

DEML owns identity, profiles, roles, Stripe billing, consent, learning content, API credentials, and the Angular surface. FORJD owns sealed intake, streaming, projections, analytics, ML, SIEM/SOAR, reports, replay, DLQ, and tenant erase. Integration contract: docs/FORJD_INTEGRATION.md. Ownership appendix: Appendix T. Observability chapter (steady state): Chapter 8.

flowchart TB
  subgraph then [Then — retired]
    A1[Apps / browser] -->|OTLP| C1[OTel Collector]
    C1 --> R1[Redpanda]
    R1 --> CH[ClickHouse]
    R1 --> W1[DEML / Railway workers]
    W1 --> PG1[(Local projections)]
    CH --> UI1[Dashboards]
    PG1 --> UI1
  end

  subgraph now [Now — steady state]
    A2[Angular / integrators] -->|Firebase or deml_ key + sealed envelope| DJ[Django BFF]
    DJ -->|fjsvc_| FJ[FORJD API + engine]
    FJ --> DF[(Dragonfly)]
    FJ --> SB[(Supabase Postgres)]
    FJ --> PR[Projections analytics ML SIEM]
    A2 -->|Product reads| DJ
    DJ -.->|BFF adapters| PR
  end

Benefits of the cutover

  1. One trust model. Partners and DEML itself authenticate to FORJD with a tenant-bound fjsvc_ token. End-user Firebase JWTs terminate at Django and are never forwarded.
  2. Ciphertext-first product lane. Sealed envelopes carry routing metadata only; plaintext lesson content, PII, and scores stay out of ingest metadata. E2EE is a real cryptographic boundary, not a marketing label on OTLP.
  3. Lean control plane. DEML no longer runs ClickHouse nodes, Redpanda clusters, or OTel Collectors for product telemetry. Fly Django + managed Postgres + Vercel stay focused on identity and BFF adaptation.
  4. Durable recovery. Replay and DLQ are FORJD contracts with idempotent projection keys—not ad-hoc "replay the topic into CH and hope the dashboard catches up."
  5. Fail-closed honesty. Steady FORJD_READ_MODE=forjd returns typed degradation; empty success is reserved for explicit cutover modes and is marked degraded: true so Angular never confuses an outage with an empty tenant.
  6. Symmetrical tenancy. Every mapped account uses the same adapter paths; capacity and RLS live in FORJD, not in per-tenant CH databases provisioned from DEML.
  7. Clear erase saga. Account deletion calls FORJD POST /api/v1/tenants/{id}/erase before local teardown—impossible to express cleanly when telemetry lived in three stores.
  8. Optional add-ons stay optional. FORJD's lean-core / fat-extension model (FORJD_ADDONS) keeps scanners and intel behind config gates without dragging pandas/OLAP into the sealed path.

What must not return

  • Do not add OpenTelemetry Collector, ClickHouse, Redpanda, Kafka, or a DEML-local projection worker "for convenience."
  • Do not query FORJD Postgres or Dragonfly from Django.
  • Do not forward Firebase tokens or deml_… keys to FORJD.
  • Do not treat missing FORJD SIEM/analytics routes as empty 200 success lists without an explicit degraded marker.
  • Do not document OTLP → columnar warehouse as the product telemetry path; optional operator tracing is a separate, non-E2EE concern.

Operator checklist after reading this chapter

  1. Confirm production flags: FORJD_WRITE_MODE=forjd, FORJD_READ_MODE=forjd (or cutover phase 2).
  2. Confirm every account that needs data-plane access has map_forjd_tenant + secret ref (never a plaintext token in git).
  3. Confirm Railway/ClickHouse/Redpanda/OTel collector services for product telemetry are absent from live topology (infrastructure/railway/README.md).
  4. Confirm dashboards and CES read through Django adapters to FORJD—not direct CH DSNs.
  5. On FORJD outage: restore FORJD; do not stand up a temporary Redpanda/ClickHouse sidecar.

The transition is complete when DEML's repository contains adapters and policy—not brokers and warehouses. Everything else in this book assumes Now.