Layer 1
Ingestion & connectivity
Two front doors: connectors, which reach into the customer's information system, and the
file drop. Connectors run inside a VPC and egress through a NAT gateway — the only
persistent network resource in the architecture. Credentials never pass through
application code: a dedicated function reads them from Secrets Manager.
Whatever breaks the inbound contract goes to quarantine with its cause and is resubmitted
once fixed. The silver layer normalizes and types what was accepted.
Layer 2
Storage
Aurora Serverless v2 (PostgreSQL 16) holds the model and the data, accessed through the
RDS Data API over HTTPS — which keeps application functions out of a VPC.
S3 holds datasets and exports, Secrets Manager the connection credentials, KMS the
encryption keys at rest.
Why one schema for every tenant rather than one per customer: the second is easier to
explain and far more expensive to run. Drift is inevitable, and every migration multiplies
by the customer count.
Layer 3
Processing & engines
128 service modules, packaged as python3.12 Lambda functions in zip format.
This is where the engines that make the product live: deterministic and probabilistic
matching, reversible survivorship, the quality rule engine, the profiler, the lineage
engine and drift detection.
Long-running work goes through an asynchronous queue and a dedicated worker: the HTTP call
returns immediately with a job identifier.
Layer 4 · AI
AI & assistance
An assistant built on Amazon Bedrock (Claude Haiku 4.5) that does three
things: it explains what is on screen, it helps decide by
ranking what to fix next, and it automates the repetitive part of
stewardship — synthesising quality rules from real profiles, inferring semantic types and
PII, and proposing mass corrections.
Retrieval is real: catalogue assets are embedded with Titan Text Embeddings v2, so
“find me something like this table” is a vector search, not a keyword match. A document
knowledge base is the next step and is marked planned on the diagram.
The discipline is in the contract between tool and model. A tool returns
a measure, never a sentence, so the assistant cites a figure instead of paraphrasing one.
And it proposes in a closed vocabulary, carrying the identifiers it saw — so a person
confirms exactly that list, not a vague “yes”.
Layer 5
API & identity
API Gateway HTTP v2, behind a dedicated domain and an ACM certificate, delegates to a JWT
authorizer that validates the token signature against the Cognito public keys. Identity
comes from Cognito (Hosted UI, OAuth 2.0 Authorization Code + PKCE).
The tenant_id and roles are read from the token claims, never from a request
parameter. That is what makes multi-tenant isolation impossible to bypass from the client.
Layer 6
Consumption & distribution
The React console is served as static files from a private S3 bucket, exposed only through
CloudFront via an Origin Access Control, with an SPA fallback for client-side routes. Other
consumers go through the Pull API, webhooks or the data product marketplace.
Masking, filtering and message contracts apply on the way out: the policy follows the
requester, not the dataset.
Cross-cutting
Security, governance, responsible AI, operations
Four concerns that cannot live inside one layer: encryption and isolation apply
everywhere; policies and contracts are evaluated on ingestion as well as on output; every
AI system is registered with its risk tier, its model card and its incidents, as the EU AI
Act requires; and observability and FinOps measure the whole.
Underneath
Secure delivery — DevSecOps
Every layer above is built and shipped through one pipeline. Today it gates on a pinned
static analyser for Python, a strict TypeScript type-check, deployment guards that fail the
build, and a Terraform plan read with -detailed-exitcode so “code waiting to be
applied” is reported rather than confused with a failure.
Security by design is the next stage, and it is not in place yet. The
tools that get it there — static analysis, dependency and IaC scanning, secret detection,
runtime protection, an SBOM, threat modelling — are named in the delivery section below,
drawn dashed on the diagram and marked planned. A security posture you have not built
yet is a plan, not a fact.
