AI tightly integrated with a data platform. Combines a lakehouse architecture (ADLS Gen2 + Databricks + Delta Lake) with Azure AI Search and Azure OpenAI. Data flows through medallion zones (raw → curated → serving) with AI at the serving layer.
The Data-Centric Lakehouse AI pattern is built on the medallion architecture — data flows through Raw, Curated, and Serving zones. Databricks handles ETL and feature engineering, three separate ADLS Gen2 accounts maintain zone isolation, and AI Search + OpenAI serve the AI layer.
This is the ideal pattern for organizations with large-scale data platforms that want to layer AI on top of existing data assets.
| GPT-4o | 40K TPM |
| text-embedding-3-large | 120K TPM |
| SKU | Standard |
| Replicas | 2 (HA) |
| Partitions | 1 |
| Semantic | Standard |
| Tier | Premium |
| VNet | Injected |
Bronze layer — ingestion & landing
{base}{env}raw
Silver/Gold — transformed & clean
{base}{env}curated
AI-ready — embeddings & vectors
{base}{env}serving
graph TB
subgraph LHVNet["Lakehouse + AI VNet 10.40.0.0/16"]
subgraph DBX["Databricks VNet-Injected"]
DBPUB["Public Subnet /22"]
DBPRV["Private Subnet /22"]
end
subgraph DataZones["Medallion Data Zones"]
RAW["ADLS Gen2 — Raw<br/>landing / raw / archive"]
CUR["ADLS Gen2 — Curated<br/>silver / gold / delta"]
SRV["ADLS Gen2 — Serving<br/>embeddings / vectors"]
end
subgraph AILayer["AI Layer"]
AIS["Azure AI Search<br/>Standard, 2 replicas"]
AOAI["Azure OpenAI<br/>GPT-4o 40K TPM"]
end
subgraph App["Application"]
APPSVC["App Service<br/>RAG Application<br/>VNet-integrated"]
end
subgraph Support["Support Services"]
KV["Key Vault"]
MON["Log Analytics + App Insights"]
end
end
DBX -->|"ETL / Feature Eng"| RAW
RAW -->|"Transform"| CUR
CUR -->|"Vectorize"| SRV
SRV --> AIS
SRV --> AOAI
APPSVC --> AIS
APPSVC --> AOAI
AIS --> MON
AOAI --> MON
DBX --> MON
AOAI --> KV
| # | Service | Resource Name | SKU / Tier | Purpose | Monthly Cost |
|---|---|---|---|---|---|
| 1 | Azure Databricks | {base}-{env}-dbw | Premium, VNet-injected | Spark ETL, feature engineering, Delta Lake | ~$2,800 |
| 2 | Azure OpenAI | {base}-{env}-openai | S0 | GPT-4o (40K TPM) + embeddings (120K TPM) | ~$2,750 |
| 3 | Azure AI Search | {base}-{env}-search | Standard, 2 replicas | Vector + semantic search (HA) | $490.56 |
| 4 | ADLS Gen2 (Raw) | {base}{env}raw | Standard LRS, HNS | Landing, raw data, archive | ~$80 |
| 5 | ADLS Gen2 (Curated) | {base}{env}curated | Standard LRS, HNS | Silver, gold, delta tables | ~$60 |
| 6 | ADLS Gen2 (Serving) | {base}{env}serving | Standard LRS, HNS | Embeddings, vector indexes, model artifacts | ~$40 |
| 7 | App Service (opt.) | {base}-{env}-app | P1v3, Linux | RAG application compute | ~$140 |
| 8 | Key Vault | {base}-{env}-kv | Standard | Connection strings, API keys | ~$5 |
| 9 | Log Analytics | {base}-{env}-law | PerGB2018, 90-day | Telemetry and audit | ~$35 |
| 10 | Application Insights | {base}-{env}-appinsights | Workspace-based | APM for App Service + AI | Incl. |
| 11 | VNet | {base}-{env}-vnet | /16 with 6 subnets | Network backbone | Free |
| 12 | Private DNS Zones | 5 zones | Global | Private name resolution | ~$2.50 |
| 13 | Private Endpoints | 7 endpoints | — | OpenAI, Search, Raw-Blob, Raw-DFS, Curated-DFS, Serving-Blob, KV | ~$51 |
| Estimated Total (Moderate Production) | ~$8,350/mo | ||||
AI Search with 2 replicas provides HA for production search queries. Databricks cost based on Premium DBU consumption.
VNet-injected Databricks for Spark-based ETL, feature engineering, and Delta Lake operations. Processes data through medallion zones. Public and private subnets each /22.
GPT-4o at 40K TPM for inference and text-embedding-3-large at 120K TPM for vectorization. Higher quota than other patterns to handle data-intensive embedding generation.
Production-ready search with 2 replicas for high availability. Standard tier with semantic ranking. Indexes vectors from the Serving zone for RAG queries.
Three separate storage accounts for medallion architecture: Raw (landing/raw/archive), Curated (silver/gold/delta), Serving (embeddings/vectors/models). All HNS-enabled.
Optional RAG application compute with VNet integration. Serves as the frontend application that queries AI Search and Azure OpenAI from data in the Serving zone.
Private Endpoints for all services: OpenAI, AI Search, Raw-Blob, Raw-DFS, Curated-DFS, Serving-Blob, Key Vault. Five Private DNS Zones for resolution.
Layer AI on top of existing data lake. Use medallion architecture to feed clean, curated data into AI Search and OpenAI.
Process streaming data through Databricks, generate embeddings in real-time, serve via AI Search for instant RAG queries.
Ingest millions of documents through medallion zones, extract insights via OpenAI, and index for semantic search.
Combine traditional ML predictions from Databricks with generative AI explanations from OpenAI for complete analytics.
| Constraint | Mitigation |
|---|---|
| Three ADLS accounts increase management overhead | Automate with Bicep; use consistent naming conventions |
| Databricks Premium is expensive | Use job clusters with auto-terminate; spot instances for ETL |
| Data movement between zones adds latency | Use Databricks Delta Live Tables for streaming ETL |
| AI Search with 2 replicas doubles search cost | Use 1 replica for dev/test, 2 for production only |
| Complex VNet with 6 subnets | Careful IP planning; document subnet allocation |
Generate a complete deployment spec sheet with GitHub Actions workflow, Bicep file structure, and prerequisite checklist.
Generate Deployment Spec Sheet