Live AI Dream Session, Vol. 3: Trustworthy AI in Practice with Sarath Kuravi and Dr. Shonraj Ballae • Nov 10 | 11:00 AM EST Reserve Your Seat Claim Free Seat

Reserve Your Seat

Data Lakehouse Architecture for AI: How to Build an AI-Native Lakehouse for the Enterprise

Enterprise data teams are being asked to deliver two things from one stack: trusted business intelligence for finance and operations, and reliable training data, features, and retrieval context for machine learning and generative AI. The warehouse-plus-lake split that most organizations inherited was never designed to carry both. In practice, that shows up as duplicated data across systems, governance scattered across tools, and compute spend climbing faster than the model teams it is supposed to serve.
An AI-native lakehouse is a data lakehouse architecture for AI that treats machine learning (ML), feature engineering, and large language model (LLM) retrieval as first-class workloads from day one, built on the same storage, catalog, and lineage that serve traditional analytics.
This guide walks through what makes an enterprise data lakehouse architecture AI-native, a reference layering model, the choice between Databricks and Snowflake for enterprise AI, how MLOps on a lakehouse works, and how to migrate without stalling existing reporting.

Key Takeaways

What Is an AI-Native Data Lakehouse?

An AI-native data lakehouse, sometimes called an AI-ready data lakehouse, is a lakehouse built so analytics, model training, feature serving, and LLM retrieval run on one governed copy of data. It stores open table formats such as Delta Lake or Apache Iceberg on object storage and adds vector search, a feature store, and unified catalog governance.

Data Lakehouse vs Data Warehouse: What Makes a Lakehouse AI-Native?

The data lakehouse vs data warehouse comparison usually gets reduced to a cost argument. The bigger difference matters more for AI. A warehouse is built for curated, structured data queried through SQL. A lake accepts everything, but without transactional guarantees or consistent schema, it rarely produces training sets a data scientist could trust twice in a row.
The lakehouse pattern consolidates both using an open table format, Delta Lake, Apache Iceberg, or Apache Hudi, on top of cheap object storage. That gives data teams ACID (atomicity, consistency, isolation, durability) transactions, schema evolution, and time travel against the same files that feed Python notebooks, SQL dashboards, and Spark jobs.
Calling it AI-native adds four things to that foundation:
The table below summarizes how the three architectures compare for AI workloads.
Capability Data warehouse Data lake AI-native lakehouse
Data types Structured All types All types
Transactions ACID None by default ACID through open table formats
ML and GenAI readiness Requires export Raw, often unreliable Native features, vectors, and models
Governance Strong, warehouse-only Fragmented One catalog for data and AI assets
Reproducibility Limited Weak Time travel on training data
Gartner analyst Prasad Pore reports 65% lakehouse adoption among Gartner’s client base, which positions the pattern as the default substrate for new AI workloads rather than a parallel experiment.[1]

Data Lakehouse Architecture for AI: A Six-Layer Reference Model

A workable reference design has six layers. Each has its own failure modes, so each needs its own ownership and controls.

1. Storage Layer: Open Table Formats on Object Storage

Object storage, Amazon S3, Azure Data Lake Storage, or Google Cloud Storage, holding Parquet files under an open table format. Delta Lake and Iceberg are the two formats most enterprise teams converge on. Both give ACID semantics; the choice usually comes down to the compute engine already in play.

2. Catalog and Governance Layer: One Control Plane for Data and AI

Unity Catalog (Databricks’ governance layer), Snowflake Horizon (Snowflake’s governance layer), or an open equivalent such as Apache Polaris (an open Iceberg catalog). This layer carries access control, lineage, and audit for every object the AI stack will touch. Data lakehouse governance decisions made here propagate to features, models, and prompts without re-implementation.

3. Transformation and Quality Layer: The Medallion Pattern

A medallion pattern of Bronze (raw ingest), Silver (cleaned and conformed), and Gold (business-ready) tables, built with Spark, dbt, or SQL. Data quality checks run as pipeline tests, not as a downstream dashboard.

4. Feature and Model Layer: Reproducible Training Data

Feature store, model registry, and experiment tracking. MLflow is the open reference. Features reference the Silver and Gold tables by version, so a training set from last quarter can be rebuilt exactly.

5. Serving and Retrieval Layer: RAG, Features, and Agents

Vector indexes for retrieval-augmented generation (RAG), online feature serving for low-latency inference, and model endpoints for both batch and real-time. For agentic AI workloads, this layer also exposes governed data to the Model Context Protocol or equivalent connectors.

6. Observability Layer: Drift, Health, and Cost

Pipeline health, data drift, model performance, and cost telemetry in one view. Without it, the lakehouse quietly degrades into another cost center.

How Do You Build a Data Lakehouse for AI Workloads?

The answer resolves at the catalog and feature layers. Those two carry the governance and reproducibility that separate production AI from disposable prototypes.

When Should You Choose a Big 4 Firm or Large Integrator for AI?

Databricks vs Snowflake for enterprise AI comes down to starting point: Databricks fits engineering- and ML-heavy teams building models and agents on open formats, while Snowflake fits SQL- and BI-heavy teams adding AI to governed analytics.
Databricks is a Spark-native environment with Delta Lake and Unity Catalog at its core. It appeals to organizations with heavy data engineering and ML workloads, deep Python usage, and a preference for notebook-driven development. MLflow, Model Serving, Vector Search, and Agent Bricks sit on the same catalog as the raw tables. For teams building retrieval pipelines, fine-tuned models, or multi-agent systems against proprietary data, this consolidation is the main argument.
Snowflake began as a SQL-first warehouse and extended into the lakehouse pattern through Iceberg table support, Snowpark for Python and ML, Cortex for LLM features, and Horizon for governance. It suits organizations with substantial SQL and BI investment that want to add AI workloads without rebuilding the analytics layer.
Factor Databricks Snowflake
Origin Spark-native lakehouse SQL-first warehouse
Default table format Delta Lake Native tables plus Iceberg
Governance Unity Catalog Horizon
ML and GenAI tooling MLflow, Model Serving, Vector Search, Agent Bricks Snowpark, Cortex
Best fit Heavy data engineering, Python, custom models, agents Heavy SQL and BI investment, AI added to analytics
Delta Lake deserves a note of its own for enterprise AI. As an open table format, Delta Lake runs under both Databricks and increasingly outside it, including through Iceberg interoperability. Time travel, schema enforcement, and change data feed are the properties that make reproducible training sets and auditable model lineage practical rather than aspirational.
The best data lakehouse architecture for AI, in practice, is the one aligned with the organization’s existing skills, cloud footprint, and governance maturity. The decision rarely turns on a feature matrix. It turns on how much of the current stack can carry forward, which is the first thing an Intuceo lakehouse readiness assessment maps.

How Does MLOps Work on a Data Lakehouse?

MLOps on a lakehouse keeps features, training sets, models, and serving endpoints under the same catalog as the source tables that produced them. That matters for three reasons.
First, reproducibility. Time travel on the underlying Delta or Iceberg tables lets a team retrain a model on the exact data version it was first trained on, which is often what regulators and internal audit actually ask for.
Second, lineage. When a model drifts, the engineering team can trace from the serving endpoint back through the feature, the Gold table, the Silver transformation, and the raw Bronze ingest, inside one lineage graph.
Third, governance. Databricks’ 2026 State of AI Agents report, based on telemetry from more than 20,000 organizations, found that companies using AI governance tools put 12 times more AI projects into production than those without.[2] On a lakehouse, that governance is applied once at the catalog layer and inherited by every downstream model, feature, and agent, the principle behind Intuceo’s DARWIN AI governance approach.
For generative AI specifically, the lakehouse also serves as the retrieval substrate for enterprise LLM infrastructure. Vector indexes sit alongside structured tables, so RAG pipelines and agents can combine semantic search with precise filters against the same governed data without copying it into a separate store.

How to Migrate From a Data Warehouse to a Lakehouse Without Breaking Reporting

A full lift-and-shift data warehouse to lakehouse migration rarely succeeds and rarely pays back inside a budget cycle. A workload-first migration does: move the jobs the warehouse handles poorly first, dual-run reporting during cutover, and retire warehouse workloads only once their replacements are proven.
Migrations that follow this pattern deliver AI workloads first, which is what justified the move in the first place, and let BI follow once the lakehouse has proven itself on harder problems.

How Intuceo Builds AI-Native Lakehouses for Regulated Industries

Intuceo designs and builds AI-native lakehouses on Databricks and Snowflake for regulated industries, including healthcare, life sciences, and the federal space, with Unity Catalog or Horizon governance applied from day one and MLflow-based model lineage tied to the underlying Delta or Iceberg tables.
Where Intuceo’s delivery differs from a generic data engineering engagement: a reference medallion design hardened across prior client engagements including Florida Blue, GuideWell Health, UF Health, Janssen Pharma, and Bausch & Lomb; accelerators such as Intuceo-Ax™ for explainable classification workloads and iPDLC™ for faster AI delivery; and a 20-year track record engineering compliant data environments for Fortune 1000 and federal clients.
Get a lakehouse readiness assessment of your current stack and a 90-day plan to serve your highest-value AI workload from a governed foundation.

Frequently Asked Questions

Compliance belongs in the catalog and storage layers, not in each downstream tool. Unity Catalog or Horizon enforces access control, row and column masking, audit logging, and lineage once, and every feature, model, and agent that reads governed tables inherits those controls. For regulated workloads such as HIPAA-compliant healthcare analytics, pair that with pipeline-level de-identification and tokenization of personally identifiable information (PII) and protected health information (PHI) so sensitive fields never reach training sets or vector stores unless an approved use case explicitly requires them. Audit evidence then flows from the catalog itself rather than from a parallel compliance report.
Both open table formats offer ACID guarantees, schema evolution, and time travel on object storage. Iceberg has broader cross-engine support across Snowflake, Trino, Spark, Flink, and DuckDB, which makes it the stronger choice when the compute strategy is multi-vendor or likely to shift. Delta Lake has deeper feature maturity on Databricks-centric stacks, including Change Data Feed, Deletion Vectors, and tighter integration with MLflow and Unity Catalog. Interoperability layers such as Delta UniForm now let teams write in one format and read through the other, which reduces the need to pick permanently on day one.
Lock-in exposure on a lakehouse is lower than on a warehouse because the data sits in open file and table formats on commodity object storage. Keep the storage layer and the chosen table format decoupled from the compute engine, treat the catalog as a governed abstraction rather than a vendor contract, and avoid encoding critical business logic in proprietary stored procedures or notebook-specific features. Portability becomes a design choice rather than a vendor feature. A lakehouse built this way can be read by multiple engines concurrently and migrated to a different compute vendor without moving the underlying data.
Agents need their own identities, scoped credentials, and audit trails distinct from the human users who deploy them. Grant least-privilege access through the catalog so an agent sees only the tables, columns, or rows its task actually requires, and route inference traffic through a gateway that enforces rate limits, prompt filtering, and output guardrails. For sensitive data, prefer a retrieval interface over direct table access: the agent requests information through a governed endpoint and never reads raw rows it has no business touching.
For most AI workloads, yes. A lakehouse stores structured and unstructured data together, adds ACID transactions through open table formats, and lets features, models, and vector indexes share one catalog with the source tables. A warehouse still suits curated SQL reporting, which is why many enterprises dual-run both during migration.
Six layers: object storage with an open table format, a catalog for governance and lineage, a transformation layer using the Bronze/Silver/Gold medallion pattern, a feature and model layer, a serving and retrieval layer for RAG and agents, and an observability layer for drift, pipeline health, and cost.

MLOps in Healthcare and Pharma: How Teams Govern Models in Production

A model that classifies adverse event reports or stratifies patient risk becomes a regulated computerized system the day it goes live, with one complication that conventional validation never had to absorb: its performance depends on data it has not seen yet. The records it produces still carry the same integrity, traceability, and privacy obligations as a batch record or a claims file. MLOps in healthcare and pharma is the discipline that keeps those two facts aligned, so that every model in production can be reproduced, explained, and defended at inspection.
Machine learning operations (MLOps) already gives teams the raw mechanics: pipelines, experiment tracking, model registries, and monitoring. This guide covers what healthcare and pharma teams add on top of those mechanics, stage by stage, building on our broader guide to MLOps for regulated industries.
What is MLOps in healthcare and pharma?
MLOps in healthcare and pharma is the practice of running machine learning pipelines so every production model can be reproduced, explained, and defended at inspection. It adds regulated controls to standard MLOps: validated releases, Part 11 audit trails, HIPAA safeguards, and change control that is approved before retraining begins.

Key Takeaways

Which Regulations Apply to MLOps in Healthcare and Pharma?

No single regulation governs machine learning (ML) in healthcare and life sciences. Teams assemble their obligations from records rules, privacy law, the good practice regulations collectively known as GxP (manufacturing, clinical, laboratory, and pharmacovigilance), and artificial intelligence (AI) guidance that is still maturing. The documents below carry the most weight for models in production.
Source Status What it asks of the pipeline
21 CFR Part 11 (Code of Federal Regulations) Binding regulation Validated systems, secure time-stamped audit trails for record creation, modification, and deletion, and change control over systems documentation[1]
Health Insurance Portability and Accountability Act (HIPAA) Security Rule, 45 CFR 164.312 Binding regulation Access control, integrity safeguards, and audit controls that record and examine activity in systems holding electronic protected health information (ePHI)[2]
U.S. Food and Drug Administration (FDA) guidance on AI for drug and biological decision-making, January 2025 Draft, nonbinding A risk-based credibility assessment for a defined context of use, with life cycle maintenance after deployment[3]
FDA and European Medicines Agency (EMA) Guiding Principles of Good AI Practice, January 2026 Joint principles Traceable data provenance in line with GxP, risk-based performance assessment, and scheduled monitoring for issues such as data drift[4]
FDA Predetermined Change Control Plan (PCCP) guidance for AI-enabled device software Final, nonbinding Pre-specified modifications, a protocol to develop, validate, and implement them, and an impact assessment[5]
European Union (EU) Good Manufacturing Practice (GMP) Annex 22, Artificial Intelligence Consultation draft Continuous oversight of AI systems, including change control, performance monitoring, and human review procedures[6]
The practical reading of the table is that Part 11 and HIPAA are enforceable today, while the AI-specific documents describe the evidence inspectors will increasingly expect to see. FDA AI compliance in 2026 therefore means satisfying binding records and privacy rules with evidence organized the way the AI guidance describes it.

How Does FDA’s Context of Use Shape AI Model Validation?

FDA’s draft guidance anchors validation to a defined question and a context of use (COU), then rates model risk on two factors. Model influence is the weight the model’s output carries in the decision relative to other evidence. Decision consequence is the significance of an adverse outcome if that decision turns out to be wrong.[3]
The guidance’s own example of high risk on both counts is a model used to decide which trial participants can skip inpatient monitoring after dosing. A model that ranks literature for a human reviewer would sit far lower, and its credibility plan can be lighter in proportion.
This framing answers how FDA expects AI models to be validated and monitored in clinical settings: through evidence scaled to risk, defined before deployment, and maintained for as long as the model is in use. Operationally, the COU statement, risk rating, and acceptance criteria belong in the model registry as metadata on every version. When an inspector asks why a model was validated the way it was, the answer then travels with the model rather than sitting in a document nobody can locate.

How to Govern MLOps in Healthcare and Pharma, Stage by Stage

Implementing MLOps in healthcare and pharma comes down to deciding what evidence each stage must produce, then making the pipeline produce it automatically.

How Is MLOps in Healthcare Different From Standard MLOps?

Standard MLOps and its regulated form share the same tooling. The difference is the evidence each step has to leave behind.
Area Standard MLOps MLOps in healthcare and pharma
Data Versioned datasets Immutable, hashed snapshots linked to each model version, with ePHI kept in access-controlled zones and de-identified before experimentation
Validation Metrics reviewed after training Acceptance criteria fixed before training, tested on held-out data, with the report assembled from pipeline metadata
Release Automated promotion to production An approval signature bound to a specific model version, with a Part 11 audit trail of who approved what and when
Monitoring Drift alerts Thresholds set by the COU risk rating; a breach opens a quality investigation with documented root cause
Retraining Scheduled or triggered automatically A retraining protocol or PCCP, approved in advance, covering triggers, data windows, acceptance criteria, and rollback
Accountability The ML team A named owner, risk tier, and review date in a model inventory, with human review of AI outputs that inform regulated decisions

Stage 1: Data Preparation and Provenance

The FDA and EMA principles ask that data provenance, processing steps, and analytical decisions be documented in a traceable, verifiable manner consistent with GxP.[4] In practice, every training set becomes an immutable, hashed snapshot that a model version points to, so retraining on identical data is always possible. A HIPAA-compliant ML pipeline adds two controls at this stage: ePHI stays inside access-controlled zones with logged activity, and data is de-identified before it reaches experimentation environments that have no need for identifiers.

Stage 2: Training and Validation

GxP MLOps for pharma uses the same automation as conventional MLOps, operated under a risk-based quality system so the automation itself generates the evidence. Each training run captures the code commit, container image, parameters, and dataset version without manual entry. Performance is tested against acceptance criteria fixed before training, on held-out data the team did not tune against, and the validation report is assembled from pipeline metadata instead of being written after the fact.

Stage 3: Release and Approval Under Part 11

Promotion from staging to production is where 21 CFR Part 11 AI validation concentrates, starting with validation of the systems involved (§11.10(a)). The promotion is an electronic record: it needs an approval signature bound to a specific model version (§11.70), and an audit trail showing who approved what and when (§11.10(e)). Part 11 also requires change control over systems documentation (§11.10(k)), which in an MLOps setting extends to pipeline definitions and infrastructure code.[1]

Stage 4: Production Monitoring and Drift

Monitoring covers three signals: shifts in input distributions, shifts in output distributions, and realized performance once ground truth arrives, such as adjudicated adverse event cases or confirmed readmissions. In healthcare MLOps, alert thresholds should follow the COU risk rating. A breach opens an investigation under the quality system with documented root cause, which keeps automatic retraining from silently masking a problem. Prediction logs are records in their own right and need retention periods aligned with the underlying regulation.

Stage 5: Change Control and Retraining

Every retrained model is a new version, and the pharma AI model lifecycle depends on deciding in advance how much revalidation each type of change requires. For AI-enabled medical devices, a PCCP lets manufacturers describe planned modifications, the methodology to develop, validate, and implement them, and an assessment of their impact, so pre-authorized changes do not each need a new marketing submission.[5] The draft EU Annex 22 points in the same direction for manufacturing, pairing change control with ongoing performance monitoring.[6] The retraining protocol, covering triggers, data windows, acceptance criteria, and rollback, should be approved before the first deployment rather than drafted during the first incident.

Evidence Each Stage Must Produce

The table below summarizes what the pipeline should generate at each stage, and which requirement drives it.
Stage Evidence the pipeline must produce Main driver
Data preparation Hashed, immutable training data snapshots tied to each model version, logged ePHI access, and de-identified data for experimentation FDA and EMA principles; HIPAA Security Rule
Training and validation Code commit, container image, parameters, and dataset version for every run, plus results against pre-set acceptance criteria GxP risk-based quality system
Release and approval An approval signature bound to the model version, an audit trail of approvals, and change control over pipeline and infrastructure code 21 CFR Part 11
Production monitoring Input, output, and performance monitoring records, investigation records for threshold breaches, and retained prediction logs FDA draft AI guidance; FDA and EMA principles
Change control An approved retraining protocol or PCCP, and revalidation evidence for every new model version FDA PCCP guidance; draft EU Annex 22
Not sure your retraining protocol would hold up at inspection? Book a governance gap review.

Why Is Human Oversight Required for AI in GxP Settings?

In an April 2026 warning letter, FDA stated that when a manufacturer uses AI to help with current good manufacturing practice (CGMP) activities, any output or recommendation from an AI agent must be reviewed and cleared by an authorized human representative of the firm’s quality unit.[7] That case involved AI-drafted specifications and procedures rather than a deployed predictive model, yet the principle carries over directly: accountability for an AI-assisted decision stays with named people.
The FDA and EMA principles reinforce this by asking that performance assessment cover the complete system, including human and AI interaction.[4] For MLOps teams, that translates into review queues for low-confidence outputs, explanations surfaced at the point of review, overrides captured in the audit trail, and override rates tracked as a monitoring signal in their own right. Sound model governance for healthcare AI wraps these controls in a model inventory that records an accountable owner, a risk tier, and a periodic review date for every model in service.

How Intuceo supports governed MLOps

Intuceo is a PhD-led AI, ML, and data analytics services firm whose life sciences work includes Janssen, Ferring Pharmaceuticals, and Bausch & Lomb. Its MLOps engagements build compliance into the pipeline from the first commit, so audit logs, lineage, and validation evidence accumulate through normal operation. Deployed models carry drift monitoring, controlled retraining loops, and interpretability views designed for the reviewers who sign off on outputs.
Explainability, bias checks, and model documentation are reviewed by Intuceo’s PhD-led Board of Science & Innovation, and the iPDLC™ delivery framework places PhD-led quality gates at every transition from development to production. Teams work across MLflow, Kubeflow, and Amazon SageMaker, so governance fits the stack a client already runs instead of forcing a migration.
Running models in production ahead of an inspection or a submission? Get a governance gap review that maps each model to its context of use, audit trail coverage, and change control readiness.

Frequently Asked Questions

Pharma companies govern ML models under Part 11 by first scoping which ML artifacts count as regulated records. FDA’s Part 11 scope and application guidance ties the rule to records that other regulations, known as predicate rules, already require. It also describes enforcement discretion for certain Part 11 requirements, while predicate rule obligations stay fully in force.[8] Pharma teams therefore classify ML artifacts individually. Model outputs that feed a GxP decision, validation reports, release approvals, and training data cited in a submission are typically treated as regulated records. Exploratory notebooks stay outside scope until their results become the basis for a regulated decision.
It looks like a validated computerized system whose validation never fully closes. The International Society for Pharmaceutical Engineering (ISPE) Good Automated Manufacturing Practice (GAMP®) Guide: Artificial Intelligence was published in July 2025 for use alongside GAMP 5 (Second Edition). It covers the AI-enabled system life cycle, the responsibilities of internal and external suppliers, and ongoing monitoring and maintenance.[9] Two practical consequences follow. Cloud ML services used in GxP work need supplier assessment and infrastructure qualification. GxP and non-GxP workspaces should also be kept separate, so exploratory work cannot reach production without passing through change control.
Healthcare organizations keep ML pipelines HIPAA-compliant by de-identifying data, scrubbing identifiers from ML outputs, and holding business associate agreements (BAAs) with vendors. Most start by deciding which de-identification route applies. HIPAA recognizes two: removing a specified list of identifiers, or a formal determination by a qualified expert applying statistical and scientific methods.[10]
Beyond that, the most common leak path in healthcare MLOps is incidental. Identifiers get written into experiment logs, prediction logs, feature store snapshots, or error traces. Most of the gaps teams find in audits close with three steps: scrubbing those outputs, keeping features to the minimum the model needs, and holding BAAs with any vendor that touches ePHI.
A typical pharma MLOps stack has five layers:
Whatever tools you choose, the configured system, including its procedures, access controls, and retention settings, must be validated. That is why tool selection should follow the governance design rather than come before it.
A PCCP lets makers of AI-enabled medical devices pre-specify planned model modifications, the protocol to develop, validate, and implement them, and an assessment of their impact. Once FDA authorizes the plan, changes made within it do not each need a new marketing submission.[5] For MLOps teams, the PCCP becomes the approved rulebook for retraining.