Databricks for Data Pipelines, LLM Integration, ML, and Predictive Analytics

Capabilities, release states and regional availability are as of 21 September 2026, taken from Databricks and Microsoft documentation. INVISOM has no partnership, reseller agreement or other commercial relationship with Databricks or its competitors.

One governance layer, four workloads

Most platform comparisons treat Databricks as a list of products: an ingestion tool, a pipeline framework, a model registry, a serving layer, an agent builder, a Postgres database. Read the documentation closely and a different picture emerges. Almost every one of these products registers its outputs in Unity Catalog: pipeline tables, models, functions, agent endpoints, MLflow traces, even agents running outside Databricks, which can now be registered as Agent Services (Beta). Lineage, permissions and attribute-based access policies then apply across all of them.

That is the platform's genuine differentiator, and it is the right lens for an enterprise decision. Our Snowflake and Databricks comparison covered placement and lock-in at the platform level. This article goes inside Databricks, workload by workload, and argues three things:

1. For data pipelines and classic ML, the platform is mature, and its core pipeline framework is now open source in Apache Spark.
2. For LLM integration and agents, the capabilities are broad but newer, serverless-only, and billed on product lines that are easy to miss.
3. In the Gulf, the feature set actually available in-region is materially smaller than the one in product announcements, and AI features in UAE North process data outside the UAE by default.

Data pipelines: mature, and increasingly open

Databricks' pipeline stack is now branded Lakeflow and has three parts.

Lakeflow Connect provides managed connectors: change data capture from databases such as SQL Server, MySQL and PostgreSQL; SaaS sources such as Salesforce, Workday and Jira; and file sources such as SharePoint and Google Drive. Ingestion pipelines write to streaming tables and run on serverless compute by default. The documentation is candid that connectors are "in various release states" and that some sources or tables do not yet support incremental ingestion, so each connector needs checking individually.

Lakeflow Spark Declarative Pipelines is the framework previously sold as Delta Live Tables. You declare streaming tables and materialized views in SQL or Python; the engine works out dependencies, orchestration, checkpoints and retries, and applies data quality expectations. In June 2025 Databricks announced it would contribute the core of this framework to Apache Spark, and Spark 4.1.0 ships it as Spark Declarative Pipelines. That matters for portability: pipeline definitions written against the open API are no longer purely proprietary, although the Databricks version adds runtime optimizations and managed operations on top.

Lakeflow Jobs orchestrates the rest: notebooks, SQL, pipelines and other tasks on a schedule or trigger.

The documented limitations are worth reading before committing:

None of these is unusual for a declarative framework. They do mean that "we will move everything into declarative pipelines" is a design project, not a lift-and-shift.

Machine learning and predictive analytics: the most mature layer

For classic ML, Databricks' position rests on MLflow, which it created and which remains open source. MLflow 3.0 was released on 10 June 2025 and the 3.x line has continued since; PyPI lists 3.16.1 as current. MLflow 3 extended the project from experiment tracking and model registry into generative AI: tracing every step of an agent's execution, evaluating with LLM judges and code-based scorers, collecting expert feedback, and monitoring production traffic with the same scorers. On Databricks, models, traces and evaluation data are stored in Unity Catalog.

Predictive analytics now also reaches SQL users directly. ai_forecast produces time-series forecasts, ai_detect_anomalies flags anomalous values (Beta), and ai_predict_class and ai_predict_value (Beta) train a classification or regression model on rows with known outcomes and fill in the rest. Importantly for cost, Databricks states that these run entirely on the compute you submit them from, with no separate inference charge.

Model Serving deploys custom MLflow models, agents, Databricks-hosted foundation models and governed access to external models (for example OpenAI and Anthropic) behind REST endpoints on serverless compute. Its documented limits are reasonable for most enterprise use: 16 MB payloads (4 MB for agent endpoints), a model execution limit of 597 seconds, and default endpoint quotas that the account team can raise. Two operational details deserve more attention than they get:

• Model images are not patched in place. Databricks does not apply security patches to existing model images, to avoid destabilizing production. A new model version produces a patched image, so regular redeployment belongs in your security process.
• Foundation Model API inputs and outputs may be retained for up to 30 days, in the workspace region, for abuse detection, and partner model providers may apply their own safety retention. This needs to be reflected in data classification decisions before sensitive data is sent to hosted models.

LLM integration: broad, newer, and metered separately

Databricks offers three routes to put LLMs to work on enterprise data. Our guide to RAG, fine-tuning and custom models covers the choice between techniques; here the question is what each route costs and where it runs.

AI Functions call models from SQL, notebooks or pipelines. Task-specific functions cover document parsing (ai_parse_document), schema-based extraction (ai_extract), classification, summarization, masking, translation and, in Beta, transcription; ai_query is the general-purpose route to any supported model endpoint. They are the fastest way to turn unstructured documents into governed tables. They also have three characteristics that matter:

• They are not available on Classic SQL warehouses.
• Task-specific functions run inference on Databricks-managed serverless GPU infrastructure that is billed in addition to the query compute, under different billing product names depending on the function.
• Language coverage varies. ai_translate, in Public Preview, lists English, French, German, Hindi, Italian, Portuguese, Spanish and Thai as supported target languages and is described as tuned for English. Arabic is not on the list.

Agent Bricks provides managed agent builders. Knowledge Assistant creates a question-answering agent with citations over documents in Unity Catalog volumes, tables or AI Search indexes, and improves it from expert feedback. Supervisor Agent orchestrates Genie agents, dashboards, agent and model endpoints, Unity Catalog functions and MCP servers. Both require serverless compute, Model Serving and a serverless usage policy with a non-zero budget. For Arabic-language corpora, note that two of the three embedding models Knowledge Assistant accepts for an existing AI Search index are English variants.

Custom agents can be written with LangGraph, LangChain, the OpenAI SDK, LlamaIndex or other libraries, traced with MLflow, and deployed through Model Serving or Databricks Apps.

Lakebase: Postgres next to the lakehouse

Lakebase, Databricks' managed Postgres, became generally available on 3 February 2026. Its use cases are specific: serving lakehouse tables to applications at low latency by syncing Unity Catalog tables into Postgres, acting as an online feature store or agent state store, and writing Postgres changes back as Delta tables (Public Preview). It supports branching, scale to zero, read replicas and Postgres 16 to 18.

It is a reasonable choice for applications built close to the lakehouse. It is not yet a default replacement for an enterprise's operational database estate, and its regional coverage is narrower than the rest of the platform, as the Gulf section shows.

Cost governance: where the bill hides

Databricks pricing is consumption-based, in Databricks Units (DBUs) at different rates per product. The difficulty is less the rate than the number of meters an AI workload touches.

Three facts from the documentation shape any cost-governance design:

• Budgets alert; they mostly do not stop spend. Account budgets track monthly spend at list price and send email alerts at up to four thresholds. Only budgets scoped to Unity Gateway can block further usage, and those do not yet track provisioned throughput.
• Attribution depends on tags you set up front. Serverless usage policies (Public Preview) attach cost attribution tags to serverless notebooks, jobs, pipelines, serving endpoints and apps. Without them, serverless spend is hard to allocate to teams.
• Visibility lags. Billable usage records in system.billing.usage are typically available within 12 hours, so near-real-time alerts on a runaway batch inference job are not possible from billing data alone.

On classic compute there is also a second invoice from the cloud provider, as our platform comparison explains. For AI workloads specifically, the practical controls are: a serverless usage policy per team before any AI feature is enabled; Unity Gateway budgets with per-user limits for pay-per-token access; a pilot run of every batch AI Functions job on a sample, with cost read from system tables, before full-table runs; and a weekly report joining system.billing.usage to jobs, pipelines and warehouses.

The Gulf view: the platform you can buy in-region is smaller

Databricks runs in the Gulf on three regions: Azure UAE North (Dubai), Azure Qatar Central (Doha) and Google Cloud me-central2 (Dammam). Databricks on AWS lists no Middle East region, so an AWS-committed enterprise in the UAE cannot run Databricks on AWS in-country.

Within those regions, features differ sharply.

Three findings stand out from the regional availability tables updated in late September 2026:

• Qatar Central has no serverless compute and none of the AI features listed: no Lakeflow Connect managed connectors, no Model Serving, no AI Functions, no agents.
• Google Cloud Dammam has serverless compute and Lakeflow Connect, but no Model Serving, no AI Search, no agent features, and no AI Functions beyond `ai_forecast` and `ai_detect_anomalies`. For Saudi data that must stay in the Kingdom, Databricks today is a pipeline and analytics platform, not an LLM platform.
• UAE North has the broadest coverage, but most LLM features depend on cross-Geo processing. Foundation Model APIs, ai_query, ai_parse_document, ai_extract, Knowledge Assistant and Supervisor Agent are supported there only with cross-Geo processing enabled. Microsoft's documentation states that cross-Geo processing is enabled by default for workspaces in Geos outside the US and EU without a compliance security profile.

The third point is the one most likely to be missed. An organization that enabled a UAE North workspace and started calling AI Functions may already be processing customer content outside the UAE. For data subject to the UAE or Saudi personal data protection laws, or to national data classification policies covered in our NDMO implementation guide, the setting "Enforce data processing within workspace Geography for Designated Services" should be an explicit, documented decision, not a default.

Microsoft has said its Saudi Arabia East region will open in November 2026 and AWS has announced a Saudi region for December 2026. Neither appears in Databricks' region lists yet; plan a re-assessment once they do.

When alternatives fit better

Databricks is a strong default for organizations whose centre of gravity is data engineering, Spark-scale processing and ML on governed data. It is not the best answer everywhere.

The general rule, which is INVISOM analysis rather than vendor guidance: choose Databricks where one governance layer across pipelines, models and agents is worth more to you than best-of-breed components, and check that the specific features you need are available, in your region, at a cost you can see.

What to do next

1. Map each target workload to the specific Databricks features it needs, then check the release state (GA, Public Preview, Beta) of each one.
2. Check every feature against the regional availability table for your workspace region, including whether it needs cross-Geo processing.
3. Set the Geo processing control deliberately on every Gulf workspace, and record the decision in your data classification and privacy documentation.
4. Create serverless usage policies and budgets before enabling AI features, and use Unity Gateway budgets for pay-per-token model access.
5. Pilot batch AI Functions on a sample and read the cost from system tables before running them on full tables.
6. Write pipelines against open APIs where possible (Spark Declarative Pipelines, Delta or Iceberg tables, MLflow) to keep exit costs low.
7. Schedule model redeployments so serving images pick up security patches.

Frequently asked questions

Is Databricks good for LLM and generative AI applications?

It is a capable platform for LLM work on governed enterprise data, through AI Functions, Model Serving, MLflow 3 tracing and evaluation, and Agent Bricks. Many of these features are newer, serverless-only and billed separately, and several are unavailable or depend on cross-region processing in Gulf regions.

What replaced Delta Live Tables?

Delta Live Tables is now Lakeflow Spark Declarative Pipelines. Databricks contributed the core framework to Apache Spark in 2025, and Spark 4.1.0 includes it as Spark Declarative Pipelines, so pipeline definitions are more portable than before.

How do you control Databricks costs for AI workloads?

Apply serverless usage policies for tagging, set account budgets with alerts, use Unity Gateway budgets with per-user limits where blocking is needed, pilot batch inference on samples, and monitor system.billing.usage, which typically lags by up to 12 hours.

Is Databricks available in Saudi Arabia and the UAE?

Yes, on Google Cloud in Dammam and Azure UAE North, and on Azure in Qatar. It is not available on AWS in the region. Model Serving, AI Functions and agent features are not offered in Dammam, and in UAE North many depend on cross-Geo processing.

Sources and further reading

• Lakeflow Spark Declarative Pipelines and limitations, Databricks; Lakeflow Connect concepts, Databricks.
• Declarative pipelines contributed to Apache Spark, Databricks, 12 June 2025; Spark 4.1.0 release notes, Apache Spark.
• Attribute-based access control in Unity Catalog, Databricks.
• Agent observability and quality with MLflow, Databricks; MLflow release history, PyPI.
• Model Serving and Model Serving limits, Databricks.
• AI Functions and ai_translate, Databricks.
• Agents on Databricks and Knowledge Assistant, Databricks.
• Lakebase Postgres, Databricks; Lakebase generally available, Databricks, 3 February 2026.
• Cost governance: budgets, serverless usage policies, billable usage system table, Databricks.
• Regions: Azure Databricks feature availability, Microsoft Learn; Databricks on Google Cloud feature availability; Azure Geographies and cross-Geo processing, Microsoft Learn; Databricks on AWS regions.