Databricks announces Dolly 2.0, the first open source, instruction-following LLM fine-tuned on a human-generated instruction corpus licensed for commercial use. Learn how you can get started!

Together, Databricks and MosaicML will enable any company to build, own and secure best-in-class generative AI models while maintaining control of their data

Introducing Dolly, a breakthrough in LLM from Databricks. Learn how Databricks open sourced the model and all its training code, enabling organizations to re-create Dolly at a minimal cost.

This blog post introduces the Pandas UDFs feature in the upcoming Apache Spark 2.3 release that substantially improves the performance and usability of user-defined functions (UDFs) in Python.

This post describes the advantage of the John Snow Labs’ Natural Language Processing library for Apache Spark and the use cases for which you should consider it for your own projects.

Read about Deep Learning Pipelines, an open-source library aimed at enabling everyone to easily integrate scalable deep learning into their workflows.

Building Enterprise GenAI Apps with Meta Llama 3 on Databricks

Databricks is launching its own large language model, DBRX, in an effort to help organizations understand and use their own private data to build their own AI systems. CEO and co-founder, Ali Ghodsi,...

Learn more about the Pytorch-based GPU-accelerated sentiment analysis package from Huggingface and how it leverages the Databricks platform to simplify and scale the analysis of text for sentiment.

Today, we announce the general availability of Apache Spark 2.0 on Databricks. This builds on what the community has learned in the past two years, doubling down on what users love and improving on...

Startup Databricks just released DBRX, the most powerful open source large language model yet—eclipsing Meta’s Llama 2.

Seven of 16 newcomers to Forbes’ annual Cloud 100 list are AI companies. More than half of Cloud 100 companies say they have launched a generative AI feature in the last eight months.

Designed in collaboration with Microsoft, Azure Databricks combines the best of Databricks and Azure to help customers accelerate innovation.

Lilac is joining Databricks! Lilac's core mission aligns with Databricks' commitment to provide customers with end-to-end generative AI capabilities.

We are thrilled to announce an entirely new and powerful development experience for the Databricks Lakehouse Platform: the Visual Studio Code Extension for Databricks.

We are excited to announce the public preview of Databricks Assistant, a context-aware AI assistant, that lets you query data through a conversational interface.

Today, we are thrill

Learn more about the new Photon and how its accelerating query performance on data lakes, especially those enabled by Delta Lake, making it easier to adopt lakehouse architectures.

Introducing the scikit-learn integration package for Apache Spark, designed to distribute the most repetitive tasks of model tuning on a Spark cluster, without impacting the workflow of data scient...

This is a guest Apache Spark community blog from Facebook Engineering. In this technical blog, authors Sital Kedia et.al share their usage of Apache Spark at terabyte scale in a production use case.

Databricks has announced a partnership and participation in the Series A funding of Mistral AI, one of Europe's leading providers of generative AI solutions. Databricks and Mistral AI now offer...

This is the fourth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark. In this blog, we will explore how easy it is to express streaming aggregations...

Learn how pandas users can run their workloads with distributed execution on a Spark cluster and how pandas API on Spark benefits users.

Introducing Dolly, a breakthrough in LLM from Databricks. Learn how Databricks open sourced the model and all its training code, enabling organizations to re-create Dolly at a minimal cost.

The Ben Horowitz-backed startup placed a huge and early bet on the power of machine learning 10 years ago.

Deep dive into Databricks’ new open source foundation model.

Get started with deep learning via a tutorial with helpful tips and resources, aimed at data scientists and engineers who need to run deep learning applications at scale.

Today we are happy t

Our viewpoint on how AI will fundamentally change data platforms, and how data will change enterprise AI

The core of Spark SQL is the Catalyst optimizer, which leverages advanced programming language features to build an extensible query optimizer.

Introducing the English SDK for Apache Spark - an innovative tool using English language input to streamline your development process, enhance code efficiency, and accelerate data insights.

Flint takes inspiration from an internal library at Two Sigma that has proven very powerful in dealing with time-series data on Apache Spark.

In this blog, we will show how Structured Streaming can be leveraged to consume and transform complex data streams from Apache Kafka. Together, you can use Apache Spark and Kafka to transform and...

Our viewpoint on how AI will fundamentally change data platforms, and how data will change enterprise AI

Databricks and Hugging Face integrate Apache Spark to more seamlessly load and transform data for AI model training and fine-tuning.

In this blog post, we introduce Spark Structured Streaming programming model in Apache Spark 2.0, a new high-level API that performs database-like query optimizations for building continuous applic...

In this first blog post in the series on Big Data at Databricks, we explore how we use Structured Streaming in Apache Spark 2.1 to monitor, process and productize low-latency and high-volume data...

We examine how Structured Streaming in Apache Spark 2.1 employs Spark SQL's built-in functions to allow you to consume data from many sources and formats (JSON, Parquet, NoSQL), and easily perform...

In this blog, we'll take a deep dive into Spark's Cost Based Optimizer (CBO) and discuss how Spark collects and stores these statistics, optimizes queries, and show its performance impact on TPC-DS...

Learn more about the record-shattering results achieved by Databricks SQL in 100TB TPC-DS, the key metric for data warehousing performance, and how it managed to beat the previous record by 2.2x...

This course is aimed at developers, data scientists, and engineers looking to build LLM-centric applications with the latest and most popular frameworks. By the end of this course, you will have...