Startup Profile: LakeSail

As big data joins the AI infrastructure, developers are hitting the limitations of software that no longer keeps pace. Apache Spark, once key to processing large amounts of data in enterprise applications, is over 15 years old and showing its age. Its Java Virtual Machine (JVM)-based structure makes it slow and expensive to run in cloud environments, where access to data lakes must be virtually instantaneous.

Enter LakeSail, a three-year-old startup based in San Francisco with a better idea than Spark. According to CEO and cofounder Shehab Amin, the company’s Sail product is “an open-source, Rust-native distributed computation framework that serves as a high-performance, drop-in replacement for Apache Spark.” The goal, LakeSail says on GitHub, is to “unify batch processing, stream processing, and AI workloads.”

There’s a lot to unpack in those descriptions. First off is the structure of Sail, which uses a stateless approach to bringing data from one location to another. An integral Model Context Protocol (MCP) server, combined with Sail’s Rust-based structure, supports agentic AI. In a blog posted last year, CEO Amin and cofounders Heran Lin and Everett Roeth stated:

“Beyond the usual compute workloads, the MCP server makes it simple and straightforward for any AI agent to query massive datasets and receive live responses, all in a natural, chat-like workflow.”

Then there’s the open-source nature of Sail. The engine is available on GitHub, though the company offers a supported version, managed in the cloud. The software is also written entirely in Rust—a significant feature, thanks to Rust’s high performance, safety and reliability compared to other languages. Rust’s efficiency drives lower cloud costs and is widely acclaimed for its substantial energy efficiency thanks to requiring fewer compute resources. Sail also utilizes Apache DataFusion, a library written in Rust that speeds up the processing of queries in big data workloads.

Sail is based on Rust but uses the Apache Spark Connect protocol to give users a Python interface. Developers don’t have to change the coding they originally used with Apache Spark—hence the drop-in nature of Sail, which is also compatible with Apache Iceberg and Databricks’ Delta Lake data formats. But Sail outperforms Spark dramatically: LakeSail claims that it improves throughput by anywhere from 4X to 8X compared to Spark and helps achieve 94% lower infrastructure costs—all while maintaining full compatibility with Spark SQL and DataFrame APIs, which allow developers to address data in rows and tables.

Building Momentum in the Market

LakeSail was founded in 2023 in San Francisco by former UC Berkeley researcher Amin, along with Lin and Roeth. LakeSail’s seed investments remain on the QT, though they include contributions from Mango Capital and a raft of private investors.

Though it seems to be barely out of stealth, LakeSail is catching on in popularity with big-data developer cognoscenti. The word is spreading about Sail’s capabilities to streamline data processing in AI workloads. At least one large-scale European online payments processing firm has deployed Sail. Other examples of its potential use include live-data healthcare monitoring, government agencies, and online e-commerce marketing firms tracking daily transactions.

LakeSail enters a hot market, competing against rivals such as Databricks and Snowflake. Unlike those solutions, LakeSail is adaptable to a range of data sources without the extra costs associated with those cloud platforms. LakeSail also claims to outperform its rivals in common benchmarks, sometimes by up to 100X, since some competing solutions still rely on underlying JVM structures.