lakeFS Acquires DVC, Uniting Data Version Control Pioneers to Accelerate AI-Ready Data
In today’s data-driven world, businesses don’t just rely on data – they are built on it. But as data infrastructure sprawls across on-prem systems, multiple cloud providers, and third-party platforms, a new challenge is taking center stage: distributed data management. It’s a silent bottleneck with loud consequences. Challenges in Distributed Data Management Managing data across […]
When I first joined lakeFS by Treeverse in 2020, we were just four engineers building an open-source solution for data versioning. It was exhilarating—being part of something from the ground up, shaping the product, and seeing it grow. But after four years, something changed. The excitement faded, and I felt like I was running in
Originally presented at Big Data LDN 2024. More than two decades ago, data warehouses outgrew the capacity of single machines, and scaling them started to become costly or inefficient. This prompted the tech industry to rethink the architecture and start to use distributed systems. If we wanted to store more data, we just bought more
Since 2021 we’ve been releasing the annual State of Data Engineering Report, a compilation of all the relevant categories that have a direct impact on data engineering infrastructure. In 2024, we see 3 primary trends that influence the categories which will be covered in this report. Trend #1: GenAI influence on software infrastructure As predicted
TL;DR: DataOps complexity arises from unclear R&Rs, a lack of standardization in interfaces, distributed technology complexities, and difficulties in implementing engineering best practices. The solution is to define clear responsibilities, address missing requirements, and manage data pipelines efficiently using emerging solutions that enhance the manageability and resilience of DataOps. What makes DataOps so hard is,
Git for data may sound odd at first. But using the Git logic and mechanisms for data lakes makes a lot of sense. After all, software developers figured out long ago how to efficiently collaborate on constantly changing source code with Git. We created lakeFS to provide similar capabilities to data folks. It provides data
This article was originally published on Datanami and is republished here with permission. As we officially kick off 2024, I realized I have a few thoughts on the direction of the data landscape that might be of interest to others. This is a recap of my “predictions.” I will admit that it’s a mix of what I
A data lake is often implemented using object storage, and our data resides in objects (files) that we can access through the storage API. Data professionals in analytics and ML are used to consuming data from a database, where there is an abstraction of a table one can access using SQL. They will find it
October 24, 2024 — TL;DR lakeFS 1.0 is generally available. You can upgrade to this version via the Assets here. Just over three years ago we publicly released the initial lakeFS. Our mission when we first launched lakeFS was to provide data practitioners with a scalable data version control system that would bring order to
This post is a recap of a talk I gave at this year’s Data + AI Summit about why I believe the Rust Programming Language and related novel technologies such as WebAssembly will play a large part in the data ecosystem in the coming years. The talk covers: The “present” of analytics (from my perspective,
A lot has happened since 2022, from the rise of Generative AI to the economic slowdown and job losses impacting data practitioners in 2023. In fact, it’s safe to say that the GenAI hype doesn’t seem to be high enough to counterbalance the influence of the struggling economy, as the demand for data practitioners is
VC surveys show that the MLOps category has significantly expanded in the past few years, with hundreds of companies defining themselves as part of this dynamically evolving niche. MLOps systems provide the infrastructure allowing ML practitioners to manage the lifecycle of their work from development to production in a robust and reproducible manner. An MLOps
We use cookies to improve your experience and understand how our site is used.
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
