lakeFS Acquires DVC, Uniting Data Version Control Pioneers to Accelerate AI-Ready Data
TL;DR In the previous blog, Introducing the LangChain lakeFS Loader, and sample notebook, we explained and demonstrated integration of lakeFS with LangChain and LLM models (specifically OpenAI models). In this blog, we will explore a new beta feature from OpenAI that enables reproducible responses from a model. Introduction Language models are Stochastic models (stochastic refers […]
Data is a valuable asset to any company today. But can you really use this massive amount of data in its raw form to train ML algorithms? Not really. Most of the time, you’re looking at noisy data full of missing data points. This is where data preprocessing comes in. Data in the actual world
lakeFS is a powerful solution for data version control that enables data practitioners to manage data as code using Git-like operations and achieve reproducible, high-quality data pipelines. While getting started with lakeFS is simple through its Quickstart guide, many seek tailored examples that integrate with their existing tech stack or address specific use cases. To
Machine learning solutions come in handy for addressing various problems and achieving a wide range of goals. However, if we look at ML applications from a distance, we’ll instantly see that the fundamental components are almost always the same. Whether you want to better understand the skeleton of machine learning solutions or start building your
In a previous article, we explored how dbt and Databricks work together. We will now review a specific feature of dbt Cloud – dbt Cloud Jobs and demonstrate how to use Databricks workflows to achieve the same functionality. If you are using dbt-core with Databricks as your data platform, this article can give you an
Large Language Models (LLMs) are pretty straightforward to use when you’re prototyping. However, incorporating an LLM into a commercial product is an altogether different story. The LLM development lifecycle is made up of several complex components, including data intake, data preparation, engineering, model fine-tuning, model deployment, model monitoring, and more. The process also calls for
In the past five years, we’ve seen many concepts and new tools in the data ecosystem contribute to implementing engineering best practices in data. This trend includes the data mesh, data quality testing, observability, and data monitoring. The practices we would like to borrow from software engineering and use in data engineering and data science
Teams looking to build machine learning applications that are scalable, easy to maintain, and highly efficient can’t omit the step of building a machine learning architecture. Developing a solid ML architecture with a well-thought-out data pipeline results in better performance from machine learning algorithms, less time spent on experimentation, development, deployment, and maintenance, and less
Data is the core of any ML project. The vast volumes of machine learning data are the most important factor in training algorithms to deliver quality output. The problem is that all datasets are faulty to some extent. That’s why taking time and effort to prepare data is a good idea. Data preparation is a
TL;DR In this blog post, we will explore how to add data versioning to an ML project; a simple end-to-end rain prediction project for the Munich area. The data assets will be stored in lakeFS and we will use the lakeFS-spec Python package for easy interaction with lakeFS. Following model training with initial data, we
Since its inception, lakeFS shipped with a full featured Python SDK. For each new version of lakeFS, this SDK is automatically generated, relying on the OpenAPI specification published by the given version. While this always ensured the Python SDK shipped with all possible features, the automatically generated code wasn’t always the nicest (or most Pythonic)
Data-driven decision-making has become the foundation of business operations across every type of company, no matter the size or industry. Large volumes of data flow from many source systems to data warehousing, data lake, or analytics solutions. What companies need to maximize their ROI from data is a fast, dependable, scalable, and user-friendly space that
We use cookies to improve your experience and understand how our site is used.
Join our community of experts:
introduce yourself, share your knowledge and discover best practices from fellow peers
