Álvaro Martín Lozano
Implementing continuous delivery in a data processing pipeline
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
00:36 MIN
The distinct roles of CI and CD pipelines
#90DaysOfDevOps - The DevOps Learning Journey
Unlock full access
Log in or set up an account to access this feature and more.
06:04 MIN
Using continuous delivery to enable business agility
The Affordances of Quality
Unlock full access
Log in or set up an account to access this feature and more.
03:47 MIN
Applying software engineering practices to data pipelines
Enjoying SQL data pipelines with dbt
Unlock full access
Log in or set up an account to access this feature and more.
00:51 MIN
Defining continuous integration, delivery, and deployment
CI/CD with Github Actions
Unlock full access
Log in or set up an account to access this feature and more.
05:34 MIN
Tracing the evolution of DevOps from silos to superhighways
Navigating the AI Wave in DevOps
Unlock full access
Log in or set up an account to access this feature and more.
04:11 MIN
Adopting trunk-based development and continuous delivery
100 times more frequent deployments: How did we create a high performance team?
Unlock full access
Log in or set up an account to access this feature and more.
01:21 MIN
Testing against production with continuous deployment
How to Destroy a Monolith?
Unlock full access
Log in or set up an account to access this feature and more.
09:45 MIN
Implementing an AI-in-the-loop continuous learning cycle
A solution to embed container technologies into automotive environments
Unlock full access
Log in or set up an account to access this feature and more.
Featured Partners
Related Videos
30:46Charting the Journey to Continuous Deployment with a Value Stream Map
Josh Armitage
43:47Industrializing your Data Science capabilities
Dubravko Dolic & Hüdaverdi Cakir
56:19Plan CI/CD on the Enterprise level!
Pawel Piwosz
25:24Integrating backups into your GitOps Pipeline
Florian Trieloff
44:40DevOps for Machine Learning
Hauke Brammer
46:06CD2CF - Continuous Deployment to Cloud Foundry
Dominik Kress
47:22Continuous testing - run automated tests for every change!
Christian Kühn
39:15Database DevOps with Containers
Rob Richardson
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
ADMIRAL Technologies
Remote
€55K
Linux
Apache Kafka
Microsoft SQL Server