Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Simplifying Data Engineering and Analytics with Delta
  • Table Of Contents Toc
  • Feedback & Rating feedback
Simplifying Data Engineering and Analytics with Delta

Simplifying Data Engineering and Analytics with Delta

By : Anindita Mahapatra
4.9 (15)
Buy this Book
close
close
Simplifying Data Engineering and Analytics with Delta

Simplifying Data Engineering and Analytics with Delta

4.9 (15)
By: Anindita Mahapatra
Buy this Book

Overview of this book

Delta helps you generate reliable insights at scale and simplifies architecture around data pipelines, allowing you to focus primarily on refining the use cases being worked on. This is especially important when you consider that existing architecture is frequently reused for new use cases. In this book, you’ll learn about the principles of distributed computing, data modeling techniques, and big data design patterns and templates that help solve end-to-end data flow problems for common scenarios and are reusable across use cases and industry verticals. You’ll also learn how to recover from errors and the best practices around handling structured, semi-structured, and unstructured data using Delta. After that, you’ll get to grips with features such as ACID transactions on big data, disciplined schema evolution, time travel to help rewind a dataset to a different time or version, and unified batch and streaming capabilities that will help you build agile and robust data products. By the end of this Delta book, you’ll be able to use Delta as the foundational block for creating analytics-ready data that fuels all AI/BI use cases.
Table of Contents (18 chapters)
close
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Section 1 – Introduction to Delta Lake and Data Engineering Principles
Section 1 – Introduction to Delta Lake and Data Engineering Principles
Free Chapter
2
Chapter 1: Introduction to Data Engineering
chevron up
Chapter 1: Introduction to Data Engineering
The motivation behind data engineering
Understanding the role of data personas
Big data ecosystem
Evolution of data systems
Distributed computing
Business justification for tech spending
Summary
3
Chapter 2: Data Modeling and ETL
Chapter 2: Data Modeling and ETL
Technical requirements
What is data modeling and why should you care?
Understanding metadata – data about data
Moving and transforming data using ETL
How to choose the right data format
Common big data design patterns
Summary
Further reading
4
Chapter 3: Delta – The Foundation Block for Big Data
Chapter 3: Delta – The Foundation Block for Big Data
Technical requirements
Motivation for Delta
Demystifying Delta
The main features of Delta
Life with and without Delta
Summary
5
Section 2 – End-to-End Process of Building Delta Pipelines
Section 2 – End-to-End Process of Building Delta Pipelines
6
Chapter 4: Unifying Batch and Streaming with Delta
Chapter 4: Unifying Batch and Streaming with Delta
Technical requirements
Moving toward real-time systems
Streaming ETL
Handling streaming scenarios
Trade-offs in designing streaming architectures
Streaming best practices
Summary
7
Chapter 5: Data Consolidation in Delta Lake
Chapter 5: Data Consolidation in Delta Lake
Technical requirements
Why consolidate disparate data types?
Delta unifies all types of data
Avoiding patches of data darkness
Curating data in stages for analytics
Ease of extending to existing and new use cases
Data governance
Summary
8
Chapter 6: Solving Common Data Pattern Scenarios with Delta
Chapter 6: Solving Common Data Pattern Scenarios with Delta
Technical requirements
Understanding use case requirements
Minimizing data movement with Delta time travel
Delta cloning
Handling CDC
Handling Slowly Changing Dimensions (SCD)
Summary
9
Chapter 7: Delta for Data Warehouse Use Cases
Chapter 7: Delta for Data Warehouse Use Cases
Technical requirements
Choosing the right architecture
Understanding what a data warehouse really solves
Discovering when a data lake does not suffice
Addressing concurrency and latency requirements with Delta
Visualizing data using BI reporting
Analyzing tradeoffs in a push versus pull data flow
Considerations around data governance
The rise of the lakehouse category
Summary
10
Chapter 8: Handling Atypical Data Scenarios with Delta
Chapter 8: Handling Atypical Data Scenarios with Delta
Technical requirements
Emphasizing the importance of exploratory data analysis (EDA)
Applying sampling techniques to address class imbalance
Addressing data skew
Providing data anonymity
Handling bias and variance in data
Compensating for missing and out-of-range data
Monitoring data drift
Summary
11
Chapter 9: Delta for Reproducible Machine Learning Pipelines
Chapter 9: Delta for Reproducible Machine Learning Pipelines
Technical requirements
Data science versus machine learning
Challenges of ML development
Formalizing the ML development process
The role of Delta in an ML pipeline
From business problem to insight generation
Summary
12
Chapter 10: Delta for Data Products and Services
Chapter 10: Delta for Data Products and Services
Technical requirements
DaaS
The need for data democratization
Delta for unstructured data
Data mashups using Delta
Facilitating data sharing with Delta
Summary
13
Section 3 – Operationalizing and Productionalizing Delta Pipelines
Section 3 – Operationalizing and Productionalizing Delta Pipelines
14
Chapter 11: Operationalizing Data and ML Pipelines
Chapter 11: Operationalizing Data and ML Pipelines
Technical requirements
Why operationalize?
Understanding and monitoring SLAs
Scaling and high availability
Planning for DR
Guaranteeing data quality
Automation of CI/CD pipelines
Data as code – An intelligent pipeline
Summary
15
Chapter 12: Optimizing Cost and Performance with Delta
Chapter 12: Optimizing Cost and Performance with Delta
Technical requirements
Improving performance with common strategies
Optimizing with Delta
Is cost always inversely proportional to performance?
Best practices for managing performance
Summary
16
Chapter 13: Managing Your Data Journey
Chapter 13: Managing Your Data Journey
Provisioning a multi-tenant infrastructure
Data democratization via policies and processes
Capacity planning
Managing and monitoring
Data sharing
Data migration
COE best practices
Summary
Why subscribe?
17
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Chapter 1: Introduction to Data Engineering

"Water, water, everywhere, nor any drop to drink...

Data data everywhere, not a drop of insight!"

With the vast exodus of data around us, it is important to crunch it meaningfully and promptly to extract value from all the noise. This is where data engineering steps in. If collecting data is the first step, drawing useful insights is the next. Data engineering encompasses several personas that come together with their unique individual skill sets and processes to bring this to fruition. Data usually outlives the technology, and it continues to grow. New tools and frameworks come to the forefront to solve a lot of old problems. It is important to understand business requirements, the accompanying tech challenges, and typical shifts in paradigms to solve these age-old problems in a better manner.

By the end of this chapter, you should have an appreciation of the data landscape, the players, and the advances in distributed computing and cloud infrastructure that make it possible to support the high pace of innovation.

In this chapter, we will cover the following topics:

  • The motivation behind data engineering
  • Data personas
  • Big data ecosystem
  • Evolution of data stores
  • Trends in distributed computing
  • Business justification for tech spending
End of Section 1
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful
Order Page Read Now

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY