Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Data Engineering with Databricks Cookbook
  • Table Of Contents Toc
  • Feedback & Rating feedback
Data Engineering with Databricks Cookbook

Data Engineering with Databricks Cookbook

By : Pulkit Chadha
4.4 (7)
close
close
Data Engineering with Databricks Cookbook

Data Engineering with Databricks Cookbook

4.4 (7)
By: Pulkit Chadha

Overview of this book

Written by a Senior Solutions Architect at Databricks, Data Engineering with Databricks Cookbook will show you how to effectively use Apache Spark, Delta Lake, and Databricks for data engineering, starting with comprehensive introduction to data ingestion and loading with Apache Spark. What makes this book unique is its recipe-based approach, which will help you put your knowledge to use straight away and tackle common problems. You’ll be introduced to various data manipulation and data transformation solutions that can be applied to data, find out how to manage and optimize Delta tables, and get to grips with ingesting and processing streaming data. The book will also show you how to improve the performance problems of Apache Spark apps and Delta Lake. Advanced recipes later in the book will teach you how to use Databricks to implement DataOps and DevOps practices, as well as how to orchestrate and schedule data pipelines using Databricks Workflows. You’ll also go through the full process of setup and configuration of the Unity Catalog for data governance. By the end of this book, you’ll be well-versed in building reliable and scalable data pipelines using modern data engineering technologies.
Table of Contents (16 chapters)
close
close
Free Chapter
1
Part 1 – Working with Apache Spark and Delta Lake
9
Part 2 – Data Engineering Capabilities within Databricks

What this book covers

Chapter 1, Data Ingestion and Data Extraction with Apache Spark, explores the fundamental processes of data ingestion and extraction using Apache Spark. From connecting to various data sources to efficiently extracting and loading data, you will gain hands-on experience in leveraging Apache Spark’s capabilities for seamless data integration.

Chapter 2, Data Transformation and Data Manipulation with Apache Spark, delves into the transformative power of Apache Spark, focusing on data transformation and manipulation techniques. You will learn how to harness Spark’s robust functionalities for reshaping and optimizing data, ensuring it aligns with specific business requirements and analytical needs.

Chapter 3, Data Management with Delta Lake, delves into Delta Lake, a critical component for effective data management. You will discover how to leverage Delta Lake’s ACID transactions and versioning capabilities to ensure data reliability, consistency, and efficient management within the Lakehouse architecture.

Chapter 4, Ingesting Streaming Data, initiates the exploration of ingesting streaming data using Apache Spark. It covers the basics of streaming data ingestion, setting the stage for understanding real-time data processing and analysis.

Chapter 5, Processing Streaming Data, completes the exploration of streaming data by focusing on advanced techniques and best practices for processing real-time data with Apache Spark. You will gain insights into handling dynamic data streams and maintaining data integrity in dynamic, fast-paced environments.

Chapter 6, Performance Tuning with Apache Spark, delves into the intricacies of performance tuning in Apache Spark. From optimizing code to fine-tuning configurations, you will learn practical strategies to enhance the efficiency and speed of Spark applications, ensuring optimal performance for large-scale data processing.

Chapter 7, Performance Tuning in Delta Lake, builds upon performance tuning principles and focuses specifically on optimizing Delta Lake workflows. You will gain insights into techniques for improving the speed and efficiency of data transactions, making data management within the Lakehouse architecture more performant.

Chapter 8, Orchestration and Scheduling Data Pipeline with Databricks Workflows, guides you through the orchestration and scheduling of workflows in Databricks. From designing automated data pipelines to scheduling tasks efficiently, you will learn how to streamline your data engineering processes and ensure the timely execution of critical workflows.

Chapter 9, Building Data Pipelines with Delta Live Tables, helps you explore the innovative Delta Live Tables, showing how to build robust and dynamic data pipelines. The focus is on leveraging Delta Live Tables to simplify data pipeline development, enhance collaboration, and ensure data consistency in real time.

Chapter 10, Data Governance with Unity Catalog, introduces the concept of data governance using Unity Catalog in Databricks. You will discover how to implement effective data governance practices, including metadata management, data lineage tracking, and access control, to ensure data quality and compliance.

Chapter 11, Implementing DataOps and DevOps on Databricks, addresses the integration of DataOps and DevOps practices within the Databricks environment. You will learn how to implement collaborative and automated development and deployment processes, fostering a culture of continuous improvement and efficiency in data engineering workflows.

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
bookmark search playlist download font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY