Sign In Start Free Trial
Account

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Book Overview & Buying Data Processing with Optimus
  • Table Of Contents Toc
  • Feedback & Rating feedback
Data Processing with Optimus

Data Processing with Optimus

By : Dr. Argenis Leon , Luis Aguirre Contreras
4.8 (4)
close
close
Data Processing with Optimus

Data Processing with Optimus

4.8 (4)
By: Dr. Argenis Leon , Luis Aguirre Contreras

Overview of this book

Optimus is a Python library that works as a unified API for data cleaning, processing, and merging data. It can be used for handling small and big data on your local laptop or on remote clusters using CPUs or GPUs. The book begins by covering the internals of Optimus and how it works in tandem with the existing technologies to serve your data processing needs. You'll then learn how to use Optimus for loading and saving data from text data formats such as CSV and JSON files, exploring binary files such as Excel, and for columnar data processing with Parquet, Avro, and OCR. Next, you'll get to grips with the profiler and its data types - a unique feature of Optimus Dataframe that assists with data quality. You'll see how to use the plots available in Optimus such as histogram, frequency charts, and scatter and box plots, and understand how Optimus lets you connect to libraries such as Plotly and Altair. You'll also delve into advanced applications such as feature engineering, machine learning, cross-validation, and natural language processing functions and explore the advancements in Optimus. Finally, you'll learn how to create data cleaning and transformation functions and add a hypothetical new data processing engine with Optimus. By the end of this book, you'll be able to improve your data science workflow with Optimus easily.
Table of Contents (16 chapters)
close
close
1
Section 1: Getting Started with Optimus
4
Section 2: Optimus – Transform and Rollout
10
Section 3: Advanced Features of Optimus

What this book covers

Chapter 1, Hi Optimus!, shows us what Optimus is, why it was created, and the goals of the project. We will get a good understanding of how Optimus works internally, how it is different from the current technology, and how it works in tandem with users to bring the best of the technology to serve users' data processing needs.

Chapter 2, Data Loading, Saving, and File Formats, is all about how to use Optimus to load and save data from text data formats such as CSV and JSON files. Also, we will explore binary files such as Excel and some optimized for columnar data processing such as Parquet, Avro, and OCR. Lastly, we will learn how to connect to databases such as SQLite and remote data storage such as Redshift.

Chapter 3, Data Wrangling, demonstrates how to concatenate data row- and column-wise and how to use SQL-like syntax to merge data using left, right, inner, and outer methods. Also, we'll learn how to pivot data tables to put data in the shape needed for the next step in the data pipeline.

Chapter 4, Combining, Reshaping, and Aggregating Data, teaches us how to group columns of data and apply summary statistics to each group of data. From count, min, and max aggregation to more advanced stats such as kurtosis and skew, we'll have all the tools to calculate any stats needed.

Chapter 5, Data Visualization and Profiling, demonstrates the profiler and the profiler data types, an Optimus DataFrame unique feature that lets the user have an overview of the data quality on every column. Data types such as email, dates, URLs, string, and float let the user easily standardize mismatches and missing data like any other library.

Chapter 6, String Clustering, uses string clustering techniques to let us easily identify groups of similar strings and replace them with a unique value.

Chapter 7, Feature Engineering, teaches us how to create new features for machine learning models to learn from. We'll look at generating them by combining fields, extracting values from messy columns, or encoding them for better results.

Chapter 8, Machine Learning, shows us how to easily create machine learning models and how Optimus will take care of the implementation details and make the feature engineering work when possible, as well as how to save the model after training and load it for future use.

Chapter 9, Natural Language Processing, shows us how to easily prepare data to apply techniques such as a word cloud, data summarization, and sentiment analysis, along with examples.

Chapter 10, Hacking Optimus, explores how to add new profiler data types to better approach quality problems. Also, we will learn how to create data cleaning and transformation functions and how to add a hypothetical new data processing engine. To close, we will talk about the Optimus Community, how to contribute to the project, and the next step in the Optimus project.

Chapter 11, Optimus as a Web Service, demonstrates how Optimus can be used as a web service with the help of various tools and plugins.

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech

Create a Note

Modal Close icon
You need to login to use this feature.
notes
bookmark search playlist download font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected

Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Delete Note

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete

Edit Note

Modal Close icon
Write a note (max 255 characters)
Cancel
Update Note

Confirmation

Modal Close icon
claim successful

Buy this book with your credits?

Modal Close icon
Are you sure you want to buy this book with one of your credits?
Close
YES, BUY