-
Book Overview & Buying
-
Table Of Contents
-
Feedback & Rating

Data Processing with Optimus
By :

Saving on Optimus can be done by simply calling any of the methods available on the save
accessor of a dataframe instance. In this section, we'll learn how to save to a local or remote filesystem, and also to a previously established database or remote storage connection.
When saving data in files, it is important to understand which format to use so that you can gain speed when reading or processing. There is plenty of information available about how to select the correct date format. I like the following for simplicity:
"Finding the right file format for your particular dataset can be tough. In general, if the data is wide, has a large number of attributes, and is write-heavy, then a row-based approach may be best. If the data is narrower, has a fewer number of attributes, and is read-heavy, then a column-based approach may be best."
(Datanami, https://www.datanami.com/2018/05/16/big-data-file-formats-demystified/)