#What Is The Advantage Of A Parquet File #spark #bigdata #parquet file format #parquet files

Advantages of Parquet Columnar Storage or parquet file format

The above characteristics of the Apache Parquet file format create several distinct benefits when it comes to storing and analyzing large volumes of data. Let’s look at some of them in more depth.

Compression

File compression is the act of taking a file and making it smaller. In Parquet, compression is performed column by column and it is built to support flexible compression options and extendable encoding schemas per data type – e.g., different encoding can be used for compressing integer and string data.

parquet file format data can be compressed using these encoding methods:

• Dictionary encoding: this is enabled automatically and dynamically for data with a small number of unique values.

• Bit packing: Storage of integers is usually done with dedicated 32 or 64 bits per integer. This allows more efficient storage of small integers.
• Run length encoding (RLE): when the same value occurs multiple times, a single value is stored once along with the number of occurrences. Parquet implements a combined version of bit packing and RLE, in which the encoding switches based on which produces the best compression results