Explore Zstandard compression

[sebastian-nagel]

(depends on #10)

Zstandard compression is directly integrated into Parquet (PARQUET-1866). Also Athena now supports Zstd as Parquet compression. Time to explore whether switching from gzip to zstd brings improvements in storage size or query processing speed. In addition to Athena and Spark, support for zstd should be verified also Hive, Arrow, Pandas and other common tools to read/process Parquet files.

[sebastian-nagel]

By passing --outputCompression=zstd the Parquet files are compressed using Zstandard compression. However, the zstd-compressed files are slightly bigger compared to gzip. There is no effect on the Parquet file size visible by adjusting the zstd compression level (eg. --conf spark.io.compression.zstd.level=22) - this option might only affect other output formats.