March 2024 Update

Documentation

• Document design philosophy

• Document custom input generators and verifiers supported in the Aggregation Fuzzer.

• Document runtime stats reported by the HashTable. #9255

• Document usage of generic types in Simple Function API. #9084

Core Library

• Add prefix-sort for fixed width sorting keys.

• Add null behavior and determinism scalar function metadata to the registry. #9209

• Add order-sensitive aggregate function metadata to the registry. #9050

• Add support for DECIMAL type to Simple Function API. #9096

• Add support for lambda functions (reduce_agg) to StreamingAggregation.

• Deprecate threshold based spilling in Aggregation and OrderBy.

• Optimize Exchange protocol used by Presto for latency. #8845

Presto Functions

• Add day(), from_ieee754_32(), hamming_distance(), map_normalize(), map_top_n() scalar functions.

• Add support for DECIMAL input type to floor() function.

• Add support for timestamp +/- IntervalYearMonth.

• Add regr_avgx(), regr_avgy(), regr_count(), regr_r2(), regr_sxx(), regr_sxy(), and regr_syy() aggregation functions.

Spark Functions

• Add array_remove(), bit_length(), bitwise_xor(), bitwise_not(), make_ym_interval(), from_utc_timestamp(), to_utc_timestamp(), make_timestamp(), map_subset(), unhex(), unix_date(), uuid() functions.

• Add regexp_replace() function.

• Add monotonically_increasing_id(), spark_partition_id() functions.

• Add kurtosis() and skewness() aggregation functions.

• Add support for DECIMAL inputs to sum() aggregation function.

• Add CAST(real as decimal).

• Add configuration property ‘spark.partition_id’.

Hive Connector

• Add support for S3 client no_proxy CIDR expression. #9160

• Add support for synthetic columns ‘$file_size’ and ‘$file_modified_time’.

• Optimize reading a small sample of rows. #8920.

• Fix Parquet reader for files with different encodings across row groups. #9129

Performance and Correctness

• Add nightly run of Aggregation fuzzer using Presto as source of truth.

• Add nightly run of Exchange fuzzer.

• Add utility to randomly trigger OOMs and integrate it into Aggregation and Join fuzzers.

• Add group execution mode to Join fuzzer.

• Add support for random frame clause generation to Window fuzzer.

• Add custom input generator for map_union_sum Presto aggregation function.

• Add custom result verifier for arbitrary Presto aggregation function.

Credits

8dukongjian, Amit Dutta, Ankita Victor, Bikramjeet Vig, Christian Zentgraf, Daniel Munoz, Deepak Majeti, Ge Gao, InitialZJ, Jacob Wujciak-Jens, Jake Jung, Jialiang Tan, Jimmy Lu, Karteekmurthys, Kevin Wilfong, Krishna Pai, Ma, Rong, Mahadevuni Naveen Kumar, Marcus D. Hanwell, Masha Basmanova, Nicholas Ormrod, Nick Terrell, Orri Erling, PHILO-HE, Patrick Sullivan, Pedro Pedreira, Pramod, Pratik Joseph Dabre, Qian Sun, Richard Barnes, Sandino Flores, Schierbeck, Cody, Sergey Pershin, Ubuntu, Wei He, Yang Zhang, Zac Wen, aditi-pandit, duanmeng, f0rest9999, hengjiang.ly, joey.ljy, lingbin, mwish, rexan, rui-mo, willsfeng, wypb, xiaodai1002, xiaoxmeng, xumingming, youxiduo, yuling.sh, zhli1142015, zky.zhoukeyong