May 2024 Update

Documentation

• Publish blog post about optimizing TRY and TRY_CAST

• Publish Technical Governance.

Core Library

• Optimize TRY and TRY_CAST for cases when many rows fail.

• Add support for LEFT SEMI FILTER and RIGHT SEMI FILTER merge joins.

• Add support for DECIMAL input to aggregations over distinct values. #9850

• Add support for specifying overwrite behavior when registering simple scalar functions. #9158

• Fix LEFT merge join with extra filter. #9862

• Fix Nested Loop join with empty build and extra filter. #9892

• Fix crash when copying complex vectors. #9725

• Fix reuse of LazyVectors in TableScan to avoid crashes in downstream operators. #9811

• Add support for including metadata identifying the operator that generated an error in error messages to ease troubleshooting. #9695

Presto Functions

• Add at_timezone() function.

• Add support for VARBINARY input to approx_distinct() aggregate function.

• Add support for CAST(varchar AS timestamp with time zone).

• Add support for DECIMAL inputs to min_by() and max_by() aggregate functions.

• Fix arrays_overlap() function for empty arrays. #9922

• Fix map_top_n() function to break ties by comparing keys.

• Fix CAST(IntervalDayTime AS Varchar) for negative intervals. #9871

• Fix from_base64() for inputs without padding. #8647

• Fix handling of equality and total ordering of NaN (Not-a-Number) floating point values in array_min(), array_sort(), array_distinct(), array_except(), array_intersect(), array_union(), array_position(), array_remove(), arrays_overlap(), contains(), map subscript and multimap_agg().

Spark Functions

• Add expm1(), get(), rint(), shuffle(), soundex(), unix_seconds(), width_bucket() functions.

• Add support for complex type inputs to hash() and xxhash64() functions.

• Fix CAST(tinyint/smallint/integer/bigint as varbinary). #9819

• Fix return type for sum() aggregate function with REAL input. #9818

Hive Connector

• Add support for projecting synthesized row-number column from Table Scan. #9174

Performance and Correctness

• Optimize memory arbitration to avoid interference between queries to reduce overall query execution time.

• Add cache expiration function to simple LRU cache to support remote IO throttling.

• Add Fuzzers for TableWriter and RowNumber operators.

• Add support for Nested Loop joins to Join Fuzzer.

• Add support for testing different sorting flags to Window Fuzzer.

• Add custom argument type generators for Presto decimal functions. #9715

• Add support for logical input types in the evaluateOnce() unit test helper method. #9708

• Re-enable testing of merge joins in Join Fuzzer.

Build System

• Upgrade aws-sdk-cpp to 1.11.321 (from 1.11.169).

• Upgrade cmake to 3.28.3 (from 3.14).

• Upgrade simdjson to 3.9.3 (from 3.8.0).

• Add support for docker image with Spark Connect server to use with Fuzzer. #9759

• Add dashboard with build time metrics.

Credits

Ankita Victor, Ashwin Krishna Kumar, Bikramjeet Vig, Bradley Dice, Daniel Munoz, Deepak Majeti, Giuseppe Ottaviano, Jacob Wujciak-Jens, Jia Ke, Jialiang Tan, Jimmy Lu, Joe Abraham, Karteekmurthys, Ke, Kevin Wilfong, Kk Pulla, Krishna Pai, Ma, Rong, Masha Basmanova, NEUpanning, PHILO-HE, Patrick Sullivan, Pedro Eugenio Rocha Pedreira, Richard Barnes, Sandino Flores, Sergey Pershin, Ubuntu, Wei He, Weihan Tang, Yang Zhang, Zac Wen, Zuyu ZHANG, aditi-pandit, chliang, duanmeng, gaoyangxiaozhu, jay.narale, joey.ljy, kevin, kikimo, lingbin, rui-mo, svm1, xiaoxmeng, xumingming, yan ma, yanngyoung, yingsu00, zhli1142015, zhouyifan279, zjuwangg, zky.zhoukeyong, 高阳阳