September 2023 Update

Documentation

• Add Chapter 2 of programming guide

• Add blog post about learning from optimizing try_cast

Core Library

• Add BaseVector::containsNullAt API #6515

• Add ContainerRowSerde::compareWithNulls API #6419

• Add Operator::initialize API to handle customized operation init #6422

• Add Pacific/Kanton, Europe/Kyiv, America/Ciudad_Juarez time zones #6670

• Add support for Array/Map/RowVector::copyRanges from UNKNOWN source #6607

• Add support for HUGEINT in function signatures #6799

• Add support for prioritizing productive sources in Exchange #6459

• Add Type::isOrderable() and Type::isComparable() APIs #6770

Presto Functions

• Add overlapped string support in strpos() function. #6410

• Add to_ieee754_64() function #6445

• Add support for all input types to IN predicate #6513

• Add support for UNKNOWN in map_from_entries() function #6353

• Add support for VARBINARY input to length() function #6795

• Add CAST DECIMAL to VARCHAR #6210

• Add CAST TIMESTAMP WITH TIME ZONE to and from TIMESTAMP #6529

• Add bitwise_xor_agg() aggregate function #6705

• Add geometric_mean() aggregate function #6678

• Add reduce_agg() aggregate function #6482

• Add support for UNKNOWN type in arbitrary() aggregate function #6557

• Add support for complex types to compare argument of min_by() and max_by() #6605

• Fix array_remove() null handling #6424

• Fix handling null string in json_extract() #6439

• Fix url_extract_path() function #6657

• Fix min(), max(), min_by(), and max_by() aggregates when input contains nested nulls #6723

• Fix set_union() aggregate #6800

• Fix lead() and lag() window function for int64 offset #6463

• Optimize exception handling in SIMD JSON functions #6776

Spark Functions

• Add comparison Spark functions #5569

• Add support for rand() function with seed specified #6112

• Add support for all patterns in CAST VARCHAR to DATE #5844

Hive Connector

• Add fast path of FlatMap column reader for scalar types #6507

• Add support for writing query results to S3 #6021

• Add support for writing sorted bucketed hive table #6142

• Pass TableScan output type to Parquet column reader #6404

• Reduce peak memory usage by roughly half in HiveDataSource #6601

Performance and Correctness

• Fix CPU time reporting by attributing LazyVector loading time to TableScan #6558

• Optimize TRY_CAST performance by reducing throwing #5913

• Optimize large chunk memory allocation in stream arena #6547

• Optimize spill performance by buffering spill data write #6509

• Tune memory arbitration and spilling for Velox batch use cases #6789

Build Systems

• Install Azure SDK needed to support Azure Storage ABFS Connector #6418

Credits

Ankita Victor, Artem Gelun, Austin Dickey, Bikramjeet Vig, Chengcheng Jin, Christian Zentgraf, Deepak Majeti, Ge Gao, George Wang, Jacob Wujciak-Jens, Jia Ke, Jialiang Tan, Jimmy Lu, Jubin Chheda, Karteekmurthys, Ke, Kevin Wilfong, Krishna Pai, Laith Sakka, Luca Niccolini, Mahadevuni Naveen Kumar, Manav Avlani, Manikandan Somasundaram, Masha Basmanova, Muir Manders, Orri Erling, PHILO-HE, Patrick Stuedi, Patrick Sullivan, Pedro Pedreira, Pramod, Pratik Joseph Dabre, Rong Ma, Sergey Pershin, Shanyue Wan, Shiyu Gan, Srikrishna Gopu, Surabhi Pandit, Wei He, Yangyang Gao, Zac, Zac Wen, aditi-pandit, duanmeng, ericyuliu, generatedunixname89002005232357, joey.ljy, lingbin, mayan, rui-mo, usurai, wypb, xiaodou, xiaoxmeng, xumingming, yingsu00, zhli1142015, 陈旭, 高阳阳