January 2024 Update


Core Library

  • Add metrics to track time spent in memory arbitration. #8497, #8482

  • Add metric to track average buffer time for exchange. #8534

  • Optimize count(distinct x) when x is of complex type. #8560

  • Optimize latency for exchange that uses arbitrary buffer. #8532, #8480

  • Optimize MallocAllocator to reduce lock contention. #8477

  • Fix aggregation over all-null keys with ignoreNullKeys = true. #8422

  • Fix race condition in task completion that caused Output buffers for task not found failures. #8357

  • Fix evaluation of CAST expression under TRY. #8365

  • Fix FlatVector<StringView>::copy for vectors with more than 2GB of data. #8516

  • Fix crash in FlatVector<bool>::ensureWritable. #8450

  • Fix interaction of spilling and yielding in Hash Join operator. #8520

  • Fix rawInputPositions metrics in Exchange operator. #8370

Presto Functions

Spark Functions

Hive Connector

  • Fix crash in Parquet reader when processing empty row groups. #8000

  • Fix data sink to avoid writing partition columns to files. #8089

Performance and Correctness

  • Add support for aggregations over distinct inputs to AggregationFuzzer.

  • Reduce memory usage of histogram metrics. #8458

  • Add Join Fuzzer run to CI that runs on each PR.

  • Add Aggregation Fuzzer run using Presto as source of truth to experimental CI.

Build System

  • Upgrade folly to v2023.12.04.00 (from v2022.11.14.00).

  • Upgrade fmt to 10.1.1 (from 8.0.1).


Amit Dutta, Benwei Shi, Bikramjeet Vig, Chen Zhang, Chengcheng Jin, Christian Zentgraf, Deepak Majeti, Ge Gao, Hongze Zhang, Jacob Wujciak-Jens, Jia Ke, Jialiang Tan, Jimmy Lu, Ke, Kevin Wilfong, Krishna Pai, Laith Sakka, Lu Niu, Ma, Rong, Masha Basmanova, Mike Lui, Orri Erling, PHILO-HE, Pedro Eugenio Rocha Pedreira, Pratik Joseph Dabre, Ravi Rahman, Richard Barnes, Schierbeck, Cody, Sergey Pershin, Sitao Lv, Taras Galkovskyi, Wei He, Yedidya Feldblum, Yuan Zhou, Yuping Fan, Zac Wen, aditi-pandit, binwei, duanmeng, hengjiang.ly, icejoywoo, lingbin, mwish, rui-mo, wypb, xiaoxmeng, xumingming, yangchuan, yingsu00, youxiduo, yuling.sh, zhli1142015, zky.zhoukeyong, zwangsheng