Apache Spark 3.3.0 是 3.x 系列的第四个版本。 在开源社区的巨大贡献下,此版本成功解决了超过 1,600 个 Jira 工单。
此版本通过 Bloom 过滤器提高了 Join 查询性能,通过支持流行的 Pandas 功能(如 datetime.timedelta 和 merge_asof)增加了 Pandas API 的覆盖范围,通过改进 ANSI 合规性并支持数十个新的内置函数来简化从传统数据仓库的迁移,并通过更好的错误处理、自动完成、性能和分析来提高开发效率。
要下载 Apache Spark 3.3.0,请访问下载页面。 您可以查阅 JIRA 以了解详细更改。 我们在此处整理了一份高级更改列表,按主要模块分组。
dropNamespace
API 的级联模式 (SPARK-37929)set catalog xxx
来更改当前目录 (SPARK-36841)支持 ILIKE (ALL | ANY | SOME) - 不区分大小写的 LIKE (SPARK-36674, SPARK-36736, SPARK-36778) |
TIMESTAMPDIFF()
的别名 (SPARK-38389)TIMESTAMPADD()
的别名 (SPARK-38332)GCM
设置为 aes_encrypt()/aes_decrypt() 中的默认模式 (SPARK-37666)mode
和 padding
参数添加到 aes_encrypt()/aes_decrypt() (SPARK-37586)ignoreCorruptFiles
和 ignoreMissingFiles
(SPARK-38767)AVG
(SPARK-37839)Sum
, Count
, Any
伴随 distinct,则无法进行部分聚合下推 (SPARK-38560)Limit
(SPARK-38768)SHOW CREATE TABLE
命令的 V1 和 V2 选项输出 (SPARK-37494)SHOW CATALOGS
(SPARK-35973)invert
(SPARK-36003)支持 str 和 timestamp 用于 (Series | DataFrame).describe() (SPARK-37657) |
DataFrame.rename
的 lambda column
参数 (SPARK-38763)SeriesGroupBy.value_counts
的 dropna
参数 (SPARK-38837)最后但并非最不重要的是,如果没有以下贡献者,此版本是不可能实现的:Abhishek Somani, Adam Binford, Alex Balikov, Alex Ott, Alfonso Buono, Allison Wang, Almog Tavor, Amin Borjian, Andrew Liu, Andrew Olson, Andy Grove, Angerszhuuuu, Anish Shrigondekar, Ankur Dave, Anton Okolnychyi, Aravind Patnam, Attila Zsolt Piros, BOOTMGR, BelodengKlaus, Bessenyei Balázs Donát, Bjørn Jørgensen, Bo Zhang, Brian Fallik, Brian Yue, Bruce Robbins, Byron, Cary Lee, Cedric-Magnan, Chandni Singh, Chao Sun, Cheng Pan, Cheng Su, Chia-Ping Tsai, Chilaka Ramakrishna, Daniel Dai, Daniel Davies, Daniel Tenedorio, Daniel-Davies, Danny Guinther, Darek, David Christle, Denis Tarima, Dereck Li, Devesh Agrawal, Dhiren Navani, Diego Luis, Dmitriy Fishman, Dmytro Melnychenko, Dominik Gehl, Dongjoon Hyun, Emil Ejbyfeldt, Enrico Minack, Erik Krogen, Eugene Koifman, Fabian A.J. Thiele, Franck Thang, Fu Chen, Geek, Gengliang Wang, Gidon Gershinsky, H. Vetinari, Haejoon Lee, Harutaka Kawamura, Herman van Hovell, Holden Karau, Huaxin Gao, Hyukjin Kwon, Igor Dvorzhak, IonutBoicuAms, Itay Bittan, Ivan Karol, Ivan Sadikov, Jackey Lee, Jerry Peng, Jiaan Geng, Jie, Johan Nystrom, Josh Rosen, Junfan Zhang, Jungtaek Lim, Kamel Gazzaz, Karen Feng, Karthik Subramanian, Kazuyuki Tanimura, Ke Jia, Keith Holliday, Keith Massey, Kent Yao, Kevin Sewell, Kevin Su, Kevin Wallimann, Koert Kuipers, Kousuke Saruta, Kun Wan, Lei Peng, Leona, Leona Yoda, Liang Zhang, Liang-Chi Hsieh, Linhong Liu, Lorenzo Martini, Luca Canali, Ludovic Henry, Lukas Rytz, Luran He, Maciej Szymkiewicz, Manu Zhang, Martin Tzvetanov Grigorov, Maryann Xue, Matthew Jones, Max Gekk, Menelaos Karavelas, Michael Chen, Michał Słapek, Mick Jermsurawong, Microsoft Learn Student, Min Shen, Minchu Yang, Ming Li, Mohamadreza Rostami, Mridul Muralidharan, Nicholas Chammas, Nicolas Azrak, Ole Sasse, Pablo Langa, Parth Chandra, PengLei, Peter Toth, Philipp Dallig, Prashant Singh, Qian.Sun, RabbidHY, Radek Busz, Rahul Mahadev, Richard Chen, Rob Reeves, Robert (Bobby) Evans, RoryQi, Rui Wang, Ruifeng Zheng, Russell Spitzer, Sachin Tripathi, Sajith Ariyarathna, Samuel Moseley, Samuel Souza, Sathiya KUMAR, SaurabhChawla, Sean Owen, Senthil Kumar, Serge Rielau, Shardul Mahadik, Shixiong Zhu, Shockang, Shruti Gumma, Simeon Simeonov, Steve Loughran, Steven Aerts, Takuya UESHIN, Ted Yu, Tengfei Huang, Terry Kim, Thejdeep Gudivada, Thomas Graves, Tim Armstrong, Tom van Bussel, Tomas Pereira de Vasconcelos, TongWeii, Utkarsh, Vasily Malakhin, Venkata Sai Akhil Gudesa, Venkata krishnan Sowrirajan, Venki Korukanti, Vitalii Li, Wang, Warren Zhu, Weichen Xu, Weiwei Yang, Wenchen Fan, William Hyun, Wu, Xiaochang, Xianjin YE, Xiduo You, Xingbo Jiang, Xinrong Meng, Xinyi Yu, XiuLi Wei, Yang He, Yang Liu, YangJie, Yannis Sismanis, Ye Zhou, Yesheng Ma, Yihong He, Yikf, Yikun Jiang, Yimin, Yingyi Bu, Yuanjian Li, Yufei Gu, Yuming Wang, Yun Tang, Yuto Akutsu, Zhen Li, Zhenhua Wang, Zimo Li, alexander_holmes, beobest2, bjornjorgensen, chenzhx, copperybean, daugraph, dch nguyen, dchvn, dchvn nguyen, dgd-contributor, dgd_contributor, dohongdayi, erenavsarogullari, fhygh, flynn, gaoyajun02, gengjiaan, herman, hi-zir, huangmaoyang2, huaxingao, hujiahua, jackierwzhang, jackylee-ch, jiaoqb, jinhai, khalidmammadov, kuwii, leesf, mans2singh, mcdull-zhang, michaelzhang-db, minyyy, nyingping, pralabhkumar, qitao liu, remykarem, sandeepvinayak, senthilkumarb, shane knapp, skhandrikagmail, sperlingxx, sudoliyang, sweisdb, sychen, tan.vu, tanel.kiis@gmail.com, tenglei, tianhanhu, tianlzhang, timothy65535, tooptoop4, vadim, w00507315, wangguangxin.cn, wangshengjie3, wayneguow, wooplevip, wuyi, xiepengjie, xuyu, yangjie01, yaohua, yi.wu, yikaifei, yoda-mon, zhangxudong1, zhoubin11, zhouyifan279, zhuqi-lucas, zwangsheng