【hive】行转列—explode()/posexplode()/lateral view 函数使用场景

2023-09-13 15:22:30


一、lateral view函数

  • 功能:

    • 用于和UDTF函数(explode,split)结合使用,把某一行数据拆分成多行数据,再将多行结果组合成一个支持别名的虚拟表。
    • 主要解决select使用UDTF做查询的过程中查询只能包含单个UDTF,不能包含其它字段以及多个UDTF的情况。
  • 语法:LATERAL VIEW udtf(expression) tableAlias AS columnAlias (‘,’ columnAlias)

    • columnAlias是给udtf(expression)列起的别名。
    • tableAlias 虚拟表的别名。
  • lateral view函数会将lUDTF生成的结果放到一个虚拟表中,然后这个虚拟表和输入行进行join来达到连接UDTF外的select字段的目的。

使用方式:

举例1:与explode() 函数连用

--测试表的数据结构,可以看到第一列是id(string类型),第二列是rule_array(array<string>类型)
> desc test_2;

--结果:
+-------------+----------------+----------+
|  col_name   |   data_type    | comment  |
+-------------+----------------+----------+
| id          | string         |          |
| rule_array  | array<string>  |          |
+-------------+----------------+----------+

--查看测试表的内容
> select * from test_2;

--结果:
+------------+----------------------------------------+
| test_2.id  |           test_2.rule_array            |
+------------+----------------------------------------+
| a          | ["501","502","503","501","512"]        |
| b          | ["511","512","513","511","512","511"]  |
| c          | ["512","513","511","512"]              |
+------------+----------------------------------------+

--拆分rule_array
> select id, rule_explode 
> from test_2
> lateral view explode(rule_array) adTable AS rule_explode;

--结果:
+-----+---------------+
| id  | rule_explode  |
+-----+---------------+
| a   | 501           |
| a   | 502           |
| a   | 503           |
| a   | 501           |
| a   | 512           |
| b   | 511           |
| b   | 512           |
| b   | 513           |
| b   | 511           |
| b   | 512           |
| b   | 511           |
| c   | 512           |
| c   | 513           |
| c   | 511           |
| c   | 512           |
+-----+---------------+

举例2:与parse_url_tuple()函数连用

  • parse_url_tuple()函数
    • 语法:parse_url(string urlString, string partToExtract [, string keyToExtract])
    • 功能:UDTF函数,解析URL字符串,它通过一次指定多个参数,从URL中解析出多个参数值返回多列。
    • 参数:partToExtract的有效值为:HOST, PATH, QUERY, REF, PROTOCOL, AUTHORITY, FILE, and USERINFO。
    • 注意:在使用 select 时不能同时返回其他字段,不能嵌套调用,不能与group by放在一起调用等。为解决此问题,可以通过测试图(Lateral View)搭配调用。
--测试表的数据结构,可以看到第一列是id(int类型),第二列是url(string类型)
> desc tb_url;

--结果:
+-----------+------------+----------+
| col_name  | data_type  | comment  |
+-----------+------------+----------+
| id        | int        |          |
| url       | string     |          |
+-----------+------------+----------+

--查看测试表的内容
> select * from tb_url;

--结果:
+------------+-------------------------------------------------+
| tb_url.id  |                   tb_url.url                    |
+------------+-------------------------------------------------+
| 1          | http://facebook.com/path/p1.php?query=1         |
| 2          | http://www.baidu.com/news/index.jsp?uuid=frank  |
| 3          | http://www.jd.com/index?source=baidu            |
+------------+-------------------------------------------------+

--提取tb_url的query部分
> select
> a.id as id,
> c.query as query
> from tb_url a
> lateral view parse_url_tuple(url,"QUERY") c as query
> ;

--结果:
+-----+---------------+
| id  |     query     |
+-----+---------------+
| 1   | query=1       |
| 2   | uuid=frank    |
| 3   | source=baidu  |
+-----+---------------+

举例3:多个lateral view连用

--提取tb_url的HOST、PATH和QUERY部分
> select
> a.id as id,
> b.host as host,
> b.path as path,
> c.query as query
> from tb_url a
> lateral view parse_url_tuple(url,"HOST","PATH") b as host,path
> lateral view parse_url_tuple(url,"QUERY") c as query
> ;

--结果:
+-----+----------------+------------------+---------------+
| id  |      host      |       path       |     query     |
+-----+----------------+------------------+---------------+
| 1   | facebook.com   | /path/p1.php     | query=1       |
| 2   | www.baidu.com  | /news/index.jsp  | uuid=frank    |
| 3   | www.jd.com     | /index           | source=baidu  |
+-----+----------------+------------------+---------------+

注意:

  • lateral view函数的位置在from后where条件前。
  • 生成虚拟表的别名不可省略。
  • from后可带多个 lateral view函数。
  • 如果要拆分的字段有null值,需要使用 lateral view outer替代,避免数据缺失。

二、explode()函数

  • 功能:UDTF函数,可以将一个array或者map展开
  • 语法:explode(map<string> ,array <string>)
    • explode(array):将array列表里的每个元素生成一行
    • explode(map):每一对元素作为一行,key为一列,value为一列

使用方式:

举例1:直接使用

--explode(array)
> select explode(array(11,22,33)) as item;

--结果:
+-------+
| item  |
+-------+
| 11    |
| 22    |
| 33    |
+-------+

--explode(map)
> select explode(map("id",11,"name","lily","age",18)) as (k,v);

--结果:
+-------+-------+
|   k   |   v   |
+-------+-------+
| id    | 11    |
| name  | lily  |
| age   | 18    |
+-------+-------+

举例2:与lateral view一起使用

  • 语法:lateral view explode(字段) 表别名 as 列别名
  • 如果字段类型是map:lateral view explode(字段) 表别名 as (key别名,value别名)
> select * from test_2;

--结果:
+------------+----------------------------------------+
| test_2.id  |           test_2.rule_array            |
+------------+----------------------------------------+
| a          | ["501","502","503","501","512"]        |
| b          | ["511","512","513","511","512","511"]  |
| c          | ["512","513","511","512"]              |
+------------+----------------------------------------+

> select id,rule
> from test_2 
> lateral view explode(rule_array) tbl as rule
> ;

--结果:
+-----+-------+
| id  | rule  |
+-----+-------+
| a   | 501   |
| a   | 502   |
| a   | 503   |
| a   | 501   |
| a   | 512   |
| b   | 511   |
| b   | 512   |
| b   | 513   |
| b   | 511   |
| b   | 512   |
| b   | 511   |
| c   | 512   |
| c   | 513   |
| c   | 511   |
| c   | 512   |
+-----+-------+

举例3:使用局限性

  • 不能关联原有的表中其它字段。例如:
> select id, explode(rule_array) as rule from test_2;

--报错:
Error: Error while compiling statement: FAILED: SemanticException [Error 10081]: UDTF's are not supported outside the SELECT clause, nor nested in expressions (state=42000,code=10081)
  • 不能与group by、cluster by 、distribute by、sort by联用。例如:
> select explode(rule_array) as rule from test_2 group by explode(rule_array);

--报错:
Error: Error while compiling statement: FAILED: SemanticException [Error 10081]: UDTF's are not supported outside the SELECT clause, nor nested in expressions (state=42000,code=10081)
  • 不能进行UDTF嵌套。例如:
> select explode(explode(rule_array)) from test_2 ;

--报错:
Error: Error while compiling statement: FAILED: SemanticException [Error 10081]: UDTF's are not supported outside the SELECT clause, nor nested in expressions (state=42000,code=10081)
  • 不允许选择其他表达式。例如:
> select explode("1,2,3,4,5") from test_2 ; 

--报错:
Error: Error while compiling statement: FAILED: UDFArgumentException explode() takes an array or a map as a parameter (state=42000,code=40000)

三、posexplode()函数

  • 功能:UDTF函数,将一个array或者map展开,可以将index(索引)和数据都取出来,使用两次posexplode()并用where语句使两次取到的index相等,从而实现对多列进行多行转换。explode()函数只能将对一列进行行转换。
  • 语法:posexplode(map<string> ,array <string>)

使用方式:

举例1:

> desc test_2;

--结果:
+-------------+----------------+----------+
|  col_name   |   data_type    | comment  |
+-------------+----------------+----------+
| id          | string         |          |
| dt_array    | array<string>  |          |
| rule_array  | array<string>  |          |
+-------------+----------------+----------+

> select * from test_2;

--结果:
+------------+----------------------------------------------------+----------------------------------------+
| test_2.id  |                  test_2.dt_array                   |           test_2.rule_array            |
+------------+----------------------------------------------------+----------------------------------------+
| a          | ["20230809","20230811","20230812","20230812","20230813"] | ["501","502","503","501","512"]        |
| b          | ["20230809","20230811","20230812","20230812","20230813","20230809"] | ["511","512","513","511","512","511"]  |
| c          | ["20230811","20230812","20230812","20230813"]      | ["512","513","511","512"]              |
+------------+----------------------------------------------------+----------------------------------------+

--注:语句中b.idxb和c.idxc分别是子视图的两个索引,我们where子句中使用b.idxb=c.idxc保证顺序的一致性。
> select a.id,b.cur_day,c.rule 
> from test_2 a
> lateral view posexplode(dt_array) b as idxb,cur_day,
> lateral view posexplode(rule_array) c as idxc,rule where b.idxb=c.idxc
> ;

--结果:
+-------+------------+---------+
| a.id  | b.cur_day  | c.rule  |
+-------+------------+---------+
| a     | 20230809   | 501     |
| a     | 20230811   | 502     |
| a     | 20230812   | 503     |
| a     | 20230812   | 501     |
| a     | 20230813   | 512     |
| b     | 20230809   | 511     |
| b     | 20230811   | 512     |
| b     | 20230812   | 513     |
| b     | 20230812   | 511     |
| b     | 20230813   | 512     |
| b     | 20230809   | 511     |
| c     | 20230811   | 512     |
| c     | 20230812   | 513     |
| c     | 20230812   | 511     |
| c     | 20230813   | 512     |
+-------+------------+---------+

四、行转列使用

单列转多行

举例1:使用explode()+lateral view

> select t.id,a.cur_day
> from test_2 t
> lateral view explode(t.dt_array) a as cur_day
> ;

--结果:
+-------+------------+
| t.id  | a.cur_day  |
+-------+------------+
| b     | 20230809   |
| b     | 20230811   |
| b     | 20230812   |
| b     | 20230812   |
| b     | 20230813   |
| b     | 20230809   |
| c     | 20230811   |
| c     | 20230812   |
| c     | 20230812   |
| c     | 20230813   |
| a     | 20230809   |
| a     | 20230811   |
| a     | 20230812   |
| a     | 20230812   |
| a     | 20230813   |
+-------+------------+

举例2:explode字段是string格式,先split()指定分隔符,如果省略split()则使用默认分隔符是英文逗号。

> desc test_3;

--结果:
+-----------+------------+----------+
| col_name  | data_type  | comment  |
+-----------+------------+----------+
| id        | string     |          |
| dt        | string     |          |
| rule1     | string     |          |
+-----------+------------+----------+

> select * from test_3;

--结果:
+------------+----------------------------------------------------+--------------------------+
| test_3.id  |                     test_3.dt                      |       test_3.rule1       |
+------------+----------------------------------------------------+--------------------------+
| a          | 20230809,20230811,20230812,20230812,20230813       | 501,502,503,501,512      |
| b          | 20230809,20230811,20230812,20230812,20230813,20230809 | 511,512,513,511,512,511  |
| c          | 20230811,20230812,20230812,20230813                | 512,513,511,512          |
+------------+----------------------------------------------------+--------------------------+

> select t.id,a.cur_day
> from test_3 t
> lateral view explode(split(t.dt,',')) a as cur_day
> ;

--结果:
+-------+------------+
| t.id  | a.cur_day  |
+-------+------------+
| a     | 20230809   |
| a     | 20230811   |
| a     | 20230812   |
| a     | 20230812   |
| a     | 20230813   |
| b     | 20230809   |
| b     | 20230811   |
| b     | 20230812   |
| b     | 20230812   |
| b     | 20230813   |
| b     | 20230809   |
| c     | 20230811   |
| c     | 20230812   |
| c     | 20230812   |
| c     | 20230813   |
+-------+------------+

多列转多行

举例1:使用posexplode()+lateral view

先测试使用explode(), 看看效果:

> select  a.id,b.cur_day,c.rule 
> from test_2 a
> lateral view explode(dt_array) b as cur_day,
> lateral view explode(rule_array) c as rule
> ;

--结果:
+-------+------------+---------+
| a.id  | b.cur_day  | c.rule  |
+-------+------------+---------+
| b     | 20230809   | 511     |
| b     | 20230809   | 512     |
| b     | 20230809   | 513     |
| b     | 20230809   | 511     |
| b     | 20230809   | 512     |
| b     | 20230809   | 511     |
| b     | 20230811   | 511     |
| b     | 20230811   | 512     |
| b     | 20230811   | 513     |
| b     | 20230811   | 511     |
| b     | 20230811   | 512     |
| b     | 20230811   | 511     |
| b     | 20230812   | 511     |
| b     | 20230812   | 512     |
| b     | 20230812   | 513     |
| b     | 20230812   | 511     |
| b     | 20230812   | 512     |
| b     | 20230812   | 511     |
| b     | 20230812   | 511     |
| b     | 20230812   | 512     |
| b     | 20230812   | 513     |
| b     | 20230812   | 511     |
| b     | 20230812   | 512     |
| b     | 20230812   | 511     |
| b     | 20230813   | 511     |
| b     | 20230813   | 512     |
| b     | 20230813   | 513     |
| b     | 20230813   | 511     |
| b     | 20230813   | 512     |
| b     | 20230813   | 511     |
| b     | 20230809   | 511     |
| b     | 20230809   | 512     |
| b     | 20230809   | 513     |
| b     | 20230809   | 511     |
| b     | 20230809   | 512     |
| b     | 20230809   | 511     |
| c     | 20230811   | 512     |
| c     | 20230811   | 513     |
| c     | 20230811   | 511     |
| c     | 20230811   | 512     |
| c     | 20230812   | 512     |
| c     | 20230812   | 513     |
| c     | 20230812   | 511     |
| c     | 20230812   | 512     |
| c     | 20230812   | 512     |
| c     | 20230812   | 513     |
| c     | 20230812   | 511     |
| c     | 20230812   | 512     |
| c     | 20230813   | 512     |
| c     | 20230813   | 513     |
| c     | 20230813   | 511     |
| c     | 20230813   | 512     |
| a     | 20230809   | 501     |
| a     | 20230809   | 502     |
| a     | 20230809   | 503     |
| a     | 20230809   | 501     |
| a     | 20230809   | 512     |
| a     | 20230811   | 501     |
| a     | 20230811   | 502     |
| a     | 20230811   | 503     |
| a     | 20230811   | 501     |
| a     | 20230811   | 512     |
| a     | 20230812   | 501     |
| a     | 20230812   | 502     |
| a     | 20230812   | 503     |
| a     | 20230812   | 501     |
| a     | 20230812   | 512     |
| a     | 20230812   | 501     |
| a     | 20230812   | 502     |
| a     | 20230812   | 503     |
| a     | 20230812   | 501     |
| a     | 20230812   | 512     |
| a     | 20230813   | 501     |
| a     | 20230813   | 502     |
| a     | 20230813   | 503     |
| a     | 20230813   | 501     |
| a     | 20230813   | 512     |
+-------+------------+---------+

出现这种情况,是因为两个并列的explode()的hql没办法识别cur_day对应的rule是什么,对于多个数组的行转列可以使用posexplode()函数。
例如使用如下查询语句:

> select a.id,b.cur_day,c.rule 
> from test_2 a
> lateral view posexplode(dt_array) b as idxb,cur_day,
> lateral view posexplode(rule_array) c as idxc,rule where b.idxb=c.idxc
> ;

--结果:
+-------+------------+---------+
| a.id  | b.cur_day  | c.rule  |
+-------+------------+---------+
| b     | 20230809   | 511     |
| b     | 20230811   | 512     |
| b     | 20230812   | 513     |
| b     | 20230812   | 511     |
| b     | 20230813   | 512     |
| b     | 20230809   | 511     |
| c     | 20230811   | 512     |
| c     | 20230812   | 513     |
| c     | 20230812   | 511     |
| c     | 20230813   | 512     |
| a     | 20230809   | 501     |
| a     | 20230811   | 502     |
| a     | 20230812   | 503     |
| a     | 20230812   | 501     |
| a     | 20230813   | 512     |
+-------+------------+---------+
更多推荐

AI 编码助手 Codewhisperer 安装步骤和使用初体验

文章作者:为了自己加油最近亚⻢逊云科技推出了一款基于机器学习的AI编程助手AmazonCodeWhisperer,可以实时提供代码建议。在编写代码时,它会自动根据现有的代码和注释给出建议。AmazonCodeWhisperer与GitHubCopilot类似,主要的功能有:代码补全注释和文档补全代码安全问题的辅助定位亚

CSS 字体:Font

文章目录CSS字体serif和sans-serif字体之间的区别CSS字型字体系列字体样式字体大小设置字体大小像素用em来设置字体大小使用百分比和EM组合CSS字体属性CSS字体CSS字体属性可以定义文本的字体系列、大小、加粗、风格(如斜体)和变形(如小型大写字母)。具体来说,CSS字体属性包括以下这些:font-fa

CSS 浮动布局

浮动的设计初衷float:left/right/both;浮动是网页布局最古老的方式。浮动一开始并不是为了网页布局而设计,它的初衷是将一个元素拉到一侧,这样文档流就能够包围它。常见的用途是文本环绕图片:浮动元素会被移出正常文档流,并被拉到容器边缘。清除浮动的原因及方法浮动元素的高度不会追加到父元素上。如果浮动的元素比容

CSS 链接:Link

文章目录CSS链接链接样式常见的链接样式文本修饰背景颜色案例1,添加不同样式的超链接2,高级-创建链接框CSS链接CSS可以用来设置链接的样式,包括未访问的链接(a:link)、已访问的链接(a:visited)、鼠标悬停在链接上时(a:hover)和链接被点击时(a:active)这四种状态。以下是一个例子:/*未被

全球汽车安全气囊芯片总体规模分析

安全气囊系统是一种被动安全性的保护系统,它与座椅安全带配合使用,可以为乘员提供有效的防撞保护。在汽车相撞时,汽车安全气囊可使头部受伤率减少25%,面部受伤率减少80%左右。汽车安全气囊芯片是整个系统的控制核心,并将所有外围系统的功能集于一身:数字碰撞传感器接口、展开气囊的点火回路驱动、大量的安全和诊断机制以供持续监测系

使用VSCode SSH实现公网远程连接本地服务器开发的详细教程

文章目录前言1、安装OpenSSH2、vscode配置ssh3.局域网测试连接远程服务器4.公网远程连接4.1ubuntu安装cpolar内网穿透4.2创建隧道映射4.3测试公网远程连接5.配置固定TCP端口地址5.1保留一个固定TCP端口地址5.2配置固定TCP端口地址5.3测试固定公网地址远程前言远程连接服务器工具

汽车OTA

汽车OTA(Over-The-Air)技术是指通过无线网络对汽车进行软件升级、数据传输和远程诊断等功能的技术。随着汽车行业的数字化和智能化发展,OTA技术在汽车领域的应用越来越广泛,对于提高汽车性能、降低维修成本和提升用户体验具有重要意义。一、汽车OTA技术的主要功能软件升级:通过OTA技术,汽车制造商可以为汽车提供实

Go 异常处理

代码在执行的过程中可能因为一些逻辑上的问题而出现错误functest1(a,bint)int{result:=a/breturnresult}funcmain(){resut:=test1(10,0)fmt.Println(resut)}panic:runtimeerror:integerdividebyzerogor

汽车红外夜视系统行业发展总体概况

汽车红外夜视系统是一种技术,旨在帮助驾驶员在夜间或低光条件下提供更好的视觉能力。它利用红外光谱的特性来检测和显示在正常光线下难以察觉的热能辐射。这使驾驶员能够在夜间或恶劣天气条件下更好地识别和辨别道路上的物体、行人、动物或其他车辆。汽车红外夜视系统通常包括以下主要组件:红外摄像机:这是系统的核心部件,它使用红外传感器来

简单聊聊G1垃圾回收算法整个流程 --- 理论篇 -- 下

简单聊聊G1垃圾回收算法整个流程---理论篇--下软实时性预测转移时间预测可信度GC暂停处理的调度并发标记中的暂停处理分代G1GC模式不同点新生代区域分代对象转移具体转移流程分代选择回收集合设置最大新生代区域数GC的切换GC执行的时机总结上一篇文章我们简单看了一下G1整个垃圾回收流程,但是关于G1如何计算区域回收价值和

ActiveMQ面试题(一)

文章目录前言一、什么是ActiveMQ二、ActiveMQ服务器宕机怎么办?三、丢消息怎么办四、持节化消息非常慢五、消息的不均匀消费总结前言什么是ActiveMQActiveMQ服务器宕机怎么办?丢消息怎么办持节化消息非常慢消息的不均匀消费一、什么是ActiveMQactiveMQ是一种开源的,实现了JMS1.1规范的

热文推荐