Linux 命令行工具之 jq 最佳实践-六虎

jq 介绍

jq 是 stedolan 开发的一个轻量级的和灵活的命令行 JSON 处理器。

它首要用于在命令行界面处理 JSON 输入，并运用给定的过滤条件来过滤符合条件的新的 JSON串。

通常在类 Unix 环境下，咱们能够快速的运用 jq 来进行 JSON 数据格局化过滤和处理。

一起需求留意的是，该命令行东西和 awk/sed/grep 东西一样，归于体系的默许命令，假如体系没有该命令，能够测验运用如下方法进行安装。

# Ubuntu 系列
$ sudo apt-get install jq 
# CentOS 系列
$ yum install jq

jq 根底运用

1. 根本语法


jq [options] <jq filter> [file...]
jq [options] --args <jq filter> [strings...]
jq [options] --jsonargs <jq filter> [JSON_TEXTS...]
# options 可选参数列表和阐明
-c               将格局化json输出为紧凑的字符串格局;
-n               运用`null`作为单个输入值;
-e               根据输出设置退出状态代码;
-s               将一切输入读取（吸取）到数组中；运用过滤器;
-r               输出原始字符串，而不是JSON文本;
-R               读取原始字符串，而不是JSON文本;
-C               为JSON输出填充色彩;
-M               单色（不要为JSON着色）;
-S               在输出上排序目标的键;
--tab            运用制表符进行缩进;
--arg a v        将变量$a设置为value<v>;
--argjson a v    将变量$a设置为JSON value<v>;
--slurpfile a f  将变量$a设置为从<f>读取的JSON文本数组;
--rawfile a f    将变量$a设置为包括<f>内容的字符串;
--args           其他参数是字符串参数，而不是文件;
--jsonargs       其他的参数是JSON参数，而不是文件;
--               停止参数处理;

当然了，咱们也能够运用 man jq 或许 jq --help 再或许官网文档 jq-doc 中查看更多运用指南。

2. 根底运用

2.1 根底字段解析

## 运用 . 参数默许格局化整个json数据
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/"}'  | jq .
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "WebSite": "https://bgbiao.top/"
}
## 运用.$name 来获取指定filed
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/"}'  | jq .Name
"CloudNativeOps"
## 解析json 中的层级数据
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact
{
  "Email": "weichuangxxb@qq.com",
  "QQ": "371990778",
  "WeChat": "GoOps"
}
### 仅输出 Contact 中的 WeChat 
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact.WeChat
"GoOps"
## 获取多个字段 (运用'.filed1,.filed2' 能够获取两个字段)
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq ".Name,.Owner"
"CloudNativeOps"
"GoOps"

2.2 列表、迭代器、管道

## 解析json 中的数组或许列表
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills
[
  {
    "name": "Python",
    "type": "dev"
  },
  {
    "name": "Golang",
    "type": "dev"
  },
  {
    "name": "Ansible",
    "type": "ops"
  },
  {
    "name": "Kubernetes",
    "type": "dev"
  },
  {
    "name": "ElasticSearch",
    "type": "ops"
  }
]
### 一起也支撑范围索引，比方 .Skills[1:3]
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills[1:3]
[
  {
    "name": "Golang",
    "type": "dev"
  },
  {
    "name": "Ansible",
    "type": "ops"
  }
]
## value 迭代器 .[]
### 前面咱们在运用数组的时分经过.Skills 直接获取到了一个数组，而经过.Skills[n:m] 来经过指定索引范围获取一个子列表
### 而经过.[] 则能够直接迭代指定filed 里的values，这儿不仅仅是列表了，比方能够经过 .Contact[] 将联系人的全部方法获取出来
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Contact[]
"weichuangxxb@qq.com"
"371990778"
"GoOps"
### 而假如是列表或许数组，则能够指定元素 (支撑倒序获取，即.[-1]表明获取列表中的最后一个元素)
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq .Skills[0]
{
  "name": "Python",
  "type": "dev"
}
## 管道 (在jq 的表达式中，能够运用 管道符号 | 来对前面的表达式成果进行再次处理)
### 比方上面的列子中，咱们运用.Skills[] 获取了子json，但是假如仅想获取子json中的某一个filed，就能够运用管道的才能
### 所以，一般管道的更多会和列表类型的迭代配合运用
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '.Skills[] | .name '
"Python"
"Golang"
"Ansible"
"Kubernetes"
"ElasticSearch"
## 值核算 
echo '{"num":12}' | jq '(.num +2)*1024'
14336

留意1: 在上面的示例中，Skills 字段是一个列表，咱们需求留意， .Skills、.Skills[]、.Skills[N]以及.Skills[N:M] 的差异

.Skills: 一般的获取 Skills 字段的 values，由所以列表，回来为一个列表，为通用方法
.Skills[]: 迭代Skills 字段里的元素，由于Skills 为元素为子json的列表，因此回来全部的子json，为通用方法
.Skills[N]: 获取列表中索引为N的元素，仅用于在字段为列表类型时
.Skills[N:M]: 截取列表中索引从N到M元素为一个新的列表，仅用于字段为列表类型时

留意2: 在上述中对列表类型的值进行迭代后经过管道取值.Skills[] | .name 和 .Skills[].name 有着相同的效果

2.3 复杂数据类型构建

列表(数组)构建[]: 能够将输出字符回来成一个列表(能够结合多字段运用)
目标构建{}: 能够联合 () 构建新的json目标
- {} 能够经过字段构建新的json结构，比方 {user, title: .titles[]} 表明将titles数组中的元素迭代出来后和user 字段从头组成多个json字串
- () 能够用于直接将value 作为 key，比方{(.user): .titles} 能够直接构建无声明key 的json串
递归下降..: 能够递归下降，将字符串的每一个value都迭代出来，用法和内置的 recurse 函数类似

## 数组构建
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '[.Skills[].name ]'
[  "Python",  "Golang",  "Ansible",  "Kubernetes",  "ElasticSearch"]
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'  | jq '[.Skills[].name,.Name]'
[  "Python",  "Golang",  "Ansible",  "Kubernetes",  "ElasticSearch",  "CloudNativeOps"]
$ echo '{"num": [1,2,3,4]}' | jq '[.num[] | . * 2 ]'
[  2,  4,  6,  8]
## 目标构建
### jq '{Name,Owner,skills: .Skills[].name}' 构建新的json串 
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}' | jq '{Name,Owner,skills: .Skills[].name}'
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Python"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Golang"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Ansible"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "Kubernetes"
}
{
  "Name": "CloudNativeOps",
  "Owner": "GoOps",
  "skills": "ElasticSearch"
}
###  jq '{(.Name) : .Skills[].name}' 构建无声明key 的json串
$ echo '{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}' | jq '{(.Name) : .Skills[].name}'
{
  "CloudNativeOps": "Python"
}
{
  "CloudNativeOps": "Golang"
}
{
  "CloudNativeOps": "Ansible"
}
{
  "CloudNativeOps": "Kubernetes"
}
{
  "CloudNativeOps": "ElasticSearch"
}
## 递归下降 运用 .. 能够将全部的子串解析出来，直到最后的叶子value
$ echo '[[{"a":1}]]' | jq '.. '
[  [    {      "a": 1    }  ]
]
[  {    "a": 1  }]
{
  "a": 1
}
1
### 递归下降获取详细的值 '.. | .filed'
#### 获取key 包括a的值
$ echo '[[{"a":1}]]' | jq '.. | .a?'
1

2.4 内置的操作符以及函数

+: 两个过滤器相加，支撑 Numbers、Arrays、Strings、Objects 类型
-: 相减，用法同上
*,/,%: 乘除余运算
length: 获取不同类型值的长度，支撑string,array,object,null(zero)
utf8bytelength: utf8 的字节长度
keys,keys_unsorted: 获取最外层的key以及排序之后的key，假如是列表或许数组，将回来索引
has(key): 回来json 中是否包括key，或许数组/列表中是否包括索引下标【需求留意的是，假如需求解析解析数组/列表内部的子串是否包括某个key，需求先运用map 函数】
in(object): 判别json中是否包括object中给定的key
max(x),map_values(x): 运用过滤器x对输入数组的元素进行挨个核算，并回来新数组；后者是给值进行核算
path(path_expression): 输出给定途径表达式中的数组表明。
del(path_expression): 删去途径表达式中的filed
getpath(PATHS): 获取指定途径的索引
setpath(PATHS; VALUE): 给指定途径设置新的值
delpaths(PATHS): 删去指定途径
to_entries, from_entries, with_entries: 二次处理json实体，比方从[{key:k1,value: v1},{key:k2,value:v2} ]转换成 {k1:v1,k2:v2}
select(boolean_expression): 运用bool 表达式进行条件过滤
arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars: 分别只选择数组、目标、可迭代目标(数组或目标)、布尔值、数字、一般数字、有限数字、字符串、空值、非空值和不行迭代目标的输入。
add: 过滤器add承受一个数组作为输入，并将数组的元素加在一起作为输出。这可能意味着根据输入数组元素的类型进行求和、衔接或兼并——规则与上面描述的+运算符的规则相同。
any,all: 从数组或许列表中判别是否存在或许全部存在
range:目标生成器
floor: 输出数字的低阶值
sqrt: 求开方
tonumber: 字符串转数字
tostring: 数字转字符串
type: 获取元素类型
sort,sort_by(path_expression): 排序
unique, unique_by(path_exp): 去重
reverse: 反转
contains(element): 判别是否包括
startswith(str): 判别前缀
endswith(str): 判别后缀
split(str): 字符串转列表
join(str): 列表转字符串
while(cond; update): 条件判别
until(cond; next): 条件判别
\(foo): 引用foo的值
tojson,fromjson: 从原始字符串转到json或许从json转到原始字符串
@base64,@base64d: base64 编码和解码
@uri,@csv, 生成uri，以及表格格局

## 模版数据
$ testJson='{"Name":"CloudNativeOps","Owner":"GoOps","WebSite":"https://bgbiao.top/", "Contact": {"Email":"weichuangxxb@qq.com","QQ":"371990778","WeChat":"GoOps"} ,"Skills": [ {"name":"Python","type":"dev" }, {"name":"Golang","type":"dev" },{"name":"Ansible","type":"ops" },{"name":"Kubernetes","type":"dev" },{"name":"ElasticSearch","type":"ops" }]}'
### keys
$ echo ${testJson} | jq 'keys'
[
  "Contact",
  "Name",
  "Owner",
  "Skills",
  "WebSite"
]
$ echo ${testJson} | jq 'keys_unsorted'
[
  "Name",
  "Owner",
  "WebSite",
  "Contact",
  "Skills"
]
### has(key)
$ echo ${testJson} | jq '.Skills | map(has("name"))'
[
  true,
  true,
  true,
  true,
  true
]
$ echo ${testJson} | jq 'has("Name") '
true
### to_entries, from_entries, with_entries
$  echo ${testJson} | jq '.Contact | to_entries'
[
  {
    "key": "Email",
    "value": "weichuangxxb@qq.com"
  },
  {
    "key": "QQ",
    "value": "371990778"
  },
  {
    "key": "WeChat",
    "value": "GoOps"
  }
]
### select(bool_exp)
### 输出技术中包括 Ansible 的技术项
$ echo ${testJson} | jq '.Skills[] | select(.name == "Ansible")'
{
  "name": "Ansible",
  "type": "ops"
}
### tonumber
$  echo ${testJson} |jq '.Contact.QQ | tonumber '
371990778
$ echo ${testJson} |jq '.Contact.QQ '
"371990778"
### tostring/fromjson
$ echo ${testJson} |jq '.Contact | tostring'
"{\"Email\":\"weichuangxxb@qq.com\",\"QQ\":\"371990778\",\"WeChat\":\"GoOps\"}"
$ echo ${testJson} |jq '.Contact | tostring | fromjson'
{
  "Email": "weichuangxxb@qq.com",
  "QQ": "371990778",
  "WeChat": "GoOps"
}
### @base64/@base64d
$ echo ${testJson} |jq '.Contact | .QQ | @base64 | @base64d'
"371990778"
### @uri
$ echo ${testJson} | jq '.Skills[] | select(.name == "Ansible") | @uri "https://www.google.com/search?q=\(.name)"'
"https://www.google.com/search?q=Ansible"

2.5 示例

# 经过接口回来数据进行过滤查找
curl -s  http://goops.top:8080/vpc/api | jq '.returnData.detail[] | select(.ipType == 41)'

Linux 命令行工具之 jq 最佳实践

jq 介绍

jq 根底运用

1. 根本语法

2. 根底运用

相关文章

如何评估机器学习模型？

JetBrains Fleet安装及使用教程

iOS ｜关于触摸事件的传递机制以及响应流程

python自动化神器专题3：slack自动化， 最好用的桌面自动化开发工具

作者信息

python自动化神器专题3：slack自动化，最好用的桌面自动化开发工具