Chapter1 starts this book by presenting a simple HTTP server. To build a working HTTP server, you need to know the internal workings of two classes in the java.net package: Socket and ServerSocket. There is sufficient background information in this chapter about these two classes for you to understand how the accompanying application works.
第一个练习的目标，创建一个简单的 web server. 服务启动后，浏览器输入地址，server 返回请求的静态资源.
逻辑层面上来说模型可以像下面这样展示，但是代码层面上却不行。
按照上面的图示，难道 client 是直接 new 一个 request 和 web server 进行交互吗？难道 web server 会 new 一个 response 发送给 client 吗? 非也。模型画成下面的样子应该更合适
Client 和 Web Server 之间通过 socket 进行交互。在 server 内部，会将 socket 分化为 input 和 output 两个 IO 流，分别对应读取 Client 发送的数据和发送给 Client 相应
publicvoidparse(){ // Read a set of characters from the socket StringBuilder request = new StringBuilder(2048); int i; byte[] buffer = newbyte[2048]; i = input.read(buffer); for (int j = 0; j < i; j++) { request.append((char) buffer[j]); } // 打印请求信息 System.out.print(request); uri = parseUri(request.toString()); }
cat countries USSR 8649 275 Asia Canada 3852 25 North America China 3705 1032 Asia USA 3615 237 North America Brazil 286 134 South America India 1267 746 Asia Mexico 762 78 North America France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
sed -n 'l' countries USSR\t8649\t275\tAsia$ Canada\t3852\t25\tNorth America$ China\t3705\t1032\tAsia$ USA\t3615\t237\tNorth America$ Brazil\t286\t134\tSouth America$ India\t1267\t746\tAsia$ Mexico\t762\t78\tNorth America$ France\t211\t55\tEurope$ Japan\t144\t120\tAsia$ Germany\t96\t61\tEurope$ England\t94\t56\tEurope$
awk ' BEGIN { FS = "\t" printf("%10s %6s %5s %s\n\n", "COUNTRY", "AREA", "POP", "CONTINENT") } { printf("%10s %6s %5d %s\n", $1, $2, $3, $4) area = area + $2 pop = pop + $3 } END { printf("\n%10s %6d %5d\n", "TOTAL", area, pop) }' countries COUNTRY AREA POP CONTINENT
USSR 8649 275 Asia Canada 3852 25 North America China 3705 1032 Asia USA 3615 237 North America Brazil 286 134 South America India 1267 746 Asia Mexico 762 78 North America France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
awk '$0 >= "M"' countries USSR 8649 275 Asia USA 3615 237 North America Mexico 762 78 North America
String-matching Patterns
Awk 支持 regular expressions
String-Matching Pattern
/regexpr/: 目标是行内容的一部分
expression ~ /regexpr/: matches if the string value of expression contains a substring matched by regexpr
expression !~ /regexpr/: 和上面的相反
1 2 3 4 5 6 7 8 9 10 11 12 13 14
awk '$4 ~ /Asia/' countries USSR 8649 275 Asia China 3705 1032 Asia India 1267 746 Asia Japan 144 120 Asia
awk '$4 !~ /Asia/' countries Canada 3852 25 North America USA 3615 237 North America Brazil 286 134 South America Mexico 762 78 North America France 211 55 Europe Germany 96 61 Europe England 94 56 Europe
octal value ddd, where ddd is 1-3 digits between 0-7
\c
any other character c literally
Compound Patterns
混合模式，即多个表示式通过逻辑运算符组合
1 2 3 4 5 6 7 8 9 10 11 12
awk '$4 == "Asia" && $3 > 500' countries China 3705 1032 Asia India 1267 746 Asia
awk '$4 == "Asia" || $4 == "Europe"' countries USSR 8649 275 Asia China 3705 1032 Asia India 1267 746 Asia France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
上面的例子是字符比较，也可以使用正则
1 2 3 4 5 6 7 8
awk '$4 ~ /^(Asia|Europe)$/' countries USSR 8649 275 Asia China 3705 1032 Asia India 1267 746 Asia France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
如果其他 field 不包含这两个关键字，还可以用逻辑或筛选
1 2 3 4 5 6 7 8 9
# 等价于 awk '/Asia|Europe/' countries awk '/Asia/||/Europe/' countries USSR 8649 275 Asia China 3705 1032 Asia India 1267 746 Asia France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
优先级：! > && > ||
同优先级(|| + &&)的操作 从左到右 的顺序计算
Range Patterns
即两个 pattern 用逗号间隔, pat1， pat2 表示取匹配的 1 行或 n 行内容. 如下面的例子，选取 Canada 出现到 USA 之间的所有行
1 2 3 4
awk '/Canada/, /USA/' countries Canada 3852 25 North America China 3705 1032 Asia USA 3615 237 North America
如果后一个 pattern 没有匹配的内容，则匹配到末尾
1 2 3 4 5
awk '/Europe/, /Africa/' countries France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
FNR: the number of the line just read from current input file
FILENAME: the filename itself
打印第一行到第五行
1 2 3 4 5 6
awk 'FNR == 1, FNR == 5 { print FILENAME ":" $0 }' countries countries:USSR 8649 275 Asia countries:Canada 3852 25 North America countries:China 3705 1032 Asia countries:USA 3615 237 North America countries:Brazil 286 134 South America
同样的效果还可以写成
1 2 3 4 5 6
awk 'FNR <= 5 { print FILENAME ": " $0 }' countries countries: USSR 8649 275 Asia countries: Canada 3852 25 North America countries: China 3705 1032 Asia countries: USA 3615 237 North America countries: Brazil 286 134 South America
还没有初始化的时候 string 默认是 “” (the null string), numberic 默认是 0
Built-in Variables
下面是一些自带的变量，FILENAME 在每次读文件时都会自动赋值。FNR，NF 和 NR 在每次读入一行时重置。
VARIABLE
MEANING
DEFAULT
ARGC
number of command line arguments
-
ARGV
array of command line arguments
-
FILENAME
name of current input file
-
FNR
record number in current file
-
FS
controls the input field separator
“ “
NF
number of fields in current record
-
NR
number of records read so far
-
OFMT
output format for numbers
“%.6g”
OFS
output field separator
“ “
ORS
output record separator
“\n”
RLENGTH
length of string matched by match function
-
RS
controls the input recrod separator
“\n”
RSTART
start of string matched by match function
-
SUBSEP
subscript separator
“\034”
Field Variables
表示当前行的 field 参数，从 $1 - $NF, $0 表示整行。运行一些例子找找感觉
1 2 3 4 5 6 7
# 第二个 field 值缩小 1000 倍并打印 awk '{ $2 = $2 / 1000; print }' countries
USSR 8.649 275 Asia Canada 3.852 25 North America China 3.705 1032 Asia ...
将 North America 和 South America 替换为简写
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
awk 'BEGIN { FS = OFS = "\t" } $4 == "North America" { $4 = "NA" } $4 == "South America" { $4 = "SA" } {print} ' countries USSR 8649 275 Asia Canada 3852 25 NA China 3705 1032 Asia USA 3615 237 NA Brazil 286 134 SA India 1267 746 Asia Mexico 762 78 NA France 211 55 Europe Japan 144 120 Asia Germany 96 61 Europe England 94 56 Europe
PS: 这里之前我倒是没有意识到，上面的做法其实就是多种情况替换的案例了
还有一些比较神奇的使用方式，比如 $(NF - 1) 可以取得倒数第二个 field。如果 field 不存在，默认值为 null string, 比如 $(NF + 1), 一个新的 field 可以通过赋值得到，比如下面的例子是在原有的数据后面添加第五列元素
1 2 3 4
awk 'BEGIN { FS = OFS = "\t" }; { $5 = 1000 * $3 / $2; print }' countries USSR 8649 275 Asia 31.7956 Canada 3852 25 North America 6.49013 ...
awk '{ print ($1 !=0 ? 1/$1 : "$1 is zero, line " NR) }'
Assignment Operators
var = expr, 下面的例子计算所有亚洲国家的人口和
1 2 3
awk '$4 == "Asia" { pop = pop + $3; n = n + 1} END { print "Total population of the ", n, "Asian countries is", pop, "million"}' countries Total population of the 4 Asian countries is 2173 million
统计人口最多的国家
1 2 3
awk '$3 > maxpop {maxpop = $3; country = $1} END { print "country with largest population:", country, maxpop }' countries country with largest population: China 1032
Increment and Decrement Oerators
n = n + 1 通常简写为 ++n 或者 n++, 区别是，如果有赋值，则 n++ 会将原始值赋给变量再自增，++n 则先自增再赋值
1 2 3 4
awk 'BEGIN { n=1; i=n++ }; END { print i }' countries 1 awk 'BEGIN { n=1; i=++n }; END { print i }' countries 2
awk '{print "total pay for", $1, "is", $2 * $3}' emp.data total pay for Beth is 0 total pay for Dan is 0 total pay for Kathy is 40 total pay for Mark is 100 total pay for Mary is 121 total pay for Susie is 76.5
Fancier Output
print 只是简单的打印内容，如果想要输出更丰富，使用 printf
Lining Up Fields printf 格式 printf(format, value1, value2..., valuen)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
awk '{printf("total pay for %s is %.2f\n", $1, $2 * $3)}' emp.data total pay for Beth is 0.00 total pay for Dan is 0.00 total pay for Kathy is 40.00 total pay for Mark is 100.00 total pay for Mary is 121.00 total pay for Susie is 76.50
awk '{printf("%-8s $%6.2f\n", $1, $2 * $3)}' emp.data Beth $ 0.00 Dan $ 0.00 Kathy $ 40.00 Mark $100.00 Mary $121.00 Susie $ 76.50
awk ' $2 > 6 { n = n+1; pay = pay+$2*$3 } END { if (n > 0) print n, "employees, total pay is", pay, "average pay is", pay/n else print "no employees are paid more than $6/hour" }' emp.data no employees are paid more than $6/hour
While Statement
while = condition + body. 下面实现一个计算存款的功能，表达式可以概括为 value = amount (1 + rate)years
private String makeNum(){ Random random = new Random(); String num = random.nextInt(9999999) + ""; StringBuffer sb = new StringBuffer(); for (int i=0; i<7-num.length(); i++) { sb.append("0"); } num = sb.toString() + num; return num; }
// loop cookies and get login cookie if exist Cookie loginCookie = null; Cookie[] cookies = req.getCookies(); for (Cookie cookie : cookies) { if (cookie.getName().equals("lastlogin")) { loginCookie = cookie; } }
// if it's first time login, print log. else print last login time if (loginCookie == null) { resp.getWriter().print("it's the first time to login..."); } else { String strDateFormat = "yyyy-MM-dd HH:mm:ss"; SimpleDateFormat sdf = new SimpleDateFormat(strDateFormat); resp.getWriter().print("last login time: " + sdf.format(new Date(Long.parseLong(loginCookie.getValue())))); }
// 三个主体方法，init + destory + service, service 包含主要转化过程 publicvoid_jspInit(){ }
publicvoid_jspDestroy(){ }
publicvoid_jspService(final javax.servlet.http.HttpServletRequest request, final javax.servlet.http.HttpServletResponse response) throws java.io.IOException, javax.servlet.ServletException {
// request type 检测 if (!javax.servlet.DispatcherType.ERROR.equals(request.getDispatcherType())) { final java.lang.String _jspx_method = request.getMethod(); if ("OPTIONS".equals(_jspx_method)) { response.setHeader("Allow","GET, HEAD, POST, OPTIONS"); return; } if (!"GET".equals(_jspx_method) && !"POST".equals(_jspx_method) && !"HEAD".equals(_jspx_method)) { response.setHeader("Allow","GET, HEAD, POST, OPTIONS"); response.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED, "JSPs only permit GET, POST or HEAD. Jasper also permits OPTIONS"); return; } }
// 声明一些内置变量 final javax.servlet.jsp.PageContext pageContext; javax.servlet.http.HttpSession session = null; // servlet context 命名为 application final javax.servlet.ServletContext application; final javax.servlet.ServletConfig config; javax.servlet.jsp.JspWriter out = null; final java.lang.Object page = this; javax.servlet.jsp.JspWriter _jspx_out = null; javax.servlet.jsp.PageContext _jspx_page_context = null;
cat test1 #!/usr/local/bin/bash # using a function in a script
function func1 { echo"This is an example of a function" }
count=1 while [ $count -le 4 ] do func1 count=$[ $count + 1 ] done
echo"This is the end of the loop" func1 echo"Now this is the end of the script"
./test1 # This is an example of a function # This is an example of a function # This is an example of a function # This is an example of a function # This is the end of the loop # This is an example of a function # Now this is the end of the script
cat test12 #!/usr/local/bin/bash # returning an array value
function arraydblr { local origarray local newarray local elements local i origarray=($(echo"$@")) newarray=($(echo"$@")) elements=$[ $# - 1 ] for (( i=0; i<= $elements; i++ )) { newarray[$i]=$[ ${origarray[$i]} * 2 ] } echo${newarray[*]} }
myarray=(1 2 3 4 5) echo"The original array is: ${myarray[*]}" arg1=$(echo${myarray[*]}) result=($(arraydblr $arg1)) echo"The new array is: ${result[*]}"
./test12 # The original array is: 1 2 3 4 5 # The new array is: 2 4 6 8 10
Function Recursion
local function variable 提供 self-containment 功能。他使得函数可以实现 recursively 的效果
cat myfuncs #!/usr/local/bin/bash # my script functions
function addem { echo $[ $1 + $2 ] }
function multem { echo $[ $1 * $2 ] }
function divem { if [ $2 -ne 0 ] then echo $[ $1 / $2 ] else echo -1 fi }
cat test14 #!/usr/local/bin/bash # using functions defined in a library file . ./myfuncs
value1=10 value2=5 result1=$(addem $value1$value2) result2=$(multem $value1$value2) result3=$(divem $value1$value2) echo"The result of adding them is: $result1" echo"The result of multiplying them is: $result2" echo"The result of dividing them is: $result3"
./test14 # The result of adding them is: 15 # The result of multiplying them is: 50 # The result of dividing them is: 2
Using Functions on the Command Line
Creating functions on the command line
方式一：终端一行定义
1 2 3 4
function doubleit { read -p "Enter value:" value; echo $[ $value * 2 ]; } doubleit # Enter value:3 6
带进度条的显示 ls -al /usr/bin | shtool prop -p "waiting..." 太快了，看不出效果
Chapter 18: Writing Scripts for Graphical Desktops
和我这次看书的目标不符，跳过
Chapter 19: Introducing sed and gawk
实际工作中，很多工作都是文字处理相关的。使用 shell 自带工具处理文字会显得很笨拙。这时候就要用到 sed 和 gawk 了。
Manipulating Text
Mac 自带的 sed 工具和书上的是不一样的，好像做了很多裁剪，很多 flag 是不支持的，可以通过 brew 重新装一个
1 2 3 4 5 6 7 8 9
brew install gnu-sed # 会给出添加 PATH 的提示，按照提示添加到配置文件中(.zshrc) brew info gnu-sed # ==> Caveats # GNU "sed" has been installed as "gsed". # If you need to use it as "sed", you can add a "gnubin" directory # to your PATH from your bashrc like:
sed 是一个流处理编辑器(stream editor)，你可以设定一系列的规则，然后通过这个流编辑器处理他。
sed 可以做如下事情
Reads one data line at a time from the input
Matches that data with the supplied editor commands
Changes data in the stream as specified in the commands
Outputs the new data to STDOUT
按行依次处理文件直到所有内容处理完毕结束，格式 sed options script file
The sed Command Options
Option
Description
-e script
Adds commands specified in the script to the commands run while processing the input
-f file
Adds the commands specified in the file to the commands run while processing the input
-n
Doesn’t produce output for each command, but waits for the print command
Defining an editor command int the command line
1 2
echo"This is a test" | sed 's/test/big test/' # This is a big test
s 表示替换(substitutes), 他会用后一个字符串替换前一个. 下面是替换文件内容的例子. sed 只会在输出内容中做修改，原文件还是保持原样
1 2 3 4 5 6 7 8 9 10
cat data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. sed 's/dog/cat/' data1.txt # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat.
Using mulitple editor commands int the command line
1 2 3 4 5
sed -e 's/brown/green/; s/dog/cat/' data1.txt # The quick green fox jumps over the lazy cat. # The quick green fox jumps over the lazy cat. # The quick green fox jumps over the lazy cat. # The quick green fox jumps over the lazy cat.
如果不想写在一行，可以写在多行
1 2 3 4 5 6 7 8
sed -e ' > s/brown/green/ > s/fox/elephant/ > s/dog/cat/' data1.txt # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat.
Reading editor commands from a file
如果命令太多，也可以将他们放到文件中
1 2 3 4 5 6 7 8 9
cat script1.sed # s/brown/green/ # s/fox/elephant/ # s/dog/cat/ sed -f script1.sed data1.txt # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat. # The quick green elephant jumps over the lazy cat.
为了便于区分 shell 文件和 sed 条件，我们将 sed 条件文件存储为 .sed 结尾
Getting to know the gawk program
sed 让你动态改变文件内容，但是还是有局限性。gawk 提供一个更程序化的方式来处理文本信息. 这个工具包默认是没有的一般需要自己手动安装。gawk 是 GNU 版本的 awk, 通过它你可以
Define variables to store data
Use arithmetic and string operatiors to perate on data
Use structured programming concepts, such as if-then statements and loops, to add logic to your data processing
Generate formatted reports by extracting data elements within the data file and repositioning them in another order or format
第四点经常用来批量处理数据使之更具可读性，典型应用就是处理 log 文件。
Visiting the gawk command format
格式 gawk options program file
The gawk Options
Option
Description
-F fs
Specifies a file separator for delineating data fields in a line
-f file
Specifies a file name to read the program from
-v var=value
Defines a variable and default value used in the gawk program
-mf N
Specifies the maximum number of fields to process in the data file
-mr N
Specifies the maximum record size in the data file
-W keyword
Specifies the compatibility mode or warning level of gawk
echo"My name is Rich" | gawk '{$4="Christine";print $0}' # My name is Christine
多行表示也是 OK 的
1 2 3 4 5 6
gawk '{ > $4="Christine" > print $0 > }' my name is Rich my name is Christine
Reading the program from a file
和 sed 一样，gawk 也支持从文件读取命令
1 2 3 4 5 6
cat script2.gawk # {print $1 "'s hoe directory is " $6} gawk -F: -f script2.gawk /etc/passwd | tail -3 # _coreml's hoe directory is /var/empty # _trustd's hoe directory is /var/empty # _oahd's hoe directory is /var/empty
gawk 文件中包含多个命令的示例
1 2 3 4 5 6 7 8 9
cat script3.gawk # { # text = "'s home directory is " # print $1 text $6 # } gawk -F: -f script3.gawk /etc/passwd | tail -3 # _coreml's home directory is /var/empty # _trustd's home directory is /var/empty # _oahd's home directory is /var/empty
cat data3.txt # Line 1 # Line 2 # Line 3 gawk 'BEGIN {print "The data3 File Contents: "} > {print $0}' data3.txt The data3 File Contents: # Line 1 # Line 2 # Line 3
Running scripts after processing data
和前面对应的还有一个 after 操作
1 2 3 4 5 6 7 8
gawk 'BEGIN {print "The data3 File Contents:"} {print $0} > END {print "End of File"}' data3.txt # The data3 File Contents: # Line 1 # Line 2 # Line 3 # End of File
at script4.gawk BEGIN { print"The latest list of users and selles" print" UserID\t Shell" print"-------\t------" FS=":" }
{ print$1" \t "$7 }
END { print"This concludes the listing" }
gawk -f script4.gawk /etc/passwd # The latest list of users and selles # UserID Shell # ------- ------ # nobody /usr/bin/false # ... # _oahd /usr/bin/false # This concludes the listing
Commanding at the sed Editor Basics
本章简要介绍一下 sed 的常规用法
Introducing more substitution options
Substituting flags
1 2 3 4 5 6
cat data4.txt # This is a test of the test script. # This is the second test of the test script. sed 's/test/trial/' data4.txt # This is a trial of the test script. # This is the second trial of the test script.
A number, indicating the pattern occurrence for which new text should be substituted
g, indicating that new text should be substituted for all occrurences of the existing text
p, indicating that the contents of the original line should be printed
w file, which means to write the results of the substitution to a file
替换指定位置的示例，下面示例中只替换了第二个位置的 test
1 2 3
sed 's/test/trial/2' data4.txt # This is a test of the trial script. # This is the second test of the trial script.
全部替换示例
1 2 3
sed 's/test/trial/g' data4.txt This is a trial of the trial script. This is the second trial of the trial script.
打印符合匹配条件的行
1 2 3 4 5 6
cat data5.txt # This is a test line. # This is a different line.
sed -n 's/test/trial/p' data5.txt # This is a trial line.
-w 指定 sed 结果输出到文件
1 2 3 4 5
sed 's/test/trial/w test.txt' data5.txt # This is a trial line. # This is a different line. cat test.txt # This is a trial line.
Replacing characters
Linux 系统中路径符号和 sed 中的符号是重的，也就是说，如果我要用 sed 替换路径的时候就必须用一中很累赘的写法, 比如 sed 's/\/bin\/bash/\/bin\/csh/' /etc/passwd
PS: Mac OS 语法和这个不一样
为了避免这么恶心的写法，我们可以用惊叹号(exclamation point) 代替原来的分割符 sed 's!/bin/bash!/bin/csh!' /etc/passwd
Using addresses
默认情况下 sed 会处理所有的行，如果你只需要处理特殊的几行，你可以使用 line address. line address 有两种模式
A numberic range of lines
A text pattern that filters out a line
两种模式的格式都是一样的 [address] command 你可以将多个命令组合到一起
1 2 3 4 5
address { command1 command2 command3 }
Addressing the numberic line
sed 会将 s 之前的内容当作行来处理, 下面的例子只替换第二行的内容
1 2 3 4 5
sed '2s/dog/cat/' data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog.
替换多行
1 2 3 4 5
sed '2,3s/dog/cat/' data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy dog.
从第 n 行还是到结束
1 2 3 4 5
sed '2,$s/dog/cat/' data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat. # The quick brown fox jumps over the lazy cat.
sed '2{ s/fox/elephant/ s/dog/cat/ }' data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown elephant jumps over the lazy cat. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog.
sed '3,${ s/fox/elephant/ s/dog/cat/ }' data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. # The quick brown elephant jumps over the lazy cat. # The quick brown elephant jumps over the lazy cat.
Deleting lines
d 用来在输出内容中删除某一行, 如果没有指定删选内容，所有输出都会被删除
1 2 3 4 5 6 7
cat data6.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is line number 4. sed 'd' data6.txt
指定删除的行
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
sed '3d' data6.txt # This is line number 1. # This is line number 2. # This is line number 4.
sed '2,3d' data6.txt # This is line number 1. # This is line number 4.
sed '3,$d' data6.txt # This is line number 1. # This is line number 2.
sed '/number 1/d' data6.txt # This is line number 2. # This is line number 3. # This is line number 4.
cat data7.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is line number 4. # This is line number 1 again. # This is text you want to keep. # This is the last line in the file.
sed '/1/,/3/d' data7.txt # This is line number 4.
sed '/1/,/5/d' data7.txt
Inserting and appending text
sed 也允许你插入，续写内容，但是有一些特别的点
The insert command(i) adds a new line before the specified line
The append commadn(a) adds a new line after the specified line
特别的点在于，你需要新起一行写这些新加的行, 格式为
1 2
sed '[address]command\ new line'
示例如下, 不过 mac 上貌似有语法错误
1 2 3 4 5 6 7 8 9 10 11 12
echo"Test Line 2" | sed 'i\Test line 1' # Test line 1 # Test Line 2
echo"Test Line 2" | sed 'a\Test line 1' # Test Line 2 # Test line 1
echo"Test Line 2" | sed 'i\ > Test Line 1' # Test Line 1 # Test Line 2
上面演示的是在全部内容之前/后添加新的行，那么怎么在特定行前后做类似的操作呢，你可以用行号指定。但是不能用 range 形式的，因为定义上，i/a 是单行操作
1 2 3 4 5 6 7 8 9 10 11 12 13
sed '3i\ This is an inserted line.' data6.txt # This is line number1. # This is line number2. # This is an inserted line. # This is line number3. # This is line number4. sed '3a\This is an appended line.' data6.txt # This is line number1. # This is line number2. # This is line number3. # This is an appended line. # This is line number4.
插入文本末尾
1 2 3 4 5 6 7
sed '$a\ > This is a new line of text.' data6.txt # This is line number1. # This is line number2. # This is line number3. # This is line number4. # This is a new line of text.
头部插入多行, 需要使用斜杠分割
1 2 3 4 5 6 7 8 9
sed '1i\ > This is one line of new text.\ > This is another line of new text.' data6.txt # This is one line of new text. # This is another line of new text. # This is line number1. # This is line number2. # This is line number3. # This is line number4.
如果不指定行号，他会每一行都 insert 啊，和之前的理解不一样. append 也是一样的效果
1 2 3 4 5 6 7 8 9
sed 'i\head insert' data6.txt # head insert # This is line number1. # head insert # This is line number2. # head insert # This is line number3. # head insert # This is line number4
也能指定 range… 前面的理解果断有问题
1 2 3 4 5 6 7
sed '1,2a\end append' data6.txt # This is line number1. # end append # This is line number2. # end append # This is line number3. # This is line number4.
Changing lines
改变行内容，用法和前面的 i/a 没什么区别
1 2 3 4 5
sed '3c\This is a chagned line of text.' data6.txt # This is line number1. # This is line number2. # This is a chagned line of text. # This is line number4.
pattern 方式替换
1 2 3 4 5 6
sed '/number3/c\ This is a changed line of text.' data6.txt # This is line number1. # This is line number2. # This is a changed line of text. # This is line number4.
pattern 替换多行
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
cat data8.txt # This is line number1. # This is line number2. # This is line number3. # This is line number4. # This is line number1 again. # This is yet another line. # This is the last line in the line.
sed '/number1/c\This is a changed line of text.' data8.txt # This is a changed line of text. # This is line number2. # This is line number3. # This is line number4. # This is a changed line of text. # This is yet another line. # This is the last line in the line.
指定行号替换的行为方式有点奇怪，他会将你指定的行中内容全部替换掉
1 2 3 4 5 6 7 8 9 10
cat data6.txt # This is line number1. # This is line number2. # This is line number3. # This is line number4.
sed '2,3c\This is a new line of text.' data6.txt # This is line number1. # This is a new line of text. # This is line number4.
sed 'y/123/789/' data8.txt # This is line number7. # This is line number8. # This is line number9. # This is line number4. # This is line number7 again. # This is yet another line. # This is the last line in the line.
而且他是全局替换，任何出现的地方都会被换掉
1 2
echo"This 1 is a test of 1 try." | sed 'y/123/456/' # This 4 is a test of 4 try.
Printing revisited
和 p flag 类似的还有两个符号，表示如下
The p command to print a text line
The equal sign(=) command to print line numbers
The l(lowercase L) command to list a line
p 使用案例, -n 可以强制只打印匹配的内容
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
echo"this is a test" | sed 'p' # this is a test # this is a test
cat data6.txt # This is line number1. # This is line number2. # This is line number3. # This is line number4.
sed -n '/number3/p' data6.txt # This is line number3.
sed -n '2,3p' data6.txt # This is line number2. # This is line number3.
找到匹配的行，先打印原始值，再替换并打印。
1 2 3 4 5 6
sed -n '/3/{ > p > s/line/test/p > }' data6.txt # This is line number3. # This is test number3.
equals 相关的案例，输出行号
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat data1.txt # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. # The quick brown fox jumps over the lazy dog. sed '=' data1.txt # 1 # The quick brown fox jumps over the lazy dog. # 2 # The quick brown fox jumps over the lazy dog. # 3 # The quick brown fox jumps over the lazy dog. # 4 # The quick brown fox jumps over the lazy dog.
搜索匹配的内容并打印行号
1 2 3 4 5 6
sed -n '/number 4/{ > = > p > }' data6.txt # 4 # This is line number 4.
l - listing lines, 打印文字和特殊字符(non-printable characters). 下面的实验中，tab 符号答应失败了，可能什么设置问题把，不过结尾符 $ 倒是么什么问题
1 2 3 4
cat data9.txt # This line contains tabs. sed -n 'l' data9.txt # This line contains tabs.$
Using files with sed
Writing to a file
通过 w 将匹配的内容写到文件 [address]w filename, 使用 -n 只在屏幕上显示匹配部分
1 2 3 4 5 6 7 8
sed '1,2w test.txt' data6.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is line number 4. cat test.txt # This is line number 1. # This is line number 2.
这个技巧在筛选数据的时候格外好用
1 2 3 4 5 6 7 8 9 10
cat data11.txt # Blum, R Browncoat # McGuiness, A Alliance # Bresnahan, C Browncoat # Harken, C Alliance
sed -n '/Browncoat/w Browncoats.txt' data11.txt cat Browncoats.txt # Blum, R Browncoat # Bresnahan, C Browncoat
Reading data from a file
The read command(r) allows you to insert data contained in a separate file. format at: [address]r filename
filename 可以是相对路径，也可以是绝对路径。你不能使用 range of address for the read command. you can only specify a single line number or text pattern address.
读取目标文件中的内容并插入到指定位置的后面
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
cat data12.txt # This is an added line. # This is the second added line.
cat data6.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is line number 4.
sed '3r data12.txt' data6.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is an added line. # This is the second added line. # This is line number 4.
pattern 同样支持
1 2 3 4 5 6 7
sed '/number 2/r data12.txt' data6.txt # This is line number 1. # This is line number 2. # This is an added line. # This is the second added line. # This is line number 3. # This is line number 4.
添加到末尾
1 2 3 4 5 6 7
sed '$r data12.txt' data6.txt # This is line number 1. # This is line number 2. # This is line number 3. # This is line number 4. # This is an added line. # This is the second added line.
将 read 和 delete 结合使用，我们就可以有类似于替换的效果了
下面例子中我们将名单用 LIST 这个单词做为占位符，将 data11.txt 中的内容替换进去
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cat notice.std # Would the following people: # LIST # please report to the ship's captain.
sed '/LIST/{ > r data11.txt > d > }' notice.std # Would the following people: # Blum, R Browncoat # McGuiness, A Alliance # Bresnahan, C Browncoat # Harken, C Alliance # please report to the ship's captain.
Chapter 20: Regular Expressions
What Are Regular Expressions
A definition
A regular expression is a pattern template you define that a Linux utility users to filter text.
Types of regular expressions
Linux 系统中，一些不同的应用采用不同的正则表达式。正则表达式通过 regular expression engine 实现。Linux 世界中有两个爆款 engines:
The POSIX Basic Regular Expression(BRE) engine
The POSIX Extened Regular Expression(ERE) engine
大多数 Linux 工具都会适配 RRE，sed 除外，它的目标是尽可能快的处理，所以只识别部分 BRE。
echo'The cost is $4.00' | sed -n '/\$/p' # The cost is $4.00
Anchor characters
Starting at the beginning
caret character(^) 指代文本开头
1 2 3 4
echo"The book store" | sed -n '/^book/p' # no matched echo"Books are great" | sed -n '/^Book/p' # Books are great
当符号出现在非开头位置，则他会被当作普通字符处理
1 2
echo"This ^ is a test" | sed -n '/s ^/p' # This ^ is a test
Looking for the ending
The dollar sign($) 指代了文本的结尾
1 2
echo"This is a good book" | sed -n '/book$/p' # This is a good book
The dot character
dot 用于指代除换行外的任何字符, 如果 . 代表的位置没有东西，则匹配失败
1 2 3 4 5 6 7 8 9 10
cat data6 # This is a test of a line. # The cat is sleeping. # That is a very nice hat. # This test is at line four. # at ten o'clock we'll go home. sed -n '/.at/p' data6 # The cat is sleeping. # That is a very nice hat. # This test is at line four.
Character classes
character class 用于限定匹配的内容，使用 square brackets([]) 表示
1 2 3
sed -n '/[ch]at/p' data6 # The cat is sleeping. # That is a very nice hat.
Negating character classes
和前面的相反，是不包含的意思
1 2
sed -n '/[^ch]at/p' data6 # This test is at line four.
Using ranges
sed -n '/^[0-9][0-9][0-9][0-9][0-9]$/p' data8 这个技巧也使用于字符
1 2 3
sed -n '/[c-h]at/p' data6 # The cat is sleeping. # That is a very nice hat.
也可以指定非连续的字符集合
1 2 3 4 5
sed -n '/[a-ch-m]at/p' data6 # The cat is sleeping. # That is a very nice hat. echo"I'm getting too fat" | sed -n '/[a-ch-m]at/p' # no matched
Special character classes
BRE Special Character classes
Class
Description
[[:alpha:]]
Matches any alphabetical character, either upper or lower case
[[:alnum:]]
Matches any alphanumberic character 0-9, A-Z or a-z
[[:blank:]]
Matches a space or Tab character
[[:digit:]]
Matches a numberical digit from 0-9
[[:lower:]]
Matches any lowercase alphabetical character a-z
[[:print:]]
Matches any printable character
[[:punct:]]
Matches a punctuation character
[[:space:]]
Matches any whitespace character: space, Table, NL, FF, VT CR
[[:upper:]]
Matches any uppercase alphabetical character A-Z
1 2 3 4 5 6 7 8 9 10
echo"abc" | sed -n '/[[:digit:]]/p' # no matched echo"abc" | sed -n '/[[:alpha:]]/p' # abc echo"abc123" | sed -n '/[[:digit:]]/p' # abc123 echo"This is, a test" | sed -n '/[[:punct:]]/p' # This is, a test echo"This is a test" | sed -n '/[[:punct:]]/p' # no matched
The asterisk
星号标识字符出现一次或 n 次
1 2 3 4 5 6
echo"ik" | sed -n '/ie*k/p' # ik echo"iek" | sed -n '/ie*k/p' # iek echo"ieeeek" | sed -n '/ie*k/p' # ieeeek
星号也可以和 character class 结合使用
1 2
echo"baeeeet" | sed -n '/b[ae]*t/p' # baeeeet
Extended Regular Expressions
gawk 识别 ERE pattern，ERE 新增了一些符号来扩展功能
Caution sed 和 gawk 采用了不同的引擎，gawk 可以适配大部分的扩展功能。sed 不能，但是 sed 更快
pipe symbol 可以让你实现 OR 的逻辑，只要有一个匹配，就算 match 了 expr1 | expr2 | ...
1 2 3 4 5 6
echo"The cat is asleep" | gawk '/cat|dog/{print $0}' # The cat is asleep echo"The dog is asleep" | gawk '/cat|dog/{print $0}' # The dog is asleep echo"The sheep is asleep" | gawk '/cat|dog/{print $0}' # no matched
结合 character class 使用
1 2
echo"He has a hat" | gawk '/[ch]at|dog/{print $0}' # He has a hat
Grouping expressions
可以使用括号(parentheses)表示 group. 这个 group 会被当成一个基本字符对待。
The single-line next command moves the next line of text from the data stream into the processing space(called the pattern space) of the sed editor.
The multiline version of the next command(which uses a captial N) adds the next line of text to the text already in the pattern space.
大写的 N 可以将两行拼成一行处理，中间用换行符隔开
1 2 3 4 5 6 7 8 9 10
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line # This is the last line
sed '/first/{N; s/\n/ /}' data2.txt # This is the header line. # This is the first data line. This is the second data line # This is the last line
上面的例子中，我们找到包含 first 的行，然后将下一行接上一起处理，处理的时候，将换行替换为空格
再举一个需要测试的数据落在两个段落中的例子
1 2 3 4 5
cat data3.txt # On Tuesday, the Linux System # Administrator's group meeting will be held. # All System Administrators should attend. # Thank you for your attendance.
第一个关键字替换失败，第二个成功，因为第一个用的换行，匹配用的空格
1 2 3 4 5
sed 'N;s/System Administrator/Desktop User/' data3.txt # On Tuesday, the Linux System # Administrator's group meeting will be held. # All Desktop Users should attend. # Thank you for your attendance
替换成功不过换行消失了
1 2 3 4
sed 'N;s/System.Administrator/Desktop User/' data3.txt # On Tuesday, the Linux Desktop User's group meeting will be held. # All Desktop Users should attend. # Thank you for your attendance.
使用两个替换分别应对换行和空格的情况
1 2 3 4 5 6 7 8
sed 'N > s/System\nAdministrator/Desktop\nUser/ > s/System Administrator/Desktop User/ > ' data3.txt # On Tuesday, the Linux Desktop # User's group meeting will be held. # All Desktop Users should attend. # Thank you for your attendance.
这里还有一个小问题，由于命令是 N 开头，他会先拿下一行到 pattern space，当处理最后一行时，下一行为空，直接结束了，如果要替换的目标在最后一行就会有遗漏
1 2 3 4 5 6 7 8 9 10 11 12
cat data4.txt # On Tuesday, the Linux System # Administrator's group meeting will be held. # All System Administrators should attend.
sed 'N s/System\nAdministrator/Desktop\nUser/ s/System Administrator/Desktop User/ ' data4.txt # On Tuesday, the Linux Desktop # User's group meeting will be held. # All System Administrators should attend.
这时你可以换一下顺序
1 2 3 4 5 6 7 8
sed ' > s/System Administrator/Desktop User/ > N > s/System\nAdministrator/Desktop\nUser/ > ' data4.txt # On Tuesday, the Linux Desktop # User's group meeting will be held. # All Desktop Users should attend.
（；￣ェ￣）简直无情，太繁琐了
Navigating the multiline delete command
当使用 N 的方式做 delete 的时候，它会将匹配到的两行内容全部删掉
1 2 3 4 5 6 7
cat data4.txt # On Tuesday, the Linux System # Administrator's group meeting will be held. # All System Administrators should attend.
sed 'N ; /System\nAdministrator/d' data4.txt # All System Administrators should attend.
sed 提供了一个只删除第一行内容的 flag - D
1 2 3
sed 'N ; /System\nAdministrator/D' data4.txt # Administrator's group meeting will be held. # All System Administrators should attend.
类似的技巧可以用来删除文章开头的空行
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat -n data1.txt # 1 # 2 This is the header line. # 3 # 4 This is a data line. # 5 # 6 This is the last line
sed '/^$/{N;/header/D}' data1.txt | cat -n # 1 This is the header line. # 2 # 3 This is a data line. # 4 # 5 This is the last line
Navigating the multiline print command
和 p 对应的还有一个 P， 用法和上面的 D 一样，如果用 p 会打印两行，而用 P 则只打印第一行
1 2 3 4 5
sed -n 'N ; /System\nAdministrator/P' data3.txt # On Tuesday, the Linux System sed -n 'N ; /System\nAdministrator/p' data3.txt # On Tuesday, the Linux System # Administrator's group meeting will be held.
Holding Space
pattern space 是 sed 用于存放正在的处理文本的空间。但是这并不是存放文本的唯一的地方，还有一个叫做 hold space. 下列是五个可以操作 hold space 的命令
The sed Editor Hold Space Commands
Command
Description
h
Copies pattern space to hold space
H
Appends pattern space to hold space
g
Copies hold space to pattern sapce
G
Appends hold space to pattern space
x
Exchanges contents of pattern and hold spaces
这些命令可以让 pattern space 空出来处理其他文本。一般来说，你在通过 h/H 将 pattern space 的内容移动到 hold space 之后，都会再通过 g/G/x 将内容在放回到 pattern space 中。
1 2 3 4 5 6 7 8 9 10
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line # This is the last line
sed -n '/first/ {h ; p ; n ; p ; g ; p }' data2.txt # This is the first data line. # This is the second data line # This is the first data line.
解析上面的命令
sed 通过 RE 过滤包含 first 的语句
匹配到目标语句后，开始执行 {} 中的内容，h 会将语句 copy 到 hold space 中
第一个 p 打印当前 pattern space 中内容
n 提取下一行内容并放到 pattern space
第二个 p 打印当前 pattern space 中内容, 即包含 second 的语句
g 将 hold space 中的内容再 copy 回去
第三个 p 打印当前 pattern space 中内容, 即包含 first 的语句
Negating a Command
使用叹号(!)对操作取反
1 2 3 4 5 6 7 8 9 10 11
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line # This is the last line sed -n '/header/p' data2.txt # This is the header line. sed -n '/header/!p' data2.txt # This is the first data line. # This is the second data line # This is the last line
N 也有取反操作, 之前的例子
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
sed 'N s/System\nAdministrator/Desktop\nUser/ s/System Administrator/Desktop User/ ' data4.txt # On Tuesday, the Linux Desktop # User's group meeting will be held. # All System Administrators should attend.
sed '$!N > s/System\nAdministrator/Desktop\nUser/ > s/System Administrator/Desktop User/ > ' data4.txt # On Tuesday, the Linux Desktop # User's group meeting will be held. # All Desktop Users should attend.
$!N 消失 最后一行不执行 N 的操作。。。
通过上面介绍的这些技巧，你可以利用 hold space 做文本倒序的功能
Place a line in the pattern space
Place the line from the pattern space to the hold space
Put the next line of text in the pattern space
Append the hold space to the pattern space
Place everything in the pttern space into the hold space
Repeat step 3-5 until you’ve put all the lines in reverse oder in the hold space
Retrieve the lines, and print them
1 2 3 4 5 6 7 8 9 10 11
cat -n data2.txt # 1 This is the header line. # 2 This is the first data line. # 3 This is the second data line # 4 This is the last line
sed -n '{1!G; h; $p}' data2.txt | cat -n # 1 This is the last line # 2 This is the second data line # 3 This is the first data line. # 4 This is the header line.
1!G 第一行时不用将 hold space 的内容 append 过来, 不加的话会多一个空行
h copy to hold space
$p 最后一行的话 打印
这尼玛也太精巧了把，我感觉我想不出来 （；￣ェ￣）
PS: 如果真要倒序，直接用 tac 即可， cat 的倒写
Changing the Flow
默认情况下 sed 是从头到尾的处理的，但是他也提供了方法改变处理顺序，感觉像有点像结构化语言
Branching
效果和叹号一样，只不过他是会根据 address 的标识批量操作而已
branch command: [address]b [label]
下面的例子中， sed 在做替换是根据 2,3b 跳过了第 2-3 行
1 2 3 4 5 6 7 8 9 10 11
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line. # This is the last line.
sed '{2,3b; s/This is/Is this/; s/line./test?/}' data2.txt # Is this the header test? # This is the first data line. # This is the second data line. # Is this the last test?
下面的例子，jump1 更像是 if, 如果 match 则跳过条件直接执行 :jump1 之后的命令
1 2 3 4 5 6 7 8 9 10 11 12 13
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line. # This is the last line.
sed '{/first/b jump1; s/This is the/No jump on/ > :jump1 > s/This is the/Jump here on/}' data2.txt # No jump on header line. # Jump here on first data line. # No jump on second data line. # No jump on last line.
当 b 匹配的内容出现，则跳过第一个替换，直接执行后一个。更骚的操作是下面的这个循环替换逗号的操作
1 2 3 4 5 6 7 8 9 10 11 12
echo"This, is, a, test, to, remove, commas." | sed -n '{ > :start > s/,//1p > b start > }' # This is, a, test, to, remove, commas. # This is a, test, to, remove, commas. # This is a test, to, remove, commas. # This is a test to, remove, commas. # This is a test to remove, commas. # This is a test to remove commas. # ^C
这个例子大致意思我是懂得，但是不清楚为什么执行操作的时候文本一直有效，不会被冲掉吗？可能要深入了解一下 pattern space 才能直到原因。这个cmd 需要 Ctrl + C 才能强制结束, 下面是改进版本
1 2 3 4 5 6 7 8 9 10 11
echo"This, is, a, test, to, remove, commas." | sed -n '{ :start s/,//1p /,/b start }' # This is, a, test, to, remove, commas. # This is a, test, to, remove, commas. # This is a test, to, remove, commas. # This is a test to, remove, commas. # This is a test to remove, commas. # This is a test to remove commas.
Testing
语法和 branch 很像 [address]t [label]
test command provide a cheap way to perform a basic if-then statement on the text in the data stream
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line. # This is the last line. sed '{ > s/first/matched/ > t > s/This is the/No match on/ > }' data2.txt # No match on header line. # This is the matched data line. # No match on second data line. # No match on last line.
如果 t 前面的 cmd 匹配则执行，不然直接执行后一个命令。之前循环替换逗号的例子用 t 的形式
1 2 3 4 5 6 7 8 9 10 11
echo"This, is, a, test, to, remove, commas." | sed -n '{ :start s/,//1p t start }' # This is, a, test, to, remove, commas. # This is a, test, to, remove, commas. # This is a test, to, remove, commas. # This is a test to, remove, commas. # This is a test to remove, commas. # This is a test to remove commas.
Replacing via a Pattern
通过 sed 做精确替换还是简单的, 比如下面的例子要在 cat 外面添加双引号
1 2
echo"The cat sleeps in his hat." | sed 's/cat/"cat"/' # The "cat" sleeps in his hat.
但是如果你想要在所有 .at 外面加双引号可能有会遇到问题了
1 2
echo"The cat sleeps in his hat." | sed 's/.at/".at"/g' # The ".at" sleeps in his ".at".
Using the ampersand
为了解决上面的问题，sed 提供了 & 符号指代匹配的字符
1 2
echo"The cat sleeps in his hat." | sed 's/.at/"&"/g' # The "cat" sleeps in his "hat".
Replacing individual words
如果你只想替换一部分内容，说人话就是支持 group 的模式减少 typing
1 2 3
echo"This System Administractor manual" | sed ' s/\(System\) Administractor/\1 User/' # This System User manual
group 需要用反斜线
指代 group 用反斜线加数子
下面的例子中我们用原句中的一部分代替原有部分
1 2
echo"That furry cat is pretty" | sed 's/furry \(.at\)/\1/' # That cat is pretty
这个技巧在插入值的时候很好用
1 2 3 4 5 6
echo"1234567" | sed '{ :start s/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/ t start }' # 1,234,567
有两个分组
.*[0-9]
[0-9]{3}
第一次替换结果为 1234,567，第二次 1,234,567
Placing sed Commands in Script
展示一些脚本中使用 sed 的技巧
Using wrappers
每次使用 sed 的时候现打会很累赘，你可以将他们写到脚本中并调用
下面的例子中，我们将之前实现的 reverse 功能通过脚本调用
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
cat reverse.sh #!/bin/bash # Shell wrapper for sed editor script. # to reverse text file lines sed -n '{1!G; h; $p}'$1
cat data2.txt # This is the header line. # This is the first data line. # This is the second data line. # This is the last line.
./reverse.sh data2.txt # This is the last line. # This is the second data line. # This is the first data line. # This is the header line.
cat fact.sh #!/usr/local/bin/bash # Add commas to number in factorial answer
factorial=1 counter=1 number=$1 # while [ $counter -le $number ] do factorial=$[ $factorial * $counter ] counter=$[ $counter + 1 ] done # result=$(echo$factorial | sed '{ :start s/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/ t start }') # echo"The result is $result" #
./fact.sh 20 # The result is 2,432,902,008,176,640,000
Creating sed Utilities
分享一些数据处理函数
Spacing with double lines
为文本中的每一行后面新家一个空行, 如果最后一行不想加空格，可以用叹号取反
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
sed 'G' data2.txt | cat -n # 1 This is the header line. # 2 # 3 This is the first data line. # 4 # 5 This is the second data line. # 6 # 7 This is the last line. # 8 sed '$!G' data2.txt | cat -n # 1 This is the header line. # 2 # 3 This is the first data line. # 4 # 5 This is the second data line. # 6 # 7 This is the last line.
sed '$!G' data6.txt | cat -n # 1 This is line number1. # 2 # 3 This is line number2. # 4 # 5 # 6 # 7 This is line number3. # 8 # 9 This is line number4.
解决方案，现将所有空格去了，再做加空行的操作
1 2 3 4 5 6 7 8
sed '/^$/d; $!G;' data6.txt | cat -A # This is line number 1.$ # $ # This is line number 2.$ # $ # This is line number 3.$ # $ # This is line number 4.$
PS: 使用 i\ 的语法添加空行会带有一个空格，推荐使用 1G 的方式
Numbering lines in a file
19 章时我们介绍过用 = 显示行号的操作
1 2 3 4 5 6 7 8 9
sed '=' data2.txt 1 This is the header line. 2 This is the first data line. 3 This is the second data line. 4 This is the last line.
这种格式有点奇怪，更友好的方式应该时行号和字串在一行中，这里可以用 N
1 2 3 4 5
sed '=' data2.txt | sed 'N; s/\n/ /' # 1 This is the header line. # 2 This is the first data line. # 3 This is the second data line. # 4 This is the last line.
cat data7.txt # This is line 1. # This is line 2. # This is line 3. # This is line 4. # This is line 5. # This is line 6. # This is line 7. # This is line 8. # This is line 9. # This is line 10. # This is line 11. # This is line 12. # This is line 13. # This is line 14. # This is line 15.
sed '{ :start $q; N; 11,$D b start }' data7.txt # This is line 6. # This is line 7. # This is line 8. # This is line 9. # This is line 10. # This is line 11. # This is line 12. # This is line 13. # This is line 14. # This is line 15.
cat -n data8.txt # 1 This is the header line. # 2 # 3 # 4 This is the first data line. # 5 # 6 This is the second data line. # 7 # 8 # 9 This is the last line. # 10
sed '/./,/^$/!d' data8.txt | cat -n # 1 This is the header line. # 2 # 3 This is the first data line. # 4 # 5 This is the second data line. # 6 # 7 This is the last line. # 8
Deleting leading blank lines
删除开头部分的空行
1 2 3 4 5 6 7 8 9 10 11
cat -n data9.txt # 1 # 2 # 3 This is line one. # 4 # 5 This is line two.
sed '/./,$!d' data9.txt | cat -n # 1 This is line one. # 2 # 3 This is line two.
任何有内容的行开始，到结束不删除
Deleting trailing blank lines
删除结尾部分的空行要比删除开头部分的空行麻烦一点，需要一些技巧和循环
1 2 3 4 5 6 7 8 9 10 11 12 13
cat -n data10.txt # 1 This is the first line. # 2 This is the second line. # 3 # 4 # 5
sed '{ :start /^\n*$/{$d; N; b start} }' data10.txt | cat -n # 1 This is the first line. # 2 This is the second line.
匹配任何只包含换行的 line, 如果是最后一行，则删除，如果不是就再次执行
Removing HTML tags
1 2 3 4 5 6 7 8 9 10 11 12 13
cat data11.txt # <html> # <head> # <title>This is the page title</title> # </head> # <body> # <p> # This is the <b>first</b> line in the Web page. # This should provide some <i>useful</i> # information to use in our sed script. # </p> # </body> # </html>
如果直接使用 s/<.*>//g 会出问题，一些文本类似 <b>abc</b> 也会被删除
1 2 3 4 5 6 7 8 9 10 11 12 13
sed 's/<.*>//g' data11.txt | cat -n # 1 # 2 # 3 # 4 # 5 # 6 # 7 This is the line in the Web page. # 8 This should provide some # 9 information to use in our sed script. # 10 # 11 # 12
这个是由于 sed 将内嵌的 > 识别为 .* 的一部分了，可以使用 s/<[^>]*>//g 修复, 再结合删除空格的语法
1 2 3 4 5
sed 's/<[^>]*>//g; /^$/d' data11.txt # This is the page title # This is the first line in the Web page. # This should provide some useful # information to use in our sed script.
Chapter 22: Advanced gawk
Using Variables
gawk 支持两种不同类型的变量
Built-in variables
User-defined variables
Built-in variables
这一节将展示 gawk 自带变量的使用办法
The field and record separator variables
The gawk Data Field and REcord Variables
Variable
Description
FIELDWIDTHS
A space-separated list of numbers defining the exact width(in spaces) of each data field
gawk '{ total = 0 i = 1 while (i<4) { total += $i i++ } avg = total / 3 print "Average:", avg }' data5 # Average: 128.333 # Average: 137.667 # Average: 176.667
支持 break，continue 打断循环
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
gawk '{ quote> total = 0 quote> i = 1 quote> while (i<4) quote> { quote> total += $i quote> if (i == 2) quote> break quote> i++ quote> } quote> avg = total/2 quote> print "The average of the first two data is:" , avg quote> }' data5 # The average of the first two data is: 125 # The average of the first two data is: 136.5 # The average of the first two data is: 157.5
The do-while statement
格式：
1 2 3 4
do { statemnets } while (condition)
1 2 3 4 5 6 7 8 9 10 11 12
gawk '{ quote> total = 0 quote> i = 1 quote> do quote> { quote> total += $i quote> i++ quote> } while (total < 150) quote> print total }' data5 # 250 # 160 # 315
# Displaying a long list ls -l # total 10112 # -rwxr--r-- 1 i306454 staff 159 May 30 15:56 badtest # -rw------- 1 i306454 staff 138 Jun 2 19:12 nohup.out
long list 显示格式说明
The file type, directory(d), regular file(-), linked file(l), character device(c) or block device(b)
The file permissions
The number of file hard links
The file owner username
The file primary group name
The file byte size
The last time file was modified
The filename or directory name
long list 是一个比较强力的模式，你可以收集到很多信息
Filtering listing output
过滤文件
1 2
ls -l bad* # -rwxr--r-- 1 i306454 staff 159 May 30 15:56 badtest
可用的过滤符
? 单个字符
* 多个字符
[] 多选，可以是 [ai], [a-i],[!a]
使用星号的过滤方法也叫做 file globbing
Handling Files
过一下常用的文件处理命令
Creating files
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
# touch 创建空文件 touch test_one ls -l test_one # -rw-r--r-- 1 i306454 staff 0 Jun 4 11:25 test_one
# 可以在不改变文件内容的情况下更新最后改动时间，这个之前倒是不知道 ls -l test_one # -rw-r--r-- 1 i306454 staff 3 Jun 4 11:27 test_one touch test_one ls -l test_one # -rw-r--r-- 1 i306454 staff 3 Jun 4 11:29 test_one
# -a 只修改最近访问时间 ls -a test_one # 不过 Mac 不支持这个参数
Copying files
format: cp source destination, copy 的文件是一个全新的文件
1 2 3 4 5 6 7 8 9 10
# -i 当文件已经存在时，询问是否覆盖 cp -i test_one test_two # overwrite test_two? (y/n [n]) n # not overwritten
# -d 只显示文件夹，不显示文件夹内容 ls -Fd tmp_folder # tmp_folder/ ls -F tmp_folder/ # test1.sh* test10b.out...
Linking files
Linux 中你的文件可以有一个物理主体和多个虚拟链接，这种链接即为 links。系统中有两种链接
A symbolic link
A hard link
A symbolic link is simply a physical file that points to another file somewhere in the virtual directory structure. The two symnolically linked together files do not share the same contents.
1 2 3 4 5 6 7 8
ln -s test_one sl_test_one ls -l *test_one # lrwxr-xr-x 1 i306454 staff 8 Jun 4 12:30 sl_test_one -> test_one # -rw-r--r-- 1 i306454 staff 3 Jun 4 11:29 test_one
file folder1 # folder1: directory file file2 # file2: empty ile search.xml # search.xml: XML 1.0 document text, UTF-8 Unicode text, with very long lines, with overstriking file badtest # badtest: Bourne-Again shell script text executable, ASCII text
ps -l | head # UID PID PPID F CPU PRI NI SZ RSS WCHAN S ADDR TTY TIME CMD # 501 647 646 4006 0 31 0 5457412 5200 - S+ 0 ttys000 0:04.03 -zsh
UID: The user responsible for launching the process
PID: The process ID of the process
PPID: The PID of the parent process(if a process is start by another process)
C: Processor utilization over the lifetime of the process
STIME: The system time when the process started
TTY: The termnal device from which the porcess was launched
TIME: The cumulative CUP Time required to run the process
CMD: The name of the program that wat started
F: System flags assigned to the process by the kernel
S: The state of the process. O-running on proecssor; S-sleeping; R-runnable, waiting to run; Z-zombie, process terminated but parent not availale; T-process stopped;
PRI: The priority of the process(higher numbers mean low priority)
NI: The nice value, which is used for determining priorites
ADDR: The memory address of the process
SZ: Approximate amount of swap space required if the process was swapped out
WCHAN: Address of the kernel function where the process is sleeping
# -c create new tar file # -v list process file # -f output result to file tar -cvf test.tar tmp_folder/ # a tmp_folder # a tmp_folder/test11.sh # a tmp_folder/test2.sh # a ...
ls test* # test.tar
# 并不会解压，只是看看 # -t list contents in tar tar -tf test.tar # tmp_folder/ # tmp_folder/test11.sh # tmp_folder/test2.sh # ...
# 解压 tar -xvf test.tar # x tmp_folder/ # x tmp_folder/test11.sh # ...
Tip 网上下的包很多都是 .tgz 格式的，是 gzipped tar files 的意思，可以用 tar -zxvf filename.tgz
Chapter 5: Understanding the Shell
这章将学习一些 shell process 相关的知识，子 shell 和 父 shell 的关系等
The user is added to a common group with group ID 100
The new user has a HOME account created in the directory /home/loginname
The account can’t be disabled when the password expires
The new account can’t be set to expire at a set date
The new account users the bin sh as the default shell
The system copies the contents of the /etc/skl directory to the user’s HOME directory
The system creates a file in the mail directory for the user account to receive mail
倒数第二个 item 说的是，在创建用户的时候，admin 可以预先设置一个模版
默认情况下，useradd 并不会为用户创建 HOME 目录，需要添加 -m 参数
1 2 3 4 5 6 7 8 9 10 11 12 13 14
ls -l /etc/skel # total 16 # drwxr-xr-x+ 2 root root 4096 Aug 11 2017 Desktop # -rw-r--r-- 1 root root 8980 Apr 20 2016 examples.desktop
useradd -m I306454 ls -l /home # total 28 # drwxr-xr-x+ 5 fuser fuser 4096 Aug 11 2017 fuser # drwxr-xr-x+ 4 I306454 I306454 4096 Jun 5 14:48 I306454 ls -lF /home/I306454/ # total 16 # drwxr-xr-x+ 2 I306454 I306454 4096 Aug 11 2017 Desktop/ # -rw-r--r-- 1 I306454 I306454 8980 Apr 20 2016 examples.desktop
The useradd Command Line Parameters
Parameter
Description
-c comment
Adds text to the new user’s comment field
-d home_dir
Specifies a different name for the HOME directory other than the login name
-e expire_date
Specifies a date, in YYYY-MM-DD format, when the account will expire
-f inactive_days
Specifies the number of days after a password expires when the account will be disabled. A value of 0 disables the accunt as soon as the password expires; a value of -1 disables this feature
-g initial_group
Specifies the group name or GID of the user’s login group
-G group
Specifies one or more supplementary groups the user belongs to
-k
Gopies the /etc/skel directory contents into the user’s HOME directory(must use -m as well)
-m
Create the user’s HOME directory
-M
Doesn’t create a user’s HOME directory(user if the default setting is to create one)
-n
Create a new group using the same name as the user’s login name
-r
Creates a system account
-p passwd
Specifies a default password for the user account
-s shell
Specifies the default login shell
-u
Specifies a unique UID for the account
如果你有很多默认配置需要修改，那么你可以通过 useradd -D 修改这些默认配置
Removing a user
userdel I306454 默认情况下并不会删除对应用户的 HOME 目录。你需要添加 -r 参数达到这个效果
Modifying a user
Linux 提供了一些不同的工具包来修改用户信息
User account Modification Utilities
Command
Description
usermod
Edits user accout fields, as well as specifying primary and seconday group membership
passwd
Changes the password for an existing user
chpassed
Reads a file of login name and password pairs, and updates the passwords
date ; who # Mon May 24 12:36:11 CST 2021 # i306454 console May 24 11:18 # i306454 ttys000 May 24 11:18 # i306454 ttys003 May 24 11:33
Creating a Script File
新建文件 mysh.sh
填入内容
设置环境
运行
However, the fi rst line of a shell script fi le is a special case, and the pound sign followed by the exclamation point tells the shell what shell to run the script under Shell script 的第一行表示你想要用哪个 Shell 运行你的脚本
1 2 3 4
#!/bin/bash # This scri displays the date and who's logged on date who
尝试运行 mysh.sh，运行失败 bash: mysh.sh: command not found
tdate=`date` echo$tdate # Mon May 24 15:46:51 CST 2021 tdate2=$(date) echo$tdate2 # Mon May 24 15:47:39 CST 2021
这只是初级用法，更骚的操作是将输出定制之后用作后续命令的输出，比如下面这个例子，将文件夹下的所有目录写进 log 文件，并用时间戳做后缀 today=$(date +%y%m%d); ls -al /usr/bin > log.$today
Caution: Command substitution creates what’s called a subshell to run the enclosed command. A subshell is a separate child shell generated from the shell that’s running the script. Because of that, any variables you create in the script aren’t available to the subshell command.
wc << EOF > test String 1 > test String 2 > test String 3 > EOF # 3 9 42
Pipes
使用方式 command1 | command2
Don’t think of piping as running two commands back to back. The Linux system actually runs both commands at the same time, linking them together internally in the system. As the fi rst command produces output, it’s sent immediately to the second command. No inter-mediate fi les or buffer areas are used to transfer the data. pipe 中的命令是同时执行的，变量传递不涉及到中间变量
Programming statements(such as if-then statements)
Functions
1 2 3 4 5 6 7 8 9 10
bc bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. 12 * 5.4 64.8 3.156 * (3 + 5) 25.248 quit
cat test9 # #!/usr/local/bin/bash # var1=$(echo "scale=4; 3.44/5" | bc) # echo This answer is $var1
sh test9 # This answer is .6880
脚本中定义的变量也可以使用
1 2 3 4 5 6 7 8
cat test10 # #!/usr/local/bin/bash # var1=100 # var2=45 # var3=$(echo "scale=4; $var1 / $var2" | bc) # echo The answer for this is $var3bash-5.1$ sh test10 # The answer for this is 2.2222
cat test1.sh # #!/usr/local/bin/bash # # testing the if statement # if pwd # then # echo "It worked" # fi chmod u+x test1.sh ./test1.sh # /Users/i306454/gitStore/hexo # It worked
negative sample 如下
1 2 3 4 5 6 7 8 9 10 11 12
cat test2.sh # # #!/usr/local/bin/bash # # # testing a bad command # if IamNotACommand # then # echo "It worked" # fi # echo "We are outside the if statement" chmod u+x test2.sh ./test2.sh # ./test2.sh: line 3: IamNotACommand: command not found # We are outside the if statement
cat test3.sh # #!/usr/local/bin/bash # # testing multiple commands in the then section # # # testuser=MySQL # # # if grep $testuser /etc/passwd # then # echo "This is my first command" # echo "This is my second command" # echo "I can even put in other commands beside echo:" # ls -a /home/$testuser/.b* # fi ./test3.sh # _mysql:*:74:74:MySQL Server:/var/empty:/usr/bin/false # This is my first command # This is my second command # I can even put in other commands beside echo: # ls: /home/MySQL/.b*: No such file or directory
PS：由于是 mac 系统，有点出入，但是目的还是达到了。顺便测了一下缩进，将 echo 的缩进去了一样 work
Exploring the if-then-else Statement
格式如下,
1 2 3 4 5 6
ifcommand then commands else commands fi
改进 test3.sh 如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
# #!/usr/local/bin/bash # # testing multiple commands in the then section # # # testuser=NoSuchUser # # # if grep $testuser /etc/passwd # then # echo "This is my first command" # echo "This is my second command" # echo "I can even put in other commands beside echo:" # ls -a /home/$testuser/.b* # else # echo "The user $testuser does not exist on this system." # echo # fi ./test4.sh # The user NoSuchUser does not exist on this system.
Nesting if
1 2 3 4 5 6 7
if command1 then commands elif command2 then more commands fi
写脚本检测帐户是否存在，然后检测用户文件夹是否存在
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
cat test5.sh # #!/usr/local/bin/bash # # testing nested ifs - use elif # # # testuser=NoSuchUser # # # if grep $testuser /etc/passwd # then # echo "The user $testuser exists on this system" # elif ls -d /home/$testuser # then # echo "The user $testuser does not exist on this system" # echo "However, $testuset has directory." # fi chmod u+x test5.sh ./test5.sh # ls: /home/NoSuchUser: No such file or directory
Tips: Keep in mind that, with an elif statement, any else statements immediately following it are for that elif code block. They are not part of a preceding if-then statement code block.(elif 之后紧跟的 else 是一对的，它不属于前面的 if-then)
多个 elif 串连的形式
1 2 3 4 5 6 7 8 9 10 11 12 13
if command1 then commandset 1 elif command2 then commandset 2 elif command3 then commandset 3 elif command4 then commandset 4 fi
Trying the test Command
if-then 条件判断只支持 exit code，为了使它更通用，Linux 提供了 test 工具集，如果 test 判定结果为 TRUE 则返回 0 否则非 0，格式为 test condition, 将它和 if-then 结合，格式如下
1 2 3 4
iftest condition then commands fi
如果没有写 condition，则默认为非 0 返回值
1 2 3 4 5 6 7 8 9 10 11 12
cat test6.sh # #!/usr/local/bin/bash # # Testing the test command # # # if test # then # echo "No expression return a True" # else # echo "No expression return a False" # fi ./test6.sh # No expression return a False
将测试条件替换为变量，输出 True 的语句
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
cat test6.sh # #!/usr/local/bin/bash # # Testing the test command # my_variable="Full" # # # if my_variable # then # echo "The $my_variable expression return a True" # else # echo "The $my_variable expression return a False" # fi ./test6.sh # The Full expression return a False
# replace my_variable="" ./test6.sh # The expression return a False
测试条件还可以简写成如下形式
Be careful; you must have a space after the first bracket and a space before the last bracket, or you’ll get an error message.
1 2 3 4
if [ condition ] then commands fi
test condition 可以测试如下三种情景
Numeric comparisons
String comparisons
File comparisions
Using numeric comparisons
Comparison
Description
n1 -eq n2
Check if n1 is equal to n2
n1 -ge n2
Check if n1 greater than or equal to n2
n1 -gt n2
Check if n1 is greater than n2
n1 -le n2
Check if n1 is less than or equal to n2
n1 -lt n2
Check if n1 less than n2
n1 -ne n2
Check if n1 is not equal to n2
test condition 对变量也是有效的，示例如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cat numeric_test.sh # #!/usr/local/bin/bash # # Using numeric test eveluations # value1=10 # value2=11 # # # if [ $value1 -gt 5 ] # then # echo "The test value $value1 is greater than 5" # fi # # # if [ $value1 -eq $value2 ] # then echo "The values are equal" # else # echo "The values are different" # fi ./numeric_test.sh # The test value 10 is greater than 5 # The values are different
Caution: Remember that the only numbers the bash shell can handle are integers.
Using string comparisons
Comparison
Description
str1 = str2
Check if str1 is the same as string str2
str1 != str2
Check if str1 is not the same as string str2
str1 < str2
Check if str1 is less than str2
str1 > str2
Check if str1 is greater than string str2
-n str1
Check if str1 has a length greater than zero
-z str1
Check if str1 has a length of zero
1 2 3 4 5 6 7 8 9 10 11 12 13
cat test8.sh # #!/usr/local/bin/bash # # Testing string equality # testuser=baduser # # # if [ $USER != $testuser ] # then # echo "This is $testuser" # else # echo "Welcome $testuser" # fi ./test8.sh # This is baduser
在处理 < 和 > 时，shell 会有一些很奇怪的注意点
< 和 > 必须使用转义符号，不然系统会把他们当作流操作
< 和 > 的用法和 sort 中的用法时不一致的
针对第一点的测试，测试中，比较符号被当成了流操作符，新生产了一个 hockey 文件。这个操作 exit code 是 0 所以执行了 true 的 loop. 你需要将条件语句改为 if [ $val1 \> $val2 ] 才能生效
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
cat badtest.sh # #!/usr/local/bin/bash # # mis-using string comparisons # val1=baseball # val2=hockey # # # if [ $val1 > $val2 ] # then # echo "$val1 is greater than $val2" # else # echo "$val1 is less than $val2" # fi /badtest.sh # baseball is greater than hockey ls # badtest.sh hockey
cat test9b.sh # #!/usr/local/bin/bash # # testing string sort order # val1=Testing # val2=testing # # # if [ $val1 \> $val2 ] # then # echo "$val1 is greater than $val2" # else # echo "$val1 is less than $val2" # fi ./test9b.sh Testing is less than testing
Note: The test command and test expressions use the standard mathematical comparison symbols for string compari-sons and text codes for numerical comparisons. This is a subtle feature that many programmers manage to get reversed. If you use the mathematical comparison symbols for numeric values, the shell interprets them as string values and may not produce the correct results.
test condition 的处理模式是 数字 + test codes(-eq); string + 运算符(</>/=)，刚好是交叉的，便于记忆
cat test10.sh # #!/usr/local/bin/bash # # testing string length # val1=testing # val2='' # # # if [ -n $val1 ] # then # echo "The string '$val1' is not empty" # else # echo "The string '$val1' is empty" # fi # # # if [ -z $val2 ] # then # echo "The string '$val2' is empty" # else # echo "The string '$val2' is not empty" # fi # # # if [ -z $val3 ] # then # echo "The string '$val3' is empty" # else # echo "The string '$val3' is not empty" # fi ./test10.sh # The string 'testing' is not empty # The string '' is empty # The string '' is empty
Using file comparisons
Comparison
Description
-d file
Check if file exists and is a directory
-e file
Check if file or directory exists
-f file
Check if file exists and is a file
-r file
Check if file exists and is readable
-s file
Check if file exists and is not empty
-w file
Check if file exists and is writable
-x file
Check if file exists and is executable
-O file
Check if file exists and is owned by the current user
-G file
Check if file exists and the default group is the same as the current user
case variable in pattern1 | pattern2) command1;; pattern3) command2;; *) default commands;; esac
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
cat test26.sh # #!/usr/local/bin/bash # # using the case command # case $USER in # rich | barbara) # echo "Welcome $USER" # echo "Please enjoy your visit";; # i306454) # echo "Special testing account";; # jessica) # echo "Do not forget to log off when you're done";; # *) # echo "Sorry, you are not allowed here";; # esac ./test26.sh # Special testing account
PS: for 和 do 也可以写一起(for var in list; do)，和 if-then 那样
Reading values in a list
1 2 3 4 5 6 7 8 9 10 11 12 13
cat test1.sh # #!/usr/local/bin/bash # # basic for command # for test in a b c d e # do # echo The character is $test # done ./test1.sh # The character is a # The character is b # The character is c # The character is d # The character is e
cat test1b.sh # #!/usr/local/bin/bash # # Testing the for variable after the looping # for test in a b c d e # do # echo The character is $test # done
# echo The last character is $test
# test=Connecticut # echo "Wait, now we're visiting $test" ./test1b.sh # The character is a # The character is b # The character is c # The character is d # The character is e # The last character is e # Wait, now we're visiting Connecticut
Reading complex values in a list
当 list 中包含一些标点时，结果可能就不是预期的那样了
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cat badtest1.sh # #!/usr/local/bin/bash # # another example of how not to use the for command # for test in I don't know if this'll work, append some thing more? # do # echo The character is $test # done ./badtest1.sh # The character is I # The character is dont know if thisll # The character is work, # The character is append # The character is some # The character is thing # The character is more?
解决方案：
给引号加转义符(for test in I don't know if this'll work, append some thing more?)
将 string 用双引号包裹(for test in I don”‘“t know if this”‘“ll work, append some thing more?)
cat badtest2.sh # #!/usr/local/bin/bash # # another example of how not to use the for command
# for test in Nevada New Hampshire New Mexico New York North Carolina # do # echo "Now going to $test" # done ./badtest2.sh # Now going to Nevada # Now going to New # Now going to Hampshire # Now going to New # Now going to Mexico # Now going to New # Now going to York # Now going to North # Now going to Carolina
# update to: for test in Nevada "New Hampshire" "New Mexico" "New York" ./badtest2.sh # Now going to Nevada # Now going to New Hampshire # Now going to New Mexico # Now going to New York
Reading a list from a variable
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
cat test4.sh # #!/usr/local/bin/bash # # using a variable to hold the list # list="Alabama Alaska Arizona Arkansas Colorado" # list=$list" Connecticut" # for state in $list # do # echo "Have you ever visited $state?" # done ./test4.sh # Have you ever visited Alabama? # Have you ever visited Alaska? # Have you ever visited Arizona? # Have you ever visited Arkansas? # Have you ever visited Colorado? # Have you ever visited Connecticut?
echo a b c > states cat test5.sh # #!/usr/local/bin/bash # # reading values from a file # file=states # for state in $(cat $file) # do # echo "Visit beautiful $state" # done ./test5.sh # Visit beautiful a # Visit beautiful b # Visit beautiful c
Changing the field separator
有一个特殊的环境变量叫做 IFS(internal field separator). 他可以作为分割 field 的依据。默认的分割符有
cat test6.sh # #!/usr/local/bin/bash # # iterate through all the files in a directory # for file in /Users/i306454/tmp/* # do # if [ -d "$file" ] # then # echo "$file is a directory" # elif [ -f "$file" ] # then # echo "$file is a file" # fi # done ./test6.sh # /Users/i306454/tmp/backup is a directory # /Users/i306454/tmp/bash_test is a directory # /Users/i306454/tmp/csv is a directory # /Users/i306454/tmp/dfile is a directory # /Users/i306454/tmp/ifenduser.sh is a file # /Users/i306454/tmp/plantuml is a directory # /Users/i306454/tmp/sh is a directory
PS: 在这个例子中有一个很有意思的点，在 test 中，将变量 file 使用双引号包裹起来了。这是因为 Linux 中带空格的文件或文件夹是合法的，如果没有引号，解析就会出错
Caution: It’s always a good idea to test each file or directory before trying to process it.
The C-Style for command
C 语言中 for 循环如下
The C language for command
1 2 3 4
for (i=0; i<10; i++) { printf("The next number is %d\n", i); }
bash 中也提供了类似的功能, 语法为 for (( variable assignment; condition; iteration process )) 例子：for(( a=1; a<10; a++ ))
限制：
The assignment of the variable value can contain space
The variable in the condition isn’t preceded with a dollar sign
The equation for the iteration process doesn’t use the expr command format
cat test8.sh # #!/usr/local/bin/bash # # Testing the C-style for loop # for ((i=1; i<= 3; i++)) # do # echo "The next number is $i" # done ./test8.sh # The next number is 1 # The next number is 2 # The next number is 3
cat test10.sh # #!/usr/local/bin/bash # # while command test # var1=3 # while [ $var1 -gt 0 ] # do # echo $var1 # var1=$[ $var1 - 1 ] # done ./test10.sh # 3 # 2 # 1
Using multiple test commands
while 判断的时候可以接多个条件，但是只有最后一个条件的 exit code 起决定作用。就算第一个条件我直接改为 cmd 每次都抛错，循环照常进行
还有，每个条件要新起一行, 当然用分号隔开也是可以的
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
cat test11.sh # #!/usr/local/bin/bash # # Testing a multicommand while loop # var1=3 # while echo $var1 # [ $var1 -gt 0 ] # do # echo "This is inside the loop" # var1=$[ $var1 - 1 ] # done ./test11.sh # 3 # This is inside the loop # 2 # This is inside the loop # 1 # This is inside the loop # 0
The until Command
语意上和 while 相反，但是用法一致
1 2 3 4
until test commands do other commands done
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat test12 # #!/usr/local/bin/bash # # using the until command # var1=100 # until [ $var1 -eq 0 ] # do # echo $var1 # var1=$[ $var1 - 25 ] # done ./test12 # 100 # 75 # 50 # 25
cat test1 # #!/usr/local/bin/bash # # changing the IFS value # IFS.OLD=$IFS # IFS=$'\n' # for entry in $(cat /etc/passwd) # do # echo "Values in $entry -" # IFS=: # for value in $entry # do # echo " $value" # done # done ./test1 # Values in _oahd:*:441:441:OAH Daemon:/var/empty:/usr/bin/false # _oahd # badtest1.sh # badtest2.sh # test1 # test12 # test14 # test15 # 441 # 441 # OAH Daemon # /var/empty # /usr/bin/false # ...
Controlling the loop
通过 break, continue 控制流程
The break command
打断单层循环, 这个语法适用于任何循环语句，比如 for, while, until 等
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
cat test17 # #!/usr/local/bin/bash # # Breaking out of a for loop # for var1 in 1 2 3 4 5 # do # if [ $var1 -eq 3 ] # then # break # fi # echo "Iteration number: $var1" # done # echo "The for loop is completed" ./test17 # Iteration number: 1 # Iteration number: 2 # The for loop is completed
cat test22 # #!/usr/local/bin/bash # # Continuing an outer loop # for (( a=1; a<=5; a++ )) # do # echo "Iteration $a:" # for (( b=1; b<3; b++)) # do # if [ $a -gt 2 ] && [ $a -lt 4 ] # then # continue 2 # fi # var3=$[ $a * $b ] # echo " The result of $a * $b is $var3" # done # done ./test22 # Iteration 1: # The result of 1 * 1 is 1 # The result of 1 * 2 is 2 # Iteration 2: # The result of 2 * 1 is 2 # The result of 2 * 2 is 4 # Iteration 3: # Iteration 4: # The result of 4 * 1 is 4 # The result of 4 * 2 is 8 # Iteration 5: # The result of 5 * 1 is 5 # The result of 5 * 2 is 10
Processing the Output of a Loop
for 中打印的语句可以在 done 后面接文件操作符一起导入，还有这种功能。。。那我之前写脚本用的 printf 不是显得有点呆
1 2 3 4 5 6 7 8 9 10 11 12 13
cat test23 # #!/usr/local/bin/bash # # redirecting the for output to a file # for (( a=1; a<=5; a++ )) # do # echo "The number is $a" # done > test23.txt cat test23.txt # The number is 1 # The number is 2 # The number is 3 # The number is 4 # The number is 5
PS: 试了一下，echo -n 也是 OK 的
同理，done 后面还可以接其他的命令, 这个扩展很赞
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cat test24 # #!/usr/local/bin/bash # # piping a loop to another command # for state in "North Dakota" Connecticut Illinois Alabama Tennessee # do # echo "$state is the next place to go" # done | sort # echo "This completes our travels" ./test24 # Alabama is the next place to go # Connecticut is the next place to go # Illinois is the next place to go # North Dakota is the next place to go # Tennessee is the next place to go # This completes our travels
cat test1 # #!/usr/local/bin/bash # # Using one command line parameter # factorial=1 # for (( number=1; number<=$1; number++ )) # do # factorial=$[ $factorial * $number ] # done # echo The factorial of $1 is $factorial ./test1 5 # The factorial of 5 is 120
多参数调用案例
1 2 3 4 5 6 7 8 9 10 11 12
cat test2 # #!/usr/local/bin/bash # # Testing two command line parameters # total=$[ $1 * $2 ] # echo The first parameter is $1 # echo The first parameter is $2 # echo The total value is $total
./test2 2 5 # The first parameter is 2 # The first parameter is 5 # The total value is 10
字符串作为参数
1 2 3 4 5 6 7
cat test3 # #!/usr/local/bin/bash # # Testing string parameters # echo Hello $1, glad to meet you
bash-5.1$ ./test3 jack # Hello jack, glad to meet you
如果字符串之间有空格，需要用引号包裹起来
当参数数量超过 9 个的时候，你需要用花括号来调用
1 2 3 4 5 6 7 8 9 10 11
at ./test4 # #!/usr/local/bin/bash # # handling lots of parameters # total=$[ ${10} * ${11} ] # echo The tenth parameter is ${10} # echo The eleventh parameter is ${11} # echo The total is $total ./test4 1 2 3 4 5 6 7 8 9 10 11 12 # The tenth parameter is 10 # The eleventh parameter is 11 # The total is 110
Reading the script name
`$0 代表了脚本文件的名字
1 2 3 4 5 6 7 8 9 10 11
cat test5 # #!/usr/local/bin/bash # # Testing the $0 parameter # echo the zero parameter is set to: $0
bash test5 # the zero parameter is set to: test5 ./test5 # the zero parameter is set to: ./test5 bash /Users/i306454/tmp/bash_test/test5 # the zero parameter is set to: /Users/i306454/tmp/bash_test/test5
cat test5b # #!/usr/local/bin/bash # # Using basename with the $0 parameter # name=$(basename $0) # echo the zero parameter is set to: $name
bash-5.1$ ./test5b # the zero parameter is set to: test5b sh test5b # the zero parameter is set to: test5b
Testing parameters
当脚本中需要用到参数，但是参数没有给，则脚本会抛异常，但是我们可以更优雅的处理这种情况
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat test7 # #!/usr/local/bin/bash # # Testing parameters before use # if [ -n "$1" ] # then # echo Hello $1, glad to meet you. # else # echo "Sorry, you did not identity yourself." # fi
./test7 jack # Hello jack, glad to meet you. ./test7 # Sorry, you did not identity yourself.
Using Special Parameter Variables
Counting parameters
$# 用于统计参数数量
1 2 3 4 5 6 7 8 9 10 11 12
cat test8 # #!/usr/local/bin/bash # # Getting the number of parameters # echo There were $# parameters supplied. ./test8 # There were 0 parameters supplied. ./test8 123 # There were 1 parameters supplied. ./test8 1 2 3 # There were 3 parameters supplied. ./test8 "jack zheng" # There were 1 parameters supplied.
根据上面的特性我们可以试着发散一下思路，尝试拿到最后一个参数
1 2 3 4 5 6
cat badtest1 # #!/usr/local/bin/bash # # Testing grabbing last parameter # echo The last parameter was ${$#} ./badtest1 1 2 3 # The last parameter was 13965
尝试失败，语法上来说，花括号中间是不允许有 $ 符号的，你可以用叹号表达上面的意思
1 2 3 4 5 6
# #!/usr/local/bin/bash # # Testing grabbing last parameter # echo The last parameter was ${!#}
./badtest1 1 2 3 # The last parameter was 3
Grabbing all the data
你可以使用 $* 或者 $@ 拿到所有的参数，区别如下
$* 会将所有的参数当作一个变量对待
$@ 会将所有的参数当作类似数组那种概念，分开对待。也就是说，你可以在 for 中循环处理
1 2 3 4 5 6 7 8 9 10 11
cat test11 # #!/usr/local/bin/bash # # Testing $* and $@ # echo # echo "Using the \$* method: $*" # echo "Using the \$@ method: $@"
./test11 a b c d
# Using the $* method: a b c d # Using the $@ method: a b c d
cat test12 # #!/usr/local/bin/bash # # Testing $* and $@ # echo # count=1 # # # for param in "$*" # do # echo "\$* Parameter #$count = $param" # count=$[ $count + 1 ] # done # # # echo # count=1 # # # for param in "$@" # do # echo "\$@ Parameter #$count = $param" # count=$[ $count + 1 ] # done ./test12 a b c d
# $* Parameter #1 = a b c d
# $@ Parameter #1 = a # $@ Parameter #2 = b # $@ Parameter #3 = c # $@ Parameter #4 = d
Being Shify
我们可以通过 shift 关键字将参数左移，默认左移一位.
PS: note that the value for variable $0, the program name, remains unchanged
PPS: Be careful when working with the shift command. When a parameter is shifted out, its value is lost and can’t be recovered.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
cat test13 #!/usr/local/bin/bash # Demostrating the shift command echo count=1 while [ -n "$1" ] do echo"Parameter #$count = $1" count=$[ $count + 1 ] shift done
./test13 a b c d
# Parameter #1 = a # Parameter #2 = b # Parameter #3 = c # Parameter #4 = d
移动多个位置测试
1 2 3 4 5 6 7 8 9 10 11 12 13 14
cat test14 #!/usr/local/bin/bash # Demostrating a multi-position shift echo echo"The original parameters: $*" shift 2 echo"The changed parameters: $*" echo"Here is the new first parameter: $1"
./test14 1 2 3 4
# The original parameters: 1 2 3 4 # The changed parameters: 3 4 # Here is the new first parameter: 3
Working with Options
介绍三种添加 Options 的方法，Options 顾名思义，就是命令中的可选参数。
Finding your options
单个依次处理法： 可以使用 case + shift 的语法识别 options。将预先设置的 Options 添加在 case 的过滤列表中，然后遍历 $1 识别它
cat test15 #!/usr/local/bin/bash # Extracting command line options as parameters # echo while [ -n "$1" ] do case"$1"in -a) echo"Found the -a option";; -b) echo"Found the -b option";; -c) echo"Found the -c option";; *) echo"$1 is not an option";; esac shift done
./test15 -a -b -c -d asd
# Found the -a option # Found the -b option # Found the -c option # -d is not an option # asd is not an option
cat test16 #!/usr/local/bin/bash # Extracting options and parameters # echo while [ -n "$1" ] do case"$1"in -a) echo"Found the -a option";; -b) echo"Found the -b option";; -c) echo"Found the -c option";; --) shift break;; *) echo"$1 is not an option";; esac shift done # count=1 for param in$@ do echo"Parameter #$count: $param" count=$[ $count + 1 ] done
./test16 -c -a -b test1 test2 test3
# Found the -c option # Found the -a option # Found the -b option # test1 is not an option # test2 is not an option # test3 is not an option
./test16 -c -a -b -- test1 test2 test3 # Found the -c option # Found the -a option # Found the -b option # Parameter #1: test1 # Parameter #2: test2 # Parameter #3: test3
cat test17 #!/usr/local/bin/bash # Extracting command line options and values # echo while [ -n "$1" ] do case"$1"in -a) echo"Found the -a option";; -b) param="$2" echo"Found the -b option, with parameter value $param" shift ;; -c) echo"Found the -c option";; --) shift break;; *) echo"$1 is not an option";; esac shift done # count=1 for param in$@ do echo"Parameter #$count: $param" count=$[ $count + 1 ] done
./test17 -a -b test1 -d
# Found the -a option # Found the -b option, with parameter value test1 # -d is not an option
Using the getopt command
介绍 getopt 工具函数，方便处理传入的参数
Looking at the command formatgetopt 可以接受一系列的 options 和 parameters 并以正确的格式返回, 语法如下 getopt optstring parameters
Tips getopt 还有一个增强版 getopts, 后面章节会介绍
测试 getopt, b 后面添加了冒号表示它是带值的可选参数. 如果输入的命令带有未定义的参数，则会给出错误信息。如果想要忽略错误信息，则需要 getopt 带 -q 参数
1 2 3 4 5 6 7 8 9
getopt ab:cd -a -b test1 -cd test2 test3 # -a -b test1 -c -d -- test2 test3
getopt ab:cd -a -b test1 -cde test2 test3 # getopt: illegal option -- e # -a -b test1 -c -d -- test2 test3
getopt -q ab:cd -a -b test1 -cde test2 test3 # -a -b 'test1' -c -d -- 'test2' 'test3'
#!/usr/local/bin/bash # Extracting command line options and values with getopt # set -- $(getopt ab:cd"$@") # 为了兼容 Mac version bash, 将参数去掉了 # set -- $(getopt -q ab:cd "$@") # echo while [ -n "$1" ] do case"$1"in -a) echo"Found the -a option";; -b) param="$2" echo"Found the -b option, with parameter value $param" shift ;; -c) echo"Found the -c option";; --) shift break;; *) echo"$1 is not an option";; esac shift done # count=1 for param in$@ do echo"Parameter #$count: $param" count=$[ $count + 1 ] done
./test18 -ac
# Found the -a option # Found the -c option
./test18 -a -b test1 -cd test2 test3 test4
# Found the -a option # Found the -b option, with parameter value test1 # Found the -c option # -d is not an option # Parameter #1: test2 # Parameter #2: test3 # Parameter #3: test4
./test18 -a -b test1 -cd"test2 test3" test4
# Found the -a option # Found the -b option, with parameter value test1 # Found the -c option # -d is not an option # Parameter #1: test2 # Parameter #2: test3 # Parameter #3: test4
cat test19 #!/usr/local/bin/bash # Simple demostration of the getopts command # echo whilegetopts :ab:c opt do case"$opt"in a) echo"Found the -a option" ;; b) echo"Found the -b option, with value $OPTARG" ;; c) echo"Found the -c option" ;; *) echo"Unknown option: $opt" ;; esac done
./test19 -ab test1 -c
# Found the -a option # Found the -b option, with value test1 # Found the -c option
./test19 -b "test1 test2" -a
# Found the -b option, with value test1 test2 # Found the -a option
cat test21 #!/usr/local/bin/bash # Testing the read command # echo -n "Enter your name: " read name echo"Hello $name, welcome to my program."
./test21 # Enter your name: jack # Hello jack, welcome to my program.
./test21 # Enter your name: jack zheng # Hello jack zheng, welcome to my program.
上面的例子中，cmd 会将所有的输入看作一个变量处理
带用户提示的输入
1 2 3 4 5 6 7 8 9 10 11
cat test22 #!/usr/local/bin/bash # Testing the read -p option # read -p "Please enter your age: " age days=$[ $age * 365 ] echo"That makes you over $days days old! "
./test22 # Please enter your age: 5 # That makes you over 1825 days old!
和第一个实验对照，cmd 也可以将所有输入当作 list 处理
1 2 3 4 5 6 7 8 9 10
cat test23 #!/usr/local/bin/bash # Testing the read command # read -p "Enter your name: " first last echo"Checking data for $last, $first"
./test23 # Enter your name: jack zheng # Checking data for zheng, jack
如果你没有为 read 指定变量，bash 会自动将这个值赋给环境变量 $REPLY
1 2 3 4 5 6 7 8 9 10 11 12
cat test24 #!/usr/local/bin/bash # Testing the REPLY Environment variable # read -p "Enter your name: " echo echo"Hello $REPLY, welcome to my program."
./test24 # Enter your name: jack zheng
# Hello jack zheng, welcome to my program.
Timing out
默认情况下 read 会一直阻塞在那里，等待用户输入，但是我们也可以设置一个等待时间
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
cat test25 #!/usr/local/bin/bash # Timing the data entry # ifread -t 5 -p "Please enter your name: " name then echo"Hello $name, welcome to my script" else echo echo"Sorry, too slow! " fi
./test25 # Please enter your name: # Sorry, too slow! ./test25 # Please enter your name: jack # Hello jack, welcome to my script
cat test26 #!/usr/local/bin/bash # Getting just one character of input # read -n1 -p "Do you want to continue [Y/N]? " answer case$answerin Y | y) echo echo"fine, continue on..." ;; N | n) echo echo OK, goodbye exit ;; esac echo"This is the end of the script"
./test26 # Do you want to continue [Y/N]? y # fine, continue on... # This is the end of the script
./test26 # Do you want to continue [Y/N]? n # OK, goodbye
Reading with no display
在输入一些敏感信息时，你不希望他显示在屏幕上，可以用 -s 参数
1 2 3 4 5 6 7 8 9 10 11
cat test27 #!/usr/local/bin/bash # hiding input data from the monitor # read -s -p "Enter your password: " pass echo echo"Is you password really $pass? "
./test27 # Enter your password: # Is you password really jack?
Reading from a file
Linux 系统中，可以通过 read 命令从文件中按行读取。当读完后，返回非 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
cat test28 #!/usr/local/bin/bash # Reading data from a file # count=1 cat test | whileread line do echo"Line $count: $line" count=$[ $count + 1 ] done echo"Finished process the file"
cat test # line 1 # line 2 ./test28 Line 1: line 1 Line 2: line 2 Finished process the file
cat test11 #!/usr/local/bin/bash # Redirecting output to different locations
exec 2> testerror
echo"This is the start of the script" echo"now redirecting all output to another location"
exec 1>testout
echo"This output should go to the testout file" echo"but this should go to the testerror file" >&2
./test11 # This is the start of the script # now redirecting all output to another location cat testout # This output should go to the testout file cat testerror # but this should go to the testerror file
当你改变了 STDOUT 或者 STDERR 后，要再改回来就不是那么容易了，如果你需要切换这些流，需要用到一些技巧，这些将在后面的 Creating Your Own Redirection 章节讲到
whileread line do echo"Line #$count: $line" count=$[ $count + 1 ] done
./test12 # Line #1: #!/usr/local/bin/bash # Line #2: # Redirecting file input # Line #3: # Line #4: exec 0< test12 # Line #5: count=1 # Line #6: # Line #7: while read line # Line #8: do # Line #9: echo "Line #$count: $line" # Line #10: count=$[ $count + 1 ] # Line #11: done # Line #12:
cat test19 #!/usr/local/bin/bash # Creating and using a temp file
tempfile=$(mktemp test19.XXXXXX)
exec 3>$tempfile
echo"This script writes to temp file $tempfile"
echo"This is the first line" >&3 echo"This is the second line" >&3 echo"This is the third line" >&3
exec 3>&-
echo"Done creating temp file. the contents are:" cat $tempfile
rm -f $tempfile 2> /dev/null
./test19 # This script writes to temp file test19.ksgCft # Done creating temp file. the contents are: # This is the first line # This is the second line # This is the third line ls test19* # test19
echo"Sending data to directory $tempdir" echo"This is a test line of data for $tempfile1" >&7 echo"This is a test line of data for $tempfile2" >&8
./test21 # Sending data to directory dir.hE8Hbr
ls -l dir* # total 16 # -rw------- 1 i306454 staff 44 May 30 16:53 temp.2V4FAk # -rw------- 1 i306454 staff 44 May 30 16:53 temp.59JjLm
cat dir.hE8Hbr/temp.2V4FAk # This is a test line of data for temp.2V4FAk cat dir.hE8Hbr/temp.59JjLm # This is a test line of data for temp.59JjLm
Logging Messages
有时你可能想一个流即打印到屏幕上，也输出到文件中，这个时候，你可以使用 tee
1 2 3 4
date | tee testfile # Sun May 30 16:58:30 CST 2021 cat testfile # Sun May 30 16:58:30 CST 2021
注意，tee 默认会覆盖原有内容
1 2 3 4 5 6
who | tee testfile # i306454 console May 27 15:12 # i306454 ttys000 May 27 16:21 cat testfile # i306454 console May 27 15:12 # i306454 ttys000 May 27 16:21
之前 date 的内容被覆盖了，你可以用 -a 做 append 操作
1 2 3 4 5 6
date | tee -a testfile # Sun May 30 17:00:41 CST 2021 cat testfile # i306454 console May 27 15:12 # i306454 ttys000 May 27 16:21 # Sun May 30 17:00:41 CST 2021
实操
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
cat test22 #!/usr/local/bin/bash # Using the tee command for logging
tempfile=test22file
echo"This is the start of the test" | tee $tempfile echo"This is the second of the test" | tee -a $tempfile echo"This is the end of the test" | tee -a $tempfile
./test22 # This is the start of the test # This is the second of the test # This is the end of the test cat test22file # This is the start of the test # This is the second of the test # This is the end of the test
Practical Example
解析一个 csv 文件，将其中的内容重组成一个 SQL 文件用作 import
1 2 3 4 5
<!-- cat members.csv --> Blum,Richard,123 Main St.,Chicago,IL,60601 Blum,Barbara,123 Main St.,Chicago,IL,60601 Bresnahan,Christine,456 Oak Ave.,Columbus,OH,43201 Bresnahan,Timothy,456 Oak Ave.,Columbus,OH,43201bash-5.1$ ./test23
cat test23 #!/usr/local/bin/bash # Read file and create INSERT statements for MySQL
outfile='members.sql' IFS=',' whileread lname fname address city state zip do cat >> $outfile << EOF INSERT INTO members (lname, fname, address, city, state, zip) VALUES ('$lanme', '$fname', '$address', '$city', '$state', '$zip'); EOF done < ${1}
./test23 members.csv cat members.sql INSERT INTO members (lname, fname, address, city, state, zip) VALUES ('', 'Richard', '123 Main St.', 'Chicago', 'IL', '60601'); INSERT INTO members (lname, fname, address, city, state, zip) VALUES ('', 'Barbara', '123 Main St.', 'Chicago', 'IL', '60601'); INSERT INTO members (lname, fname, address, city, state, zip) VALUES ('', 'Christine', '456 Oak Ave.', 'Columbus', 'OH', '43201');
Script Control
Handling Signals
Signal
Name
Description
1
SIGHUP
Hangs up
2
SIGINT
Interrupts
3
SIGQUIT
Stops running
9
SIGKILL
Unconditionally terminates
11
SIGSEGV
Produces segment violation
15
SIGTERM
Terminates if possible
17
SIGSTOP
Stops unconditionally, but doesn’t terminate
18
SIGTSTP
Stops or pauses, but continues to run in background
19
SIGCONT
Resumes execution after STOP or TSTP
默认情况下，bash shell 会忽略 QUIT 和 TERM 这两个信息，但是可以识别 HUP 和 INT。
Generating signals
通过键盘操作你可以产生两种信号
Interrupting a processCtrl + C 可以生成 SIGINT 信号并把它发送到任何终端正在执行的进程中
Pausing a processCtrl + Z 停止一个进程
1 2 3 4 5 6
sleep 100 # ^Z # [1]+ Stopped sleep 100 exit # exit # There are stopped jobs.
cat test1.sh #!/usr/local/bin/bash # Testing signal trapping # trap"echo ' Sorry! I have trapped Ctrl-C'" SIGINT # echo This is a test script # count=1 while [ $count -le 10 ] do echo"Loop #$count" sleep 1 count=$[ $count + 1 ] done # echo"This is the end of the test script"
./test1.sh # This is a test script # Loop #1 # Loop #2 # Loop #3 # Loop #4 # ^C Sorry! I have trapped Ctrl-C # Loop #5 # Loop #6 # Loop #7 # Loop #8 # Loop #9 # Loop #10 # ^C Sorry! I have trapped Ctrl-C # This is the end of the test script
Trapping a script exit
trap 命令还可以做到，当脚本结束时执行命令的效果，即使是通过 Ctrl + C 结束也会被出发。
cat test2.sh #!/usr/local/bin/bash # Trapping the script exit # trap"echo Goodbye..." EXIT # count=1 while [ $count -le 5 ] do echo"Loop #$count" sleep 1 count=$[ $count + 1 ] done # echo"This is the end of the test script"
cat test5.sh #!/usr/local/bin/bash # Test running in the background with output # echo"Start the test script" count=1 while [ $count -le 5 ] do echo"Loop #$count" sleep 1 count=$[ $count + 1 ] done # echo"Test script is complete"
./test5.sh & # [1] 16474 # bash-5.1$ Start the test script # Loop #1 # Loop #2 # Loop #3 # Loop #4 # Loop #5 # Test script is complete
# [1]+ Done ./test5.sh
Running multiple background jobs
如果你想启动多个 background job 只需要终端运行多个 xx.sh & 即可。每次系统多会分配一个 job id 和 process id 给后台进程，可以通过 ps 查看
nohup ./test1.sh & cat nohup.out # This is a test script # Loop #1 # Loop #2 # Loop #3 # Loop #4 # Loop #5 # Loop #6 # Loop #7 # Loop #8 # Loop #9 # Loop #10 # This is the end of the test script # [1]+ Done nohup ./test1.sh
Controlling the Job
job control 即 开始/通知/kill/重启 jobs 的动作
Viewing jobs
jobs cmd 让你可以查看 shell 正在运行的 jobs
下面的例子中，我们启动一个计时器脚本。第一次运行，中间通过 Ctrl + Z stop 它。第二次采用后台运行。然后通过 jobs 命令观察这两个 job 的状态。jobs -l 可以显示 PID
cat test10.sh #!/usr/local/bin/bash # Test job control # # $$ to display the PID of process running this script echo"Script Process ID: $$" # count=1 while [ $count -le 10 ] do echo"Loop #$count" sleep 1 count=$[ $count + 1 ] done # echo"End of script..."
Note 以前还有一个 batch 命令可以让你在 low useage 状态下运行 script，现在它只是通过调用 at + b queue 完成的定时脚本。队列的字母顺序越靠后，优先级越低。默认 job 优先级为 a
Retrieving job output
Linux 系统中，当 job 运行时是没有监测的地方的。系统会将内容记录到邮件中并发送给联系人
1 2 3 4 5 6 7 8 9 10 11
cat test13.sh #!/usr/local/bin/bash # Test using at command # echo"This script ran at $(date +%B%d,%T)" echo sleep 5 echo"This is the script's end" #
at -f test13.sh now # job 1 at Thu Jun 3 12:27:19 2021
如果你系统没有配置邮箱，那就收不到邮件了，你可以直接指定输入到文件
PS: 这个实验失败，运行后我并没有看到 out 文件
1 2 3 4 5 6 7 8 9 10 11 12 13
cat test13b.sh #!/usr/local/bin/bash # Test using at command # echo"This script ran at $(date +%B%d,%T)" > test13b.out echo >> test13b.out echo sleep 5 echo"This is the script's end" >> test13b.out #
# 查了一下 at 并没有 -M 这个可选参数啊。。 at -M -f test13b.sh now # job 3 at Thu Jun 3 12:34:11 2021
Listing pending jobs
显示所有 pendng 的 job, 我还以为也是用 jobs 呢，忙乎了半天
1 2 3 4 5 6 7
atq # 1 Thu Jun 3 12:27:00 2021 # 4 Thu Jun 3 12:35:00 2021 # 5 Thu Jun 3 12:35:00 2021 # 2 Thu Jun 3 12:32:00 2021 # 3 Thu Jun 3 12:34:00 2021 # 6 Thu Jun 3 16:00:00 2021
Removing jobs
删除 at queue 中的 job
1 2 3 4 5 6 7
atrm 1 atq # 4 Thu Jun 3 12:35:00 2021 # 5 Thu Jun 3 12:35:00 2021 # 2 Thu Jun 3 12:32:00 2021 # 3 Thu Jun 3 12:34:00 2021 # 6 Thu Jun 3 16:00:00 2021
