Tag Archives: linux

Bash array tutorial

Like some of other advanced program language, Bash also has Array data structures. There are some basic array tutorials can be found in The Ultimate Bash Array Tutorial with 15 Examples

Syntax:
declare -a Unix=('Debian' 'Red hat' 'Ubuntu' 'Suse' 'Fedora' 'UTS' 'OpenLinux');

Also the keyword declare -a can be omitted.

There are still some tips in daily really shell scripts:

Declare Array more simple

Array is created automatically when a variable is used in the format like,

name[index]=value

Length of the Array vs Length of the nth Element

We can get the length of an array using the special parameter called $#.

${#arrayname[@]} gives you the length of the whole array.

But, if the @ sign replace with nth of the element (>=1), then gives you the length of the nth Element in an array. Also if omit the [nth], the nth Defaults to the first element of the array.

echo ${#Unix[@]} # Number of elements in the array. => 7
echo ${#Unix}  # Number of characters in the first element located at index 1. i.e Debian => 6
echo ${#Unix[2]} # Echo the 2th element 'Red hat' length => 7 

Difference between @ and * when referencing array values

This Bash guide says:

If the index number is @ or *, all members of an array are referenced.

LIST=(1 2 3)
for i in "${LIST[@]}"; do
  echo "example.$i "
done

Gives: example.1 example.2 example.3 (desired result).

But if use ${LIST[*]}, The loop will get example.1 2 3 instead.

when using echo, @ and * actually do give the same results like,

echo ${LIST[@]}
echo ${LIST[*]}

both echos get the desired result: 1 2 3

The difference is subtle; $* creates one argument, while $@ will expand into separate arguments, so:

for i in "${LIST[@]}"

will deal with the list (print it) as multiple variables

but

for i in "${LIST[*]}"

will deal with the list as one variable.

Read Content of a File into an Array

You can load the content of the file line by line into an array by cat, example like,

$ cat loadcontent.sh

filecontent=( `cat "logfile" `)
for t in "${filecontent[@]}"; do
  echo $t
done
echo "Read file content!"

Also you can use [read][3] for more duplex through for loop. such as Reading Columns like,

var="one two three"
read -r col1 col2 col3 <<< "$var"
printf "col1: %s, col2: %s, col3 %s\n" "$col1" "$col2" "$col3"

Dump first column value of each line

while read -r -a line; do
  i=$((${#line[@]} - 1));
  [ $i -eq -1 ] || echo "${line["$i"]}";
done <~/.ssh/config

Parse predefine config in ~/.ssh/config like,

$ cat ~/.ssh/config
#def USER_NAME apps
#def HOST_PREFIX 10.200.51
Host *
    ControlMaster auto
    ControlPath ~/.ssh/master-%r@%h:%p
...
while read -r x k v; do
  if [ "$x" == "#def" ]; then
    echo "{$k/$v}";
  fi;
done <~/.ssh/config

In the above example, the k/v prefixing with #def has printed through for loop.

CentOS how to tips

How to remove RPM packages with several dependencies

If you are using fedora, simply use this simple script but be careful when answering y/N:

yum remove $(rpm -qa | grep PACKAGENAME)

  • Change PACKAGENAME with your Package name
  • For disabling plugins just add --disableplugin=PLUGIN-NAME
  • If you can’t access the Internet, just add this options to the line above --disablerepo=*

Find out what files are in my rpm package

Use following syntax to list the files for already INSTALLED package:

The –v (verbose) option can give you more information on the files when used with the various query options.

rpm -ql package-name

Use following syntax to list the files for RPM package:

rpm -qlp package-name

Type the following command to list the files for gitlab*.rpm package file:

rpm -qlp gitlab-7.1.1_omnibus-1.el6.x86_64.rpm

See also: HowTo: Extract an RPM Package Files Without Installing It

Update yum repositories for CentOS, RHEL Systems

Get the latest yum repos from one of the two links below, selecting to match your host’s architecture:

# CentOS/RHEL 6, 64 Bit (x86_64):
rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Then enjoy update with yum update yum-updatesd

Change CentOS language

vi /etc/sysconfig/i18n

check the lang is your expected, such as:

LANG="en_US.UTF-8"  <<-----
SUPPORTED="en_US.UTF-8:en_US:en" 
SYSFONT="latarcyrheb-sun16"

and re-login with you user/passwd, check it with command locale

Yum install/update with specific repository

# update git with rpmforge-extras repository
yum --disablerepo=base,updates --enablerepo=rpmforge-extras update git

Linux daily skills (continuous updating)

Cleanup process list

Kill these process that zombie or stopped.

# kill zombie process list
ps -A -ostat,ppid | grep -e '[zZ]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

# cleanup stopped process list
ps -A -ostat,pid | grep -e '[T]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

Mandatory logged out user session

Tips: 当出现服务器用户数过多,造成别人登陆不上去,管理员可强行踢出用户

w list current logon sessions, and the kill it with pkill -kill -t [tty]

pkill -kill -t pts/2

Make Tab auto-completion case-insensitive

# If ~./inputrc doesn't exist yet, first include the original /etc/inputrc so we don't override it
if [ ! -a ~/.inputrc ]; then echo "\$include /etc/inputrc" > ~/.inputrc; fi

# Add option to ~/.inputrc to enable case-insensitive tab completion
echo "set completion-ignore-case On" >> ~/.inputrc

Note: to make this change for all users, edit /etc/inputrc

Get the networking connection statistics

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

TIME_WAIT 22
ESTABLISHED 3254   # data transfer state
LAST_ACK 236
FIN_WAIT_1 648
FIN_WAIT_2 581
CLOSING 7
CLOSE_WAIT 4916

Kill the process that bind the specific port

GistURL: https://gist.github.com/allex/8b399a93749703acd780

#!/bin/sh

port=$1
while [[ -z "${port}" ]]; do
    read -p "Please type the port you wanna kill (q to exit): " port
done
[ "${port}" = "q" ] && exit 0;

PID=""
case "`uname `" in
    Linux*)  PID=`lsof -t -i:${port}` ;;
    Darwin*) PID=`lsof -i -n -P | grep ":${port} (LISTEN)" | awk '{print $2}'` ;;
esac

if [ -n "$PID" ];
then
    kill -9 $PID && echo "procss with pid: ${PID} on port ${port} had killed success!"
else
    echo "The specific process with port '${port}' not found. exit";
fi

Usage: ./killport.sh [port]

Find IP address in shell

# OS X may not work except
ip = `hostname -I | cut -d' ' -f1`

# or use complex Linux shell
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`

Example:

#!/bin/sh
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`
wget -q -O - http://${ip}:8092/yn/build.hash -H 'Host: st.comm.miui.com' | base64 -d

Compare differences between directories

cp -R $local $bak
rsync $server:$remdir/* $local/
rsync $local/ $server:$remdir/*
diff -wur $local $bak

CentOS LOG – Safety optimizations

Kernel optimization vi /etc/sysctl.conf

We can view the system kernel settings by sysctl -a.

# Not accept ICMP redirects (prevent MITM attacks)
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0

优化内核阻挡SYN洪水攻击 sysctl -a | grep syn

# 设置syncookies:
sysctl -w net.ipv4.tcp_syncookies=1
sysctl -w net.ipv4.tcp_max_syn_backlog=3072
sysctl -w net.ipv4.tcp_synack_retries=0
sysctl -w net.ipv4.tcp_syn_retries=0
sysctl -w net.ipv4.conf.all.send_redirects=0
sysctl -w net.ipv4.conf.all.accept_redirects=0
sysctl -w net.ipv4.conf.all.forwarding=0
sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1

# 防止PING:
sysctl -w net.ipv4.icmp_echo_ignore_all=1

Add iptables to avoid Sync Flood Attack

# 防止Sync Flood, 缩短SYN- Timeout时间 (-limit 1/s 限制SYN并发数每秒1次,可以根据自己的需要修改)
iptables -A FORWARD -p tcp --syn -m limit --limit 1/s -j ACCEPT
iptables -A INPUT -i eth0 -m limit --limit 1/sec --limit-burst 5 -j ACCEPT

# 防止各种端口扫描
iptables -A FORWARD -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 1/s -j ACCEPT

# 防止 Ping of Death 攻击
iptables -A FORWARD -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT

# 每秒 最多3个 syn 封包 进入
iptables -N syn-flood
iptables -A INPUT -p tcp --syn -j syn-flood
iptables -A syn-flood -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j RETURN
iptables -A syn-flood -j REJECT

# 拦截具体IP范围 (eg. 10.0.0.0/8)
iptables -A INPUT -s 10.0.0.0/8 -i eth0 -j Drop

开防火墙,不用的端口都禁止掉

iptables -F
iptables -A INPUT -p tcp -i vnet0 –dport ssh -j ACCEPT
iptables -A INPUT -p tcp -i vnet0 –dport 80 -j ACCEPT
iptables -A INPUT -i vnet0 -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p ICMP -j DROP
iptables -A INPUT -i vnet0 -j DROP

修改好之后重启iptables

/etc/init.d/iptables restart

libpcre.so.1: cannont open shared object file: No such file or directory.

after installed nginx, got some error message when raunch as: `/opt/nginx/sbin/nginx`

/opt/nginx/sbin/nginx: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory

still i’m sure i’ve installed the latest pcre. and also i can find the libpcre.so.1 with:

find /usr/ -name "libpcre.so.1"
/usr/local/lib/libpcre.so.1

ok, why cannot found the libpcre.so.1 in `/user/local/lib` ??

strace /opt/nginx/sbin/nginx

So how does the dynamic loader know where to look for executables? As with many things on Linux, there is a configuration file in /etc. In fact, there are two configuration files, /etc/ld.so.conf and /etc/ld.so.cache. Note that /etc/ld.so.conf specifies that all the .conf files from the subdirectory ld.so.conf.d should be included.
Dynamic library configuration

ldconfig -p | grep "libpcre.so.1"

not found any matches.

so the problem is the dynamic loader not serach for my lib dir in /usr/local/lib

Then Use shared libraries in /usr/local/lib??

For the current session you can

export LD_LIBRARY_PATH=/lib:/usr/lib:/usr/local/lib

or to make the change permanent you can add /usr/local/lib to /etc/ld.so.conf (or something it includes) and run `ldconfig` as root.

keep reading. If not, read aboout ldconfig first.

after all, grep it:

ldconfig -p | grep "libpcre.so.1"
libpcre.so.1 (libc6,x86-64) => /usr/local/lib/libpcre.so.1

ok, got it!
run again /opt/nginx/sbin/nginx

that all..

these are some keyword need to read for details: `strace`, `ldconfig`, `/etc/ld.conf`, `/etc/ld.conf.cache`.

Upgrade Bash to 4+ on OS X and Use GNU Instead of BSD Command

Install latest bash

Install by brew for OS X user brew install bash

Compiling bash with source code: http://ftp.gnu.org/gnu/bash/

  • Add /usr/local/bin/bash to /etc/shells
  • Change the default shell with chsh -s /usr/local/bin/bash
  • To see your current version of bash do the following: echo $BASH_VERSION

Install and Use GNU Command Line Tools on Mac OS X

If you are moving onto Mac OS X from Linux, you would probably find out that the command line tools shipped with Mac OS X are not as powerful and easy to use as the tools in Linux. The reason is that Mac OS X uses the BSD version command line tools, which are different from the Linux version, while they are both compliant with POSIX standards.

  • Install the GNU Command Line Tools

    Get the latest GNU Coreutils and install to /usr/local/coreutils/

    wget http://ftpmirror.gnu.org/coreutils/coreutils-8.22.tar.xz
    tar xzf coreutils-8.22.tar.xz && cd coreutils-8.22
    ./configure --prefix=/usr/local/coreutils-8.22
    sudo make install
    sudo ln -sf /usr/local/coreutils-8.22 /usr/local/coreutils
    
  • Enable GNU coreutils, add coreutils bin folder to $PATH

    add script to .profile or .bash_profile

    export PATH="/usr/local/coreutils/bin:$PATH"
    

Links:

disable X from loading on start up

Assumes that you have the X Window System installed and will be using either GNOME or KDE as your graphical X Window System.

In order to disable X from loading during the boot sequence, it is recommended that you boot your system to runlevel 3. Booting your system to runlevel 3 will still allow you to log in as a system user but will provide a command line login instead of a graphical login. Once you are successfully logged in, you can then use the startx command to bring your sy…

Edit /etc/inittab with your favourite editor and change the 5 to a 3:

vim /etc/inittab

# Default runlevel. The runlevels used by RHS are:
#   0 - halt (Do NOT set initdefault to this)
#   1 - Single user mode
#   2 - Multiuser, without NFS (The same as 3, if you do not have networking)
#   3 - Full multiuser mode
#   4 - unused
#   5 - X11
#   6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:

Reboot and DONE!