Category Archives: linux

Bash array tutorial

Like some of other advanced program language, Bash also has Array data structures. There are some basic array tutorials can be found in The Ultimate Bash Array Tutorial with 15 Examples

Syntax:
declare -a Unix=('Debian' 'Red hat' 'Ubuntu' 'Suse' 'Fedora' 'UTS' 'OpenLinux');

Also the keyword declare -a can be omitted.

There are still some tips in daily really shell scripts:

Declare Array more simple

Array is created automatically when a variable is used in the format like,

name[index]=value

Length of the Array vs Length of the nth Element

We can get the length of an array using the special parameter called $#.

${#arrayname[@]} gives you the length of the whole array.

But, if the @ sign replace with nth of the element (>=1), then gives you the length of the nth Element in an array. Also if omit the [nth], the nth Defaults to the first element of the array.

echo ${#Unix[@]} # Number of elements in the array. => 7
echo ${#Unix}  # Number of characters in the first element located at index 1. i.e Debian => 6
echo ${#Unix[2]} # Echo the 2th element 'Red hat' length => 7 

Difference between @ and * when referencing array values

This Bash guide says:

If the index number is @ or *, all members of an array are referenced.

LIST=(1 2 3)
for i in "${LIST[@]}"; do
  echo "example.$i "
done

Gives: example.1 example.2 example.3 (desired result).

But if use ${LIST[*]}, The loop will get example.1 2 3 instead.

when using echo, @ and * actually do give the same results like,

echo ${LIST[@]}
echo ${LIST[*]}

both echos get the desired result: 1 2 3

The difference is subtle; $* creates one argument, while $@ will expand into separate arguments, so:

for i in "${LIST[@]}"

will deal with the list (print it) as multiple variables

but

for i in "${LIST[*]}"

will deal with the list as one variable.

Read Content of a File into an Array

You can load the content of the file line by line into an array by cat, example like,

$ cat loadcontent.sh

filecontent=( `cat "logfile" `)
for t in "${filecontent[@]}"; do
  echo $t
done
echo "Read file content!"

Also you can use [read][3] for more duplex through for loop. such as Reading Columns like,

var="one two three"
read -r col1 col2 col3 <<< "$var"
printf "col1: %s, col2: %s, col3 %s\n" "$col1" "$col2" "$col3"

Dump first column value of each line

while read -r -a line; do
  i=$((${#line[@]} - 1));
  [ $i -eq -1 ] || echo "${line["$i"]}";
done <~/.ssh/config

Parse predefine config in ~/.ssh/config like,

$ cat ~/.ssh/config
#def USER_NAME apps
#def HOST_PREFIX 10.200.51
Host *
    ControlMaster auto
    ControlPath ~/.ssh/master-%r@%h:%p
...
while read -r x k v; do
  if [ "$x" == "#def" ]; then
    echo "{$k/$v}";
  fi;
done <~/.ssh/config

In the above example, the k/v prefixing with #def has printed through for loop.

Add cron jobs to delete logs periodically

From Wikipedia:

cron is the time-based job scheduler in Unix-like computer operating systems. cron enables users to schedule jobs (commands or shell scripts) to run periodically at certain times or dates. It is commonly used to automate system maintenance or administration.

The final cron job script is here https://gist.github.com/c487d6d8b2b52219a87b

sudo crontab –e
2 0 * * * /opt/tools/delete_old_logs.sh /var/log >> /var/log/cron_job.log 2>&1

Crontab format

The basic format for a crontab is:

minute hour day_of_month month day_of_week command [args]
  • 1: minute (0-59)
  • 2: hour (0-23)
  • 3: day_of_month (0-31)
  • 4: month (0-12 [12 == December])
  • 5: day_of_week Day of the week(0-7 [7 or 0 == sunday])
  • /path/to/command – Script or command name to schedule

Easy to remember format:

* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

Multiple times may be specified with a comma(,), a range can be given with a hyphen(-), and the asterisk symbol(*) is a wildcard character. Spaces are used to separate fields. For example, the line:

*/5 9-16 * 1-5,9-12 1-5 ~/bin/i_love_cron.sh

Will execute the script i_love_cron.sh at five minute intervals from 9 AM to 4:55 PM on weekdays except during the summer months (June, July, and August). More examples and advanced configuration techniques can be found below.

Basic commands for cron management

Crontabs should never be edited directly; instead, users should use the [crontab][1] program to work with their crontabs. To be granted access to this command, user must be a member of the users group (see the gpasswd command).

To edit their crontabs, they may use:

$ crontab -e

Note: By default the crontab command uses the vi editor. To change it, export EDITOR or VISUAL, or specify the editor directly: EDITOR=vim crontab -e.

To view their crontabs, users should issue the command:

$ crontab -l

To remove their crontabs, they should use:

$ crontab -r

Remove or delete single cron job using command

crontab -l | grep -v '/var/crontab/xxx.sh' | crontab -

crontab -l lists the current crontab jobs

grep -v filter some line

crontab - adds all the printed stuff into the crontab file.

Related Articles

Running compass watch in the background

Since a long time ago, i wanna write a script to auto compile sass -> css. A slight disadvantage (besides of having a ton of advantages) of using Sass over native CSS is that you need to compile your Sass files to CSS before loading them up in your browser by compass command.

In bash shell script, the ampersand “&” is used to fork processes and runn in the background:

compass watch [path/to/project] &

Will cause the find command to be forked and run in the background (you can always kill it by it’s PID)

The problem with using the & (ampersand) for forking a process in the shell is that whenever you close the shell the process is going to be killed because the parent process is killed.

& runs the whole thing in the background, giving you your prompt back immediately.

nohup allows the background process to continue running even after the user logs out (or exits the initiating shell).

nohup compass watch &

Explanation:

Every Linux process opens three I/O channels, an input “stdin”, a standard output “stdout” and a standard error output “stderr”. They can be used for binary but are traditionally text. When most programs see stdin close, they exit (this can be changed by the programmer).

When the parent shell exits, stdin is closed on the children, and (often, usually) the children exit as well. In addition the children receive a software signal, SIGHUP, indicating the user has “hung up” (formerly, the modem) and the default here is to exit as well. (Note, a programmer can change all of this when writing the program).

So, what nohup does is give the child process a separate I/O environment, tying up the ins and outs to something not tied to the parent shell, and shielding the child from the SIGHUP signal. Once the user disconnects, you will see the nohup background process owned by init (process 1), not the user’s shell.

Save following script as ~/bin/sass-watch and chmod +x for x permission.

cmd_dir=`pwd`
pidfile="$cmd_dir/.sass.pid"
logfile="$cmd_dir/.sass.log"
if [ -f $pidfile ]; then
    kill -9 `cat $pidfile` >/dev/null 2>&1
    [ "$1" = "stop" ] && echo 'shutdown!' && exit 0;
fi
nohup compass watch>$logfile 2>&1&
echo $! >$pidfile
echo "compass watch success, pid: $!";

Also the final script is https://gist.github.com/allex/965f3e1a0876592db33f

Run sass-watch on your sass project for launch daemon.

Some helper reference at internal variables

$! represents the PID of the last process executed.

$$ means the process ID that the script file is running under. For any given script, when it is run, it will have only one “main” process ID. Regardless of how many subshells you invoke, $$ will always return the first process ID associated with the script.

CentOS how to tips

How to remove RPM packages with several dependencies

If you are using fedora, simply use this simple script but be careful when answering y/N:

yum remove $(rpm -qa | grep PACKAGENAME)

  • Change PACKAGENAME with your Package name
  • For disabling plugins just add --disableplugin=PLUGIN-NAME
  • If you can’t access the Internet, just add this options to the line above --disablerepo=*

Find out what files are in my rpm package

Use following syntax to list the files for already INSTALLED package:

The –v (verbose) option can give you more information on the files when used with the various query options.

rpm -ql package-name

Use following syntax to list the files for RPM package:

rpm -qlp package-name

Type the following command to list the files for gitlab*.rpm package file:

rpm -qlp gitlab-7.1.1_omnibus-1.el6.x86_64.rpm

See also: HowTo: Extract an RPM Package Files Without Installing It

Update yum repositories for CentOS, RHEL Systems

Get the latest yum repos from one of the two links below, selecting to match your host’s architecture:

# CentOS/RHEL 6, 64 Bit (x86_64):
rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Then enjoy update with yum update yum-updatesd

Change CentOS language

vi /etc/sysconfig/i18n

check the lang is your expected, such as:

LANG="en_US.UTF-8"  <<-----
SUPPORTED="en_US.UTF-8:en_US:en" 
SYSFONT="latarcyrheb-sun16"

and re-login with you user/passwd, check it with command locale

Yum install/update with specific repository

# update git with rpmforge-extras repository
yum --disablerepo=base,updates --enablerepo=rpmforge-extras update git

Grub Booting the ISO

安装多个 Linux 版本时,通常情况下我们拿到的都是在 iso 文件, 那么如何用 grub 引导iso镜像呢?
So, 我们需要在GRUB引导菜单列表来添加一个启动项。

环境:Ubuntu 13.10 / x64

sudo vim /boot/grub/grub.cfg

添加 Grub 启动项:

menuentry "Ubuntu 13.10 ISO" {
    # set isofile="/home/<username>/Downloads/ubuntu-13.10-desktop-amd64.iso"
    # or set
    isofile="/allex/Downloads/ubuntu-13.10-desktop-amd64.iso"
    # if you use a single partition for your $HOME
    loopback loop (hd0,8)$isofile
    linux (loop)/casper/vmlinuz.efi boot=casper iso-scan/filename=$isofile noprompt noeject
    initrd (loop)/casper/initrd.lz
}

Note:

  1. 找到 iso 文件的所属分区路径。df -h 查看目录所在分区,修改到上面的 loopback loop (hd0,8)
  2. 同步自己 iso 镜像里的 casper/ 目录下面的内核引导文件名称,vmlinuz.*, initrd.lz
  3. /boot/grub/grub.cfg 文件是自己动生成的,为防止系统擦掉,可以把上面代码放在 /etc/grub.d/40_custom, 完了执行一下 update-grub
  4. Windows用户可参考 GRUB4DOS, doc

Reference Links:

Linux daily skills (continuous updating)

Cleanup process list

Kill these process that zombie or stopped.

# kill zombie process list
ps -A -ostat,ppid | grep -e '[zZ]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

# cleanup stopped process list
ps -A -ostat,pid | grep -e '[T]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

Mandatory logged out user session

Tips: 当出现服务器用户数过多,造成别人登陆不上去,管理员可强行踢出用户

w list current logon sessions, and the kill it with pkill -kill -t [tty]

pkill -kill -t pts/2

Make Tab auto-completion case-insensitive

# If ~./inputrc doesn't exist yet, first include the original /etc/inputrc so we don't override it
if [ ! -a ~/.inputrc ]; then echo "\$include /etc/inputrc" > ~/.inputrc; fi

# Add option to ~/.inputrc to enable case-insensitive tab completion
echo "set completion-ignore-case On" >> ~/.inputrc

Note: to make this change for all users, edit /etc/inputrc

Get the networking connection statistics

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

TIME_WAIT 22
ESTABLISHED 3254   # data transfer state
LAST_ACK 236
FIN_WAIT_1 648
FIN_WAIT_2 581
CLOSING 7
CLOSE_WAIT 4916

Kill the process that bind the specific port

GistURL: https://gist.github.com/allex/8b399a93749703acd780

#!/bin/sh

port=$1
while [[ -z "${port}" ]]; do
    read -p "Please type the port you wanna kill (q to exit): " port
done
[ "${port}" = "q" ] && exit 0;

PID=""
case "`uname `" in
    Linux*)  PID=`lsof -t -i:${port}` ;;
    Darwin*) PID=`lsof -i -n -P | grep ":${port} (LISTEN)" | awk '{print $2}'` ;;
esac

if [ -n "$PID" ];
then
    kill -9 $PID && echo "procss with pid: ${PID} on port ${port} had killed success!"
else
    echo "The specific process with port '${port}' not found. exit";
fi

Usage: ./killport.sh [port]

Find IP address in shell

# OS X may not work except
ip = `hostname -I | cut -d' ' -f1`

# or use complex Linux shell
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`

Example:

#!/bin/sh
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`
wget -q -O - http://${ip}:8092/yn/build.hash -H 'Host: st.comm.miui.com' | base64 -d

Compare differences between directories

cp -R $local $bak
rsync $server:$remdir/* $local/
rsync $local/ $server:$remdir/*
diff -wur $local $bak

Use cron job to cleanup log files

Linux system various kinds logs and tmp generated in /var/log/, /tmp, How to clean these files automatically?

Using tmpwatch to automate temporary file cleanup

first we need install the 3rd tool tmpwatch

yum install tmpwatch -y

once tmpwatch is installed run command

/usr/sbin/tmpwatch -am 12 /tmp

this will delete all files over 12 hours old

next, we will configure your server to do this automatically.

from SSH type: crontab -e

go to the very bottom and paste

0 4 * * * /usr/sbin/tmpwatch -am 12 /var/log

For more daily job script:

$ cat /etc/cron.daily/tmpwatch

flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
    -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix 240 /tmp
/usr/sbin/tmpwatch "$flags" 720 /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
  if [ -d "$d" ]; then
    /usr/sbin/tmpwatch "$flags" -f 720 "$d"
  fi
done

-x is an entry to be excluded from the clean up operation.


Using a shell script do the same thing if none tmpwatch

find /var/log -type f -name "*.tmp" -exec rm {} \+

Normally we can execute as find /path -name "*.tmp" -exec rm {} \;
This may sometimes fail to work because the argument list may grow larger (in bytes) than the maximum allowed by the shell (getconf ARG_MAX). This may be solved by xargs with the -L option.

Also configure as a cron job to run automatically.

find /var/log -type f -mtime +12 -print0 | xargs -0 -L 5000 rm

Reference Links:

sudo & redirect output

Like normally command append host sudo echo 127.0.0.1 local.host > /etc/hosts, but we’ll get the Permission denied error message.

sudo testparm /etc/samba/smb.conf.master > /etc/samba/smb.conf
bash: /etc/samba/smb.conf: Permission denied

Then how to sudo and redirect output to a file?

One solution is to use sh -c option:

sudo sh -c "testparm /etc/samba/smb.conf.master > /etc/samba/smb.conf"

Or you could use a tee command like this:

testparm /etc/samba/smb.conf.master | sudo tee /etc/samba/smb.conf

More about tee usage

http://linux.101hacks.com/unix/tee-command-examples/

Get absolute path from a shell path

While coding shell script, normally get the current directory of the script file like:

sh_dir=`cd -P — “$(dirname “$0″)” && pwd -P`

But, if running by source like source foo.sh, we may got the unexpected result.

Here is a safer and more readable way to do this job:

# get current script directory
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

# get current executing script full path
sh_path=$( unset CDPATH && cd "$(dirname "$0")" && echo $PWD/$(basename "$0") )

Notes:

  • If $0 is a bare filename with no preceding path, the original script will fail but the one given here will work. (Not a problem with $0 but could be in other applications.)
  • Either approach will fail if the path to the file doesn’t actually exist. (Not a problem with $0, but could be in other applications.)
  • The unset is essential if your user may have CDPATH set.
  • Unlike readlink -f or realpath, this will work on non-Linux versions of Unix (e.g., Mac OS X).
DIR=$(cd `dirname "${BASH_SOURCE[0]}"` && pwd)/

Using ${BASH_SOURCE[0]} instead of $0 produces the same behaviour
regardless of whether the script is invoked as

name.sh or source name.sh

For more details about difference in linux between ‘source’ and ‘sh’

Enable Root User ( Super User ) in Ubuntu

Question: I’m unable to do su – on Ubuntu. It says “su: Authentication failure”. How do I fix it? Also, is it possible for me to login to Ubuntu using root account directly?

Answer: Let us address these two question one by one.

Warning: Enabling root is not recommended. If possible, you should always try to perform all administrative tasks using sudo.

Question 1: I’m unable to login using su command. How to fix this?

By default, root account password is locked in Ubuntu. So, when you do su -, you’ll get Authentication failure error message as shown below.

$ su -
Password:
su: Authentication failure

Enable super user account password on Ubuntu

First, set a password for root user as shown below.

$ sudo passwd root
[sudo] password for ramesh:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Now with the new password you can login as super user with su command

$ su -
Password:
#

Disable super user account password on Ubuntu

Later if you don’t want to use su anymore, you can lock the root user password using one of the methods shown below

$ sudo passwd -l root

( or )

$ sudo usermod -p '!' root

Step 2: In the Security tab, click on the check box “Allow local system administrator” as shown below. After this change, reboot the system and login directly using root account.