Author Archives: Allex Wang

About Allex Wang

focus on web front development, mobile, *unix.

PHP_FPM of unix sockets vs TCP ports

When setup PHP-FPM for nginx pass_proxy, we may setup the php-fpm.conf include one of the config below:

listen = /var/run/php5-fpm.sock –> [nginx.conf] fastcgi_pass unix:/var/run/php5-fpm.sock;

listen = 127.0.0.1:9000 –> [nginx.conf] fastcgi_pass 127.0.0.1:9000;

Performance of unix sockets vs TCP ports

When you are using TCP, you are also using the whole network stack. Even if you are on the same machine, this implies that packets are encapsulated and decapsulated to use the network stack and the related protocols.

If you use unix domain sockets, you will not be forced to go through all the network protocols that are required otherwise. The sockets are identified solely by the inodes on your hard drive.

Make PHP-FPM Listen at “IPAddress:Port” Instead of “/var/run/php5-fpm.sock;”

Sockets are slightly faster as compared to TCP/IP connection. But they are less scalable by default.

If you start getting errors like below

connect() to unix:/var/run/php5-fpm.sock failed or **apr_socket_recv: Connection reset by peer (104)**

Then it means you need to either switch to TCP/IP or tweak with linux-system parameter so that your OS can handle large number of connections.

So, for high-load cases this is what it’s supposed to be: listen = 127.0.0.1:9000 and that fixed everything!

Reference Links

Remove PHP X-Powered-By & Nginx Version

For some website’s security reason, we need remove X-Powered-By and NGINX Version from response headers.

PHP:

To remove X-Powered-By completely, search line in php.ini.

expose_php = Off

or add the following directive to the Nginx configuration:

# Prevent version info leakage
fastcgi_hide_header X-Powered-By;

Nginx:

To remove Server Version from Header, server_tokens should be disabled in nginx.conf.

# http://wiki.nginx.org/HttpCoreModule#server_tokens
server_tokens off;

Change server string by recompiling Nginx source:

vim +49 src/http/ngx_http_header_filter_module.c

Find the lines:

static char ngx_http_server_string[] = "Server: nginx" CRLF;
static char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF;

see also: Customize Your Nginx Server Name After Compiling From Source

Maven deploy and release

The “SNAPSHOT” term means that the build is a snapshot of your code at a given time.

It usually means that the version is a version still under heavy development.

When it comes time to release your code, you will want to change the version listed in the pom. So instead of having a “1.0.0-SNAPSHOT” you would have something like “1.0.0″.

If your project.version contains SNAPSHOT (f.e., 1.2-SNAPSHOT) and you execute mvn deploy, artifacts will be deployed to your snapshot repository. If it doesn’t (f.e., 1.2) – they will be deployed to your release repository.

Maven 项目管理的时候,日常发布版本,我们都是直接 mvn deploy 到一个 SNAPSHOT 版本,那如何发布 stable 版本呢?

本篇描述了,如何使用 mvn 命令对一个项目进行从 1.0.0-SNAPSHOT 版本升级到 1.0.0,同时在 scm 中进行打 tag 后,修改版本到 1.0.1-SNAPSHOT,同时 commit 到 branch。

#1. 添加 scm 和 plugin 定义

<scm>
  <developerConnection>scm:git:git@git.n.xiaomi.com:allex/yp-assist.git</developerConnection>
</scm>
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-release-plugin</artifactId>
  <version>2.5</version>
  <!--
  <configuration>
    <tagBase>https://code.miliao.com/repos/xiaomi-miliao/tags/src/cloud/yp-assist/</tagBase>
  </configuration>
  -->
</plugin>

为了保障 svn 目录清洁,建议在使用 svn 作为 scm 时,添加上面注释的段落,git 项目不用加

#2. release:prepare 准备工作

$ mvn release:prepare
  • 忽略 SNAPSHOT 依赖,强制发布,mvn release:prepare -DignoreSnapshots=true – 和 -DskipTest=true 一样令人讨厌,不建议使用
  • 此操作会向 git/svn 写入内容
  • 新手建议:-DdryRun=true 此命令加 dryRun 是演习,不会向 git/svn checkin 任何内容。-DdryRun=true 可用 mvn release:clean 回滚。
  • 会检查各个版本是否是 SNAPSHOT,会在运行中要求你回答使用什么版本。
  • 准备工作的作用:加版本 & checkin + 打 tag 到 git/svn 中

#3. 发包工作

$ mvn release:perform
  • 这一步,会去一个临时目录中,把 commit 的代码抓出来 build 之后 deploy。

#4. 清理工作

$ mvn release:clean

References Pages:

CentOS how to tips

How to remove RPM packages with several dependencies

If you are using fedora, simply use this simple script but be careful when answering y/N:

yum remove $(rpm -qa | grep PACKAGENAME)

  • Change PACKAGENAME with your Package name
  • For disabling plugins just add --disableplugin=PLUGIN-NAME
  • If you can’t access the Internet, just add this options to the line above --disablerepo=*

Find out what files are in my rpm package

Use following syntax to list the files for already INSTALLED package:

The –v (verbose) option can give you more information on the files when used with the various query options.

rpm -ql package-name

Use following syntax to list the files for RPM package:

rpm -qlp package-name

Type the following command to list the files for gitlab*.rpm package file:

rpm -qlp gitlab-7.1.1_omnibus-1.el6.x86_64.rpm

See also: HowTo: Extract an RPM Package Files Without Installing It

Update yum repositories for CentOS, RHEL Systems

Get the latest yum repos from one of the two links below, selecting to match your host’s architecture:

# CentOS/RHEL 6, 64 Bit (x86_64):
rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Then enjoy update with yum update yum-updatesd

Change CentOS language

vi /etc/sysconfig/i18n

check the lang is your expected, such as:

LANG="en_US.UTF-8"  <<-----
SUPPORTED="en_US.UTF-8:en_US:en" 
SYSFONT="latarcyrheb-sun16"

and re-login with you user/passwd, check it with command locale

Yum install/update with specific repository

# update git with rpmforge-extras repository
yum --disablerepo=base,updates --enablerepo=rpmforge-extras update git

vimdiff ignore white space

To ignore white spaces while using vimdiff.

set diffopt+=iwhite

From the command line use:

vimdiff -c 'set diffopt+=iwhite' ...

To have vimdiff ignore whitespace while normal vim doesn’t, simply put this into your .vimrc:

if &diff
    " diff mode"
    set diffopt+=iwhite
endif

Grub Booting the ISO

安装多个 Linux 版本时,通常情况下我们拿到的都是在 iso 文件, 那么如何用 grub 引导iso镜像呢?
So, 我们需要在GRUB引导菜单列表来添加一个启动项。

环境:Ubuntu 13.10 / x64

sudo vim /boot/grub/grub.cfg

添加 Grub 启动项:

menuentry "Ubuntu 13.10 ISO" {
    # set isofile="/home/<username>/Downloads/ubuntu-13.10-desktop-amd64.iso"
    # or set
    isofile="/allex/Downloads/ubuntu-13.10-desktop-amd64.iso"
    # if you use a single partition for your $HOME
    loopback loop (hd0,8)$isofile
    linux (loop)/casper/vmlinuz.efi boot=casper iso-scan/filename=$isofile noprompt noeject
    initrd (loop)/casper/initrd.lz
}

Note:

  1. 找到 iso 文件的所属分区路径。df -h 查看目录所在分区,修改到上面的 loopback loop (hd0,8)
  2. 同步自己 iso 镜像里的 casper/ 目录下面的内核引导文件名称,vmlinuz.*, initrd.lz
  3. /boot/grub/grub.cfg 文件是自己动生成的,为防止系统擦掉,可以把上面代码放在 /etc/grub.d/40_custom, 完了执行一下 update-grub
  4. Windows用户可参考 GRUB4DOS, doc

Reference Links:

After upgrade to OSX Mavericks

After upgrade to OSX Mavericks, and reinstall xcode, if make some project, You’ll get the error a bit more…

“Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.”

– What?

Ah! Ok, X-code was obviously re-installed with OSX Mavericks.

sudo xcodebuild -license

This allowed me to view the X-code licence, and then agree to the terms. Voila, that’s it. Everything worked just fine after that.

Linux daily skills (continuous updating)

Cleanup process list

Kill these process that zombie or stopped.

# kill zombie process list
ps -A -ostat,ppid | grep -e '[zZ]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

# cleanup stopped process list
ps -A -ostat,pid | grep -e '[T]' | tail -n +2 | awk '{ print $2 }' | xargs kill -9

Mandatory logged out user session

Tips: 当出现服务器用户数过多,造成别人登陆不上去,管理员可强行踢出用户

w list current logon sessions, and the kill it with pkill -kill -t [tty]

pkill -kill -t pts/2

Make Tab auto-completion case-insensitive

# If ~./inputrc doesn't exist yet, first include the original /etc/inputrc so we don't override it
if [ ! -a ~/.inputrc ]; then echo "\$include /etc/inputrc" > ~/.inputrc; fi

# Add option to ~/.inputrc to enable case-insensitive tab completion
echo "set completion-ignore-case On" >> ~/.inputrc

Note: to make this change for all users, edit /etc/inputrc

Get the networking connection statistics

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

TIME_WAIT 22
ESTABLISHED 3254   # data transfer state
LAST_ACK 236
FIN_WAIT_1 648
FIN_WAIT_2 581
CLOSING 7
CLOSE_WAIT 4916

Kill the process that bind the specific port

GistURL: https://gist.github.com/allex/8b399a93749703acd780

#!/bin/sh

port=$1
while [[ -z "${port}" ]]; do
    read -p "Please type the port you wanna kill (q to exit): " port
done
[ "${port}" = "q" ] && exit 0;

PID=""
case "`uname `" in
    Linux*)  PID=`lsof -t -i:${port}` ;;
    Darwin*) PID=`lsof -i -n -P | grep ":${port} (LISTEN)" | awk '{print $2}'` ;;
esac

if [ -n "$PID" ];
then
    kill -9 $PID && echo "procss with pid: ${PID} on port ${port} had killed success!"
else
    echo "The specific process with port '${port}' not found. exit";
fi

Usage: ./killport.sh [port]

Find IP address in shell

# OS X may not work except
ip = `hostname -I | cut -d' ' -f1`

# or use complex Linux shell
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`

Example:

#!/bin/sh
ip=`ifconfig | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p' | grep -v '127.0.' | head -n 1`
wget -q -O - http://${ip}:8092/yn/build.hash -H 'Host: st.comm.miui.com' | base64 -d

Compare differences between directories

cp -R $local $bak
rsync $server:$remdir/* $local/
rsync $local/ $server:$remdir/*
diff -wur $local $bak

Use cron job to cleanup log files

Linux system various kinds logs and tmp generated in /var/log/, /tmp, How to clean these files automatically?

Using tmpwatch to automate temporary file cleanup

first we need install the 3rd tool tmpwatch

yum install tmpwatch -y

once tmpwatch is installed run command

/usr/sbin/tmpwatch -am 12 /tmp

this will delete all files over 12 hours old

next, we will configure your server to do this automatically.

from SSH type: crontab -e

go to the very bottom and paste

0 4 * * * /usr/sbin/tmpwatch -am 12 /var/log

For more daily job script:

$ cat /etc/cron.daily/tmpwatch

flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
    -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix 240 /tmp
/usr/sbin/tmpwatch "$flags" 720 /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
  if [ -d "$d" ]; then
    /usr/sbin/tmpwatch "$flags" -f 720 "$d"
  fi
done

-x is an entry to be excluded from the clean up operation.


Using a shell script do the same thing if none tmpwatch

find /var/log -type f -name "*.tmp" -exec rm {} \+

Normally we can execute as find /path -name "*.tmp" -exec rm {} \;
This may sometimes fail to work because the argument list may grow larger (in bytes) than the maximum allowed by the shell (getconf ARG_MAX). This may be solved by xargs with the -L option.

Also configure as a cron job to run automatically.

find /var/log -type f -mtime +12 -print0 | xargs -0 -L 5000 rm

Reference Links: