Tag Archives: cron

Add cron jobs to delete logs periodically

From Wikipedia:

cron is the time-based job scheduler in Unix-like computer operating systems. cron enables users to schedule jobs (commands or shell scripts) to run periodically at certain times or dates. It is commonly used to automate system maintenance or administration.

The final cron job script is here https://gist.github.com/c487d6d8b2b52219a87b

sudo crontab –e
2 0 * * * /opt/tools/delete_old_logs.sh /var/log >> /var/log/cron_job.log 2>&1

Crontab format

The basic format for a crontab is:

minute hour day_of_month month day_of_week command [args]
  • 1: minute (0-59)
  • 2: hour (0-23)
  • 3: day_of_month (0-31)
  • 4: month (0-12 [12 == December])
  • 5: day_of_week Day of the week(0-7 [7 or 0 == sunday])
  • /path/to/command – Script or command name to schedule

Easy to remember format:

* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

Multiple times may be specified with a comma(,), a range can be given with a hyphen(-), and the asterisk symbol(*) is a wildcard character. Spaces are used to separate fields. For example, the line:

*/5 9-16 * 1-5,9-12 1-5 ~/bin/i_love_cron.sh

Will execute the script i_love_cron.sh at five minute intervals from 9 AM to 4:55 PM on weekdays except during the summer months (June, July, and August). More examples and advanced configuration techniques can be found below.

Basic commands for cron management

Crontabs should never be edited directly; instead, users should use the [crontab][1] program to work with their crontabs. To be granted access to this command, user must be a member of the users group (see the gpasswd command).

To edit their crontabs, they may use:

$ crontab -e

Note: By default the crontab command uses the vi editor. To change it, export EDITOR or VISUAL, or specify the editor directly: EDITOR=vim crontab -e.

To view their crontabs, users should issue the command:

$ crontab -l

To remove their crontabs, they should use:

$ crontab -r

Remove or delete single cron job using command

crontab -l | grep -v '/var/crontab/xxx.sh' | crontab -

crontab -l lists the current crontab jobs

grep -v filter some line

crontab - adds all the printed stuff into the crontab file.

Related Articles

Use cron job to cleanup log files

Linux system various kinds logs and tmp generated in /var/log/, /tmp, How to clean these files automatically?

Using tmpwatch to automate temporary file cleanup

first we need install the 3rd tool tmpwatch

yum install tmpwatch -y

once tmpwatch is installed run command

/usr/sbin/tmpwatch -am 12 /tmp

this will delete all files over 12 hours old

next, we will configure your server to do this automatically.

from SSH type: crontab -e

go to the very bottom and paste

0 4 * * * /usr/sbin/tmpwatch -am 12 /var/log

For more daily job script:

$ cat /etc/cron.daily/tmpwatch

flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
    -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix 240 /tmp
/usr/sbin/tmpwatch "$flags" 720 /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
  if [ -d "$d" ]; then
    /usr/sbin/tmpwatch "$flags" -f 720 "$d"
  fi
done

-x is an entry to be excluded from the clean up operation.


Using a shell script do the same thing if none tmpwatch

find /var/log -type f -name "*.tmp" -exec rm {} \+

Normally we can execute as find /path -name "*.tmp" -exec rm {} \;
This may sometimes fail to work because the argument list may grow larger (in bytes) than the maximum allowed by the shell (getconf ARG_MAX). This may be solved by xargs with the -L option.

Also configure as a cron job to run automatically.

find /var/log -type f -mtime +12 -print0 | xargs -0 -L 5000 rm

Reference Links:

Rotate Nginx log files

Nginx is a great web server, however a default install will not rotate log files for you. there’s no size limit, it will keep getting bigger until your disk is full.

this is a problem especially on busy sites, as the access log can eat up disc space quite quickly.

In this tutorial, I will show you how to rotate ngnix log files automatically, my version is nginx/1.4.3. but any modern distribution should function in a similar way.

Manual rotating nginx log files via Cron

First we need to create the job bash script for cron that will do the log rotation.

sudo vi /usr/local/sbin/rotate_nginx_log.sh

Here are the contents of the script (this is based off the example from the Nginx wiki):

#!/bin/bash
# <https://gist.github.com/allex/10360845>

# Set variable
logs_path="/var/log/nginx/"
old_logs_path=${logs_path}/old
nginx_pid=`cat /var/run/nginx.pid`

time_stamp=`date -d "yesterday" +%Y-%m-%d`

mkdir -p ${old_logs_path}

# Main
for file in `ls $logs_path | grep log$ | grep -v '^20'`
do
    if [ ! -f ${old_logs_path}/${time_stamp}_$file ]
    then
        dst_file="${old_logs_path}/${time_stamp}_$file"
    else
        dst_file="${old_logs_path}/${time_stamp}_$file.$$"
    fi
    mv $logs_path/$file $dst_file
    gzip -f $dst_file  # do something with access.log.0
done

kill -USR1 $nginx_pid

Note:

First, we move the current log to a new file for archiving. A common scheme is to name the most recent log file with a suffix of current time stamp. e.g, $(date "+%Y-%m-%d").

The command that actually rotates the logs is kill -USR1 $(cat /var/run/nginx.pid). This does not kill the Nginx process, but instead sends it a SIGUSR1 signal causing it to re-open its logs.

THen execute sleep 1 to allow the process to complete the transfer. We can then zip the old files or do whatever post-rotation processes we would like.

Next please make sure that the script file is executable by running

chmod +x /usr/local/sbin/rotate_nginx_log.sh

In our final step we will create a crontab file to run the script we just created.

sudo crontab -e

In this file let’s create a cron job to run every day at 1am

Add the following lines to the file as follows:

00 01 * * * /usr/local/sbin/rotate_nginx_log.sh &> /dev/null

Also we can config suppressing cron jobs status email notifications. see Suppressing Cron Job Email Notifications


Log Rotation With Logrotate

The logrotate application is a simple program to rotate logs.

sudo vim /etc/logrotate.d/nginx

Put this content inside and modify the first line to match your Nginx log file

/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    dateformat .%Y-%m-%d
    compress
    delaycompress
    notifempty
    create 640 nginx adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] &amp;&amp; kill -USR1 `cat /var/run/nginx.pid`
    endscript
}

Wait 24 hours until cron daily runs and check out if you see any .gz file inside your logs directory, if you see some gzipped files, your Nginx rotation is working fine :D


Related References: