Surviving Linux

Author: Kasper B. Graversen, 21/02/08 Keywords: Linux, Unix, Help Abstract: This article describes "how to survive as a web developer in a Linux environment". The focus is not to give a general introduction to linux or its massive selection of programs, rather to focus on what is interesting from a developers perspective.

subscribe to my RSS feed

Surviving Linux

Tasks and services

• Scheduled execution of tasks are controlled by cron to get an overview and configuration type crontab -e sThis automatically restarts the crontab service.
• To edit the list of scheduled tasks /var/spool/cron/root and run service crond restart or use crontab -e, which upon saving (press ":" and type "wq") does not require a restart
  

  *    *    *    *    *  command to be executed
  |    |    |    |    |
  |    |    |    |    +----- day of week (0 - 6) (Sunday=0)
  |    |    |    +------- month (1 - 12)
  |    |    +--------- day of month (1 - 31)
  |    +----------- hour (0 - 23)
  +------------- min (0 - 59)
  

  Each argument can have different forms

  * * * * *      - Run every minute
  */2 * * * *    - Run every second minute
  10,20 * * * *  - Run every hour at 10 and 20 past
  

  It is easy to make mistakes, but a rule of thumb is that 99% of the cases the first two arguments should not be * so always be aware of that. Also be aware that you cannot specify a year so tasks running quaterly but on different dates each year (many businesses have different quaters each year) this cannot be specified.
• See all running tasks by you ps see all running tasks in the system ps ax or see them as process/subprocesses ps axf
• See if tomcat 5 has stopped ps -a | grep jsvc

Ressource overview

• Disk usage:

  # df -h
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/sda1             186G   19G  158G  11% /
  varrun                503M  188K  502M   1% /var/run
  varlock               503M  4.0K  503M   1% /var/lock
  

• Find all files greater than 10mb find / -size +10000k -printf %p\\t%s\\n
• sar shows various system ressources throughout the last many hours.

  # sar
  06:00:01 AM       CPU     %user     %nice   %system   %iowait     %idle
  06:10:01 AM       all     28.27      1.52      1.78      2.12     66.31
  06:20:01 AM       all     10.53      1.45      1.69      2.12     84.20
  06:30:01 AM       all      3.18      1.46      1.54      2.08     91.74
  

  # sar -r
  06:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
  06:10:01 AM      7840   7997628     99.90     74712    628688    461520    590728     56.14      7880
  06:20:01 AM      8008   7997460     99.90     72632    657424    461520    590728     56.14      7880
  06:30:01 AM      7056   7998412     99.91     70232    668528    461520    590728     56.14      7880
  

  A high memory usage (%memused and %swpused columns) as shown here indicate a need for a RAM upgrade.
• free -m shows the current memory status. Supposedly, this is much more accurate than top

  # free -m
      total used   free  shared buffers cached
  Mem: 3959  3948   11    0      58      1984
  -/+ buffers/cache: 1905 2053
  Swap: 1027 261    766
  

  The top row 'used' (3948) value will almost always nearly match the top row total mem value (3959) since Linux likes to use any spare memory to cache disk blocks (cached 1984). The key figure to look at is the buffers/cache row used value (1905). This is how much space your applications are currently using. For best performance, this number should be less than your total (3959) memory.
• watch [-n seconds] [-d] executes a program periodically, showing output, e.g. watch -n 1 -d free -m
• ps shows you where all your memory is going

  # ps aux
  USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
  apache 475 0.0 0.1 21400 5360 ? S 10:30 0:00 /usr/sbin/httpd
  apache 498 0.0 0.1 21380 5368 ? S 10:31 0:00 /usr/sbin/httpd
  

  Using the f flag shows subprocesses of a process in the list which often is very useful.
• top shows a live feed of the system processes, their memory and cpu usage

  top - 19:20:47 up 67 days, 22:12,  2 users,  load average: 0.24, 0.09, 0.02
  Tasks: 111 total,   2 running, 109 sleeping,   0 stopped,   0 zombie
  Cpu(s):  0.7% us,  0.3% sy,  0.0% ni, 99.0% id,  0.0% wa,  0.0% hi,  0.0% si
  Mem:   1028224k total,   609100k used,   419124k free,   158040k buffers
  Swap:  1494004k total,    56976k used,  1437028k free,   269648k cached
  
    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
   5230 donald    15   0  6408 1312  876 S  0.3  0.1   0:00.74 watch
   5656 matthew   16   0 10568 1280  948 R  0.3  0.1   0:00.02 top
      1 root      16   0  2636  440  412 S  0.0  0.0   0:00.84 init
      2 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0
      3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
  

  • Use f to pick columns
  • Use F to choose sort order
  • Use c to display path to process
  Where SWAP denotes the amount of swapped out memory. This not only memory moved to the swap but also for example memory-mapped files, which makes this value rathe useless.
• List all services on the box ls /etc/init.d
• vmstat helps you to see, among other things, if your server is swapping. If it's swapping you may be running towards trouble

  # vmstat 1 2
  procs memory swap io system cpu
  r b swpd free buff cache si so bi bo in cs us sy id wa
  1 0 267516 12548 59888 2003012 0 0 0 1 1 1 1 1 0 1
  3 0 267516 12344 59888 2002788 0 0 20 2316 701 5649 57 7 29 6
  

  where the most useful columns probably are
  • r - the number of processes waiting for acccess to the cpu
  • si — the amount of memory swapped in from disk
  • so — the amount of memory swapped out to disk
  • wa — I/O waiting
• iostat displays an overview of cpu utilization combined with I/O statistics the disk drives.
• mpstat shows multiprocessor information about the cpu and I/O stats

  # mpsatt -P ALL
  07:49:45 PM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
  07:49:45 PM  all   19.25    0.85    2.06    3.08    0.04    0.46   74.25   2753.67
  07:49:45 PM    0   16.15    0.58    2.48    1.20    0.15    1.24   78.20   1688.68
  07:49:45 PM    1   17.96    0.96    2.24    4.80    0.01    0.23   73.81    504.21
  07:49:45 PM    2   23.27    0.86    1.58    1.81    0.01    0.15   72.33     75.97
  07:49:45 PM    3   19.62    1.02    1.94    4.53    0.00    0.22   72.67    484.81
  

In BASH shell

for i in *; do echo $i; done

for i in `find . -type f`; do echo $i; done

for i in `find . -name "*.htm"`; do echo $i; done

for i in `find . -name "*.htm"`; do sed 's/\/99\//\/memorabilia\//' $i > $i; done

for i in `find . -name "*.htm"`; do sed 's/\/99\//\/memorabilia\//' $i > ${i}_tmp && mv -f ${i}_tmp ${i}; done

Process operations

Many applications maintain a so-called lock file, an empty file that denotes the program is already running. This prevents several instances of the same program to run simultaniously. Consequently, you may need to delete the lock file. Often it is found in /var/lock/ or /var/lock/subsys/

File operations

• List a directory with readable file sizes ls -hal [directory]
• ln -s sourcepath linkname adds a symbolic link called path+linkname pointing to the sourcepath.

  I have found it advantegous to create soft links for various ressources such as tomcat, java etc. This enables you to easily up- or downgrade a part of your software simply by changing the softlinks. Further, when all your scripts / cron jobs refer to the softlinks, no changes are needed when changing up/down-grading. E.g. ln -s /opt/ibm/java2-x86_64-50/ /var/java

  I have had bad experiences with sharing .jar files and other ressources unless the software sharing these ressources are exactly the same.
• Find all html files containing links to "www.foo" find . -type f -iname '*.htm*' -exec grep -l -i "a href=\"http://www.foo" {} \;
• scp source destination enables you to copy files between servers. To copy /opt/foo from current server to another server scp -r /opt/foo/ root@server:destination
• bzip one or more files tar -cjf destination.tar.bz2 [files..] untar/unzip them using tar -xjf foo.tar.gz
• Mass-edit files in bash: for f in `find . -name config.xml`; do vim $f; done
• sed 's/searchpat/replacetext/' file > newfile replaces occurences of 'searchpat' with the text 'replacetext' and redirects the output to 'newfile'
• mac2unix or dos2unix Converts a file from MAC or PC
• iconv Changes encoding of a file
• Mail program: mutt press SHIFT-D to deletes mails according to a pattern
• Send attachments: mutt -a /root/tmp/FRmonthly.csv Trudi.Winchester@famousretail.com < /dev/null

Preferences

• Edit personal settings edit ~.bash_profile
• /etc/DIR_COLORS holds definitions of colours for e.g. dir listings
• Create a new command alias

Misc

• See currently logged in users users. See history of lastest logging in users users /var/log/wtmp
• See the lastest invoked commands

  # history
  534  sudo rm -rf *
  535  ll
  536  cd ..
  

Net

• ifconfig shows the IP addresses of the computer much like
• iptables is the inbuild firewall in Linux (read by the kernel). To see the configuration iptables -L and the config is residing in /etc/sysconfig/iptables.
  An often used trick involving testing a configuration of iptables is to set a process to switch the firewall off after a set number of seconds in case of problems. This is particularly important if the computer is not physically reachable.

  ( sleep 30 ; /etc/init.d/iptables stop ) & /etc/init.d/iptables start 

  If you lose access the firewall will be automatically stopped after 30 seconds, if it all works you can kill the sleeping job using:

  kill %1
  

• To see your external ip address (e.g. if behind a router or other stuff), visit www.whatsmyip.com

Comments

Help spread the word