• Good bye 2.6

    Another proud moment for linux lovers , kernel 3 released !!!

    Linux successfullly completed 2 decades and entering to the 3rd decade with 3rd generation kernel.




    Refer the official details from : Linus Torvalds  and  Kernel Org

    You can view the 20 years story here : The Story

    Good bye 2.6  !!!!!!

  • Difference between Ext2,Ext3 and Ext4 File Systems

    Overview :-

    1) Difference between file systems
    2) How to create file system
    3) How to convert a file system to other
    4) How to find the file system type of your server
    5) I/O performance comparison using chart.

    (Warning :-  Never try the filesystem conversion commands in your live or production servers )

     

    Ext2

    • stands for second extended file system and it was introduced by Remy card in 1993.
    • Maximum individual file size can be from 16 GB to 2 TB and overall ext2 file system size can be from 2 TB to 32 TB

    How to create an ext2 filesystem
    # mke2fs /dev/sda1

     

    Ext3

    • stands for third extended file system and it was introduced  by Stephen Tweedie in 2001
    • The main difference between ext2 and ext2 is , ext3 will allow journaling . (journaling is a type of log file , which will track all the file system changes)
    • Maximum individual file size can be from 16 GB to 2 TB and overall ext3 file system size can be from 2 TB to 32 TB
    • We can convert ext2 to ext3 directly

    How to create ext3 file system :-
    # mkfs.ext3 /dev/sda1
    (or)
    # mke2fs –j /dev/sda1
    ( -j for adding journaling capability )

    How to  convert  ext2 to ext3 :-
    # umount /dev/sda2
    # tune2fs -j /dev/sda2
    # mount /dev/sda2  /var

     

    Ext4

    • stands for fourth extended file system and it was introduced in 2008 with Linux Kernel 2.6.19 .
    • Ext4 will support huge individual file systems also it can handle huge overall file system.
    • Maximum individual file size can be from 16 GB to 16 TB
    • Overall maximum ext4 file system size is 1 EB (exabyte). 1 EB = 1024 PB (petabyte). 1 PB = 1024 TB (terabyte).
    • In ext3 the number of sub directories that a directory can contain is limited to 32,000. This limit has been raised to 64,000 in ext4,
    • Extents replace the traditional block mapping scheme used by ext2/3 filesystems. An extent is a range of contiguous physical blocks, improving large file performance and reducing fragmentation.
    • In ext4, unallocated block groups and sections of the inode table are marked as such. This enables e2fsck to skip them entirely on a check and greatly reduces the time it takes to check a file system . This feature is implemented in version 2.6.24 of the Linux kernel.

    Creating ext4 file system :-
    # mkfs.ext4 /dev/sda1
    (or)
    # mke2fs -t ext4 /dev/sda1

    Converting ext3 to ext4
    ( Warning :- Never try this live or production servers )
    # umount /dev/sda2
    # tune2fs -O extents,uninit_bg,dir_index  /dev/sda2
    # e2fsck -pf /dev/sda2
    # mount /dev/sda2 /var

     

    Find your servers filesystem type
    We can find the filesystem type used in our servers using any one of the following commands
    # mount
    /dev/sda3 on / type ext3 (rw)
    proc on /proc type proc (rw)
    /dev/sda1 on /boot type ext3 (rw)
    tmpfs on /dev/shm type tmpfs (rw)

    # file -sL /dev/sda1
    /dev/sda1: Linux rev 1.0 ext3 filesystem data (needs journal recovery)

    # df -T | awk ‘{print $1,$2,$7}’ | grep “^/dev”
    /dev/sda3 ext3 /
    /dev/sda1 ext3 /boot

     

    Read/Write Performance chart of  ext2,ext3 and ext4.

    Reference : http://www.delltechcenter.com/page/A+Comparison+of+Ext2,+Ext3+and+Ext4+Performance

  • Difference between lilo and grub

    Difference between lilo and grub

    Lilo is a boot strap program and its stands for Linux loader.  lilo knows how to locate kernel and load that kernel. configuration file is /etc/lilo.conf
    If you never need to make changes at boot time, then LILO will work just fine for you.

    GRUB stands for GRand Unified Bootloader and is a GNU Bootloader that can boot a variety of operating systems from Linux, , DOS, NT 3.51 etc. Its config file is usually in /boot/grub and might be called grub.conf or menu.lst. When you change the config file, you merely reboot to read the changes. The structure of the config file is very different than that of LILO, even though they each convey essentially the same information. When you boot with GRUB, you have a vast array of options available to you. If you ever find that you need to make changes at boot time, usually you will find GRUB more useful.

    Difference at a glance :-

    * LILO has no interactive command interface, whereas GRUB does.

    * LILO does not support booting from a network, whereas GRUB do.
    * LILO only loads linux and other boot loaders. and GRUB loads a large number of OS
    * LILO works by loading itself into a space that will fit on the MBR. Grub has two stages

    example grub config :-

    # cat /boot/grub/grub.conf
    # grub.conf generated by anaconda
    #
    # Note that you do not have to rerun grub after making changes to this file
    # NOTICE:  You have a /boot partition.  This means that
    #          all kernel and initrd paths are relative to /boot/, eg.
    #          root (hd0,0)
    #          kernel /vmlinuz-version ro root=/dev/sda3
    #          initrd /initrd-version.img
    #boot=/dev/sda
    default=0
    timeout=5
    splashimage=(hd0,0)/grub/splash.xpm.gz
    hiddenmenu
    title CentOS (2.6.18-194.32.1.el5.centos.plus)
    root (hd0,0)
    kernel /vmlinuz-2.6.18-194.32.1.el5.centos.plus ro root=LABEL=/
    initrd /initrd-2.6.18-194.32.1.el5.centos.plus.img

    title windows
    rootnoverify (hd0,4)
    chainloader(hd0,0)+1

    >>> default 0
    Default will tell which kernel should load , zero stands for the first one. ( The first menu entry title linux is the one to boot by default.)

    >>> timeout 5
    After five seconds without any user input, GRUB automatically boots the default entry. To deactivate automatic boot, delete the timeout line. If you set timeout 0, GRUB boots the default entry immediately.

    >>> title
    Is responsible for booting Linux. The kernel (vmlinuz) is located in the first logical partition (the boot partition) of the first hard disk.  The root partition is specified according to the Linux naming convention (/dev/sda3/), because this information is read by the kernel and has nothing to do with GRUB. The initrd is also located in the first logical partition of the first hard disk.

    >>> rootnoverify
    rootnoverify command is for OS filesystems not specifically recognized by GRUB, so that GRUB will not try to mount the partition. Next, the chainloader command will use the first sector of the partition of device (hd0,0) and attempt to boot whatever it finds there. This is a common means of booting OSes that install their own boot loaders in the first sector of the partition where they are installed

  • Difference between insmod and modprobe

    Both insmod and modprobe is using to insert kernel module.
    But everyone will prefer modprobe because insmod have no capability to resolve dependency issue . But modprobe can do that.

    While we are installing a module using modprobe , first it will check the dependancies in /lib/modules/<kernel-version>/modules.dep. This file contains entries for a module and its corresponding dependencies .  modules.dep file is generated by “depmode” command.

    modprobe in action :-

    List available Kernel modules
    # modprobe -l | more

    List Currently loaded modules
    # lsmod | more

    Install new module
    # modprobe <module name >
    example :-  modprobe ip_tables

    Remove a module
    # modprobe -r <module name>

    we can use ” rmmod ” , also to delete a module. But admins prefer modprobe with -r option , because modprobe is cleverer than insmod and rmmod.


  • Create Amazon Elastic Compute Cloud (Amazon EC2)

    Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.

    You can read more about ec2 here : Adminlogs-amazonEC2

    I would like to document how I have created an instance in amazon and successfully logged in to shell. Amazon ec2 will not allow password authentication , it will support only key authentication.

    1) Login to your amazon account ( or create a new account )

    2) Create an EC2 instance

    Once you have logged in using your account , you can see a well organized admin panel . From here you can create ,edit ,monitor and delete your instances

    From admin panel , Select EC2 and appropriate region you wish to host your instance . Once you selected these two things then you can press the button “launch instance ”

    3) Select appropriate OS for your instance( Choose an AMI )

    Here I have selected 32 bit CentOs from the Community AMI’s

    4) Select the type of instance ( Ram and CPU )

    I have selected the small instance with 1.7Gb ram and 1 cpu.  You can select any one of the following spec.

    Next step is to setup the Advanced Instance options, you can keep the default settings as it is.  Also in this step you can specify whether you want to use amazon detailed monitoring.

    5) Add tags to your Instance

    6) Create a keypair

    As I have told before amazon will not allow password authentication. You can give a specific name for your key.  And there is an option to create and download your key file (.pem format ). You should keep this in a safe place. If you are creating multiple instances you can use the same keypair or generate new keys for each instance.

    7) Configure Firewall.

    In this steps you can configure access control for your server . For example allow port 22 only from your network , allow http port 80 to public etc.  Your settings will be saved as security rule. Just like keypair , you can use a default security group for your all the instances or you can write specific firewall rules .

    8 ) Review

    Your ec2 instance configuration is over, In this step you can review your configurations and make necessary changes if  you need.

    If you are ok with the current configurations then you can proceed with ” Launch ” Instance . This will create an instance for you with the given specifications. ( It will take 2-3 minutes to complete this )

    9) Manage an Instance

    Once the instance is created , then you can view the instance in the admin dash board. From here you can stop/start/reboot your instance just like a normal linux server.

    From the Instance tab you can view your newly created instance . This will give you basic details of your instance like instance name , id etc.

    If you want to know about more about your instance just click on the instance and this will display a detailed info about your instance just show above.

    In the above diagram , I have projected ” public dns ” , this is your server ip/hostname to connect remotely to the server. By default amazon will not give you a public IP . Next step

    I will describe how to setup a public ip for  your instance.

    10 ) Create additional volume and setup a new public IP

    By default for the “small instance ” type amazon will give only 10Gb disk space . We can add additional volumes and attach to the corresponding instance.  Also we can add public ip  to your instance using the tab ” Elastic IP’s ” . Note that if you restart , the public ip will be detached automatically . So each time you need to attach public ip after a reboot.  Also note that after each restart you will get a different public DNS name and private ip . Once you added/attached  a public ip and this ip will be stable .

    Now everything is fine with the EC2 configuration.

    11) Remote login to the newly created Instance.

    We need to create/generate a public and private key pair using the security key (.pem) generated through amazon admin panel ( Refer step 6 )

    To generate the key pairs , we can use the software putty gen . Load the amazon key to putty gen and save public and private keys in to a safe place.

    12 ) Configure putty to use private key authentication.

    Use the “public dns ” ( If you are purchased/added a elastic ip then you can use that ) as the hostname/server ip in putty and point the corresponding private key to putty as follows

    Once you setup the putty as above then you can connect to the server/instance using root.

    Here may be you have noticed /dev/sdb with 147Gb space. Actually its a free space given by amazon . But please note that whenever you restart the instance all the data’s in this drive will be lost.  If you want additional space other than /dev/sda1 , then you should create new volumes using the admin panel and attach to your instance.

    Yes , you have successfully created your own ec2 instance. Now you can work on the server just like a normal linux server. Enjoy !! 🙂

     

  • The Story of Linux: Commemorating 20 Years of the Linux Operating System

  • Process management Tips

    Its necessary for a system admin to know about the process states and its management .

    In Linux every activity is controlled by certain process. or processes are programs which are in execution.
    A process consists of resources such as open file, internal kernel data ,an address space and its executing program code etc.
    Each process will run with its own process id and all the running process are stored as files inside a virtual directory /proc

    Process states :-

    1. Running  : This is a state where a process is either in running or ready to run.
    2. Interruptible  : This state is a blocked state of a process which awaits for an event or a signal from another process
    3. Uninterruptible: It is also a blocked state. The process is forced to halt for certain condition that a hardware status is waited and a signal could not be handled.
    4. Stopped : Once the process is completed, this state occurs. This process can be restarted
    5. Zombie  : In this state, the process will be terminated and the information will still be available in the process table.

    Zombie is a process state when the child dies before the parent process. In this case the structural information of the process is still in the process table. Since this process is not alive, it cannot react to signals. Zombie state can finish when the parent dies. All resources of the zombie state process are cleared by the kernel

    System Calls used for Process management :-

    Fork()         :- Used to create a new process
    Exec()         :- Execute a new program
    Wait()          :- wait until the process finishes execution
    Exit()           :- Exit from the process
    Getpid()      :- get the unique process id of the process
    Getppid()   :- get the parent process unique id
    Nice()          :- to bias the existing property of process

    Commands used to manage process :-

    1) PS
    ps is used to list all the process which are presently running in the server

    # ps aux

    ( ps is the one command which is not using “-” with options )
    a : all process
    u : with username
    x : processes w/o controlling ttys
    A/ e : also will list all the process

    # ps aux | grep java
    the above command will list all the java process

    # ps aux –forest

    This command will output a process tree structure

    2) Pgrep

    pgrep looks through the currently running processes and lists the process IDs which matches the selection criteria

    # pgrep -u root sshd
    will only list the processes called sshd AND owned by root

    # pgrep java
    will all the java processes

    3) top
    The  top  program provides a dynamic real-time view of a running system.  It can display system summary information as well as a list of tasks currently being managed by the Linux
    kernel. top is just like a the ps command . Main difference between top and ps is , top will give dynamic results but ps will give a static output.

    # top -c
    May be this will be the command mostly used by a system admin in his/her servers. Option “c” will give a command mode interface also
    We can sort ” top ” results like memorywise process wise etc

    shift+ m :  will sort the result in memory wise
    shift+ p : will sort the result in cpu usage

    # top -c -u root
    will give the dynamic display of process which are owned by root user.

    Process Investigation

    Its always better to know about the details like from where its running and which are the files opened by the specific process etc of a process . This
    knowledge will help us to manage/control process running in our server.
    As mentioned earlier all the running process are stored inside a virtual directory /proc . Each process will be a folder inside /proc

    For example , from top I found one perl process is taking more memory resouce and its causing the server load issue. So i decided to investigate about this process.

    # cat /proc/<pid from top>/environ
    the above will give you a clear idea about the running process .
    Or the best and efficient command to get the process details is lsof
    # lsof -p  <pid from top> | more
    this is my favorite command 😉

    this will clearly mention from which location this file is running and which are the open files .

    Process Killing
    Usually we are using process killing stop some unwanted/resource using process.
    There are lots of ways to kill a running process

    # kill -9  <pid>
    Kill the process which having the specific pid
    # pkill <process name >
    kill/stop the process which are running under a specific service
    # fuser -k < process deamon >
    example : fuser -k /usr/local/apache/bin/httpd
    This will kill all the process which are running due to apache service.  You should restart the service once you have run the above.
    This command is very useful while we are facing high load issues on the server. Its very quick and efficient.

  • Linux Find command tips

    Find is a most important command for an admin for his/her daily system administration.

    Find command will search recursively search and display the results .

    I would like to share/document some of the find command examples that I am using in my day to day use.

    Find a file name

    # find . -name “admin.txt”
    Search for the file admin.txt in present working directory

    # find /usr -name “admin.txt”
    this command will search for the exact name “admin.txt” in /usr

    # find /usr -iname “admin.txt”
    this will search for the name admin.txt and ignore the case.
    For example this will display Admin.txt and admin.txt

    Find a file owned by a specific user

    # find  /  -user  admin

    Find everything under /var ending with “.log”

    # find /var -name \*.log

    Find files with respect to the date of creation

    # find /backup -mtime +60

    Find files with specific permission and owenrship

    # find /home -perm 777  -user nobody

    Find a files with specified size

    # find /home -size 1024k

    Find files with zero size  ( empty files )

    # find /home -empty

    Find files with file type ( ” f  ” for files and ” d”  for directory types )

    # find /home -type f -name index
    # find /home -type d -name  public

    Find files specified with in the dept

    touch /passwd

    # find /  -maxdepth 3 -name password
    it will display the following ( root and two levels down )
    /usr/bin/passwd
    /passwd
    /etc/passwd
    /etc/pam.d/passwd

    Find and write

    a) Find and write the result to new file

    # find . -iname test -type f > /tmp.txt
    or
    # find . -iname test -type f -fprint /tmp.txt

    Find and remove

    # find /backup -mtime +4  -type d \( -iname “*-2011*” \) -exec rm -rf {} \;

    The above will find and delete for all the directories which are older than 4 days and its name contains the string “-2011”

    # find . -name ‘*.class’ | xargs /bin/rm -f

    This command will remove all the files which are having extenstion .class

    Find the 5 biggest files  in /home

    find /home -type f -exec ls -s {} \; | sort -n -r | head -5

    Find and zip the files

    find /var -name \*.log -exec bzip {} \;

    Find and change the ownership of files

    find /home/admin -user nobody -exec chown admin {} \;

    Above command will search for all the files which are under nobody user ownership and change the user ownership to admin.

    you can suggest your own and I am happy to include your suggestions to this doc

  • SVN branching ( Branching / Tagging )

    Scenario :- For our new project , today my boss asked me to setup a branch of the existing svn with a new name.

     

    One of the features of version control systems is the ability to isolate changes onto a separate line of development. This line is known as a branch. Branches are often used to try out new features without disturbing the main line of development with compiler errors and bugs. As soon as the new feature is stable enough then the development branch is merged back into the main branch (trunk).

     

    You can use the following doc to create a branch

     

    1) Right click on the source and select “ Svn branch /tag “ and this will give the following popup. Specify your new url and recent message .

     

     

     

     

     

     

     

     

     

     

     

    http://adminlogs.info/wp-content/uploads/2011/05/svn_branch.jpg

    That’s its , you created the new branch.

    Refer the following url to get more idea about branching :

    http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-branchtag.html#tsvn-dug-branch-1

     

    The above method is just like a link in unix , You can create a separate folder/svn url using the following svn commands

    svn copy file:///adminlogs.info/yourName/svn/yourProject/trunk file:///adminlogs.info/yourName/yourProject/NewBranches/newProject-1.0

    For the above command you need to create a new directory NewBranches first .

  • Could not copy file.: /public_html : wordpress upgrade

    Today was trying to upgrade my wordpress and it was failing with the following error

    Could not copy file.: /public_html

    As per the wordpress , it was due  the pure-ftpd server and the suggestion was to switch over to Proftpd.

    I have contacted my support and the reply was like ” it is not possible to change default configuration of ftp on shared server “

    So I have decided to dig in this issue and found a nice word press plugin , which will help you to fix the bug with pure-ftpd

    FTP Upgrade Fix

    I tried to install this plugin , but got the same error.  So i have removed the folder ” upgrade ”  (wp-content == > upgrade ) and recreate the same with 777 permision.

    This time i was able to install this plugin successfully .

    Yes, its a nice one . After that I did the upgrade with in seconds !!!!!

    Hope this will help you once you stuck with your wordpress upgrade 😉