• Welcome to Bashguru

    Linux is one of popular version of UNIX operating System. It is open source as its source code is freely available. It is free to use. Linux was designed considering UNIX compatibility. It's functionality list is quite similar to that of UNIX and become very popular over the last several years. Our Basic motive is to provide latest information about Linux Operating system.

  • Python Programming

    Python is a comparatively simple programming language, compared to c++. Although some of the benefits of c++ are abstracted away in python, they are replaced with an overall easier to learn language with many “intuitive” features. For this reason it is common and recommended by most professionals that people new to programming start with python.

  • Perl Programming

    Perl is an open-source, general-purpose interpreted programming language. Used often for CGI, Perl is also used for graphics programming, system administration, network programming, finance, bioinformatics, and other applications. The Perl languages borrow features from other programming languages including C, shell scripting (sh), AWK, and sed. They provide powerful text processing facilities without the arbitrary data-length limits of many contemporary UNIX command line tools, facilitating easy manipulation of text files.

  • Android

    Android is an operating system based on the Linux kernel, and designed primarily for touch screen mobile devices such as smart phones and tablet computers. Android is a Linux-based software system, and similar to Linux, is free and open source software. This means that other companies can use the Android operating developed by Google and use it in their mobile devices.Android gives you a world-class platform for creating apps and games for Android users everywhere, as well as an open marketplace for distributing to them instantly.

Friday, May 14, 2010

Posted by venu k
54 comments | 11:45 PM

 There are many ways to handle any task on a Unix platform,  but some
techniques that are used to process a file waste a lot of CPU time.
Most of the wasted time is spent in unnecessary variable assignment and
continuously opening and closing the same file over and over. Using a
pipe also has a negative impact on the timing.

 In this article I will explain various techniques for parsing a file
line by line. Some techniques are very fast and some make you wait for
half a day. The techniques used in this article are measurable, and I
tested each technique with time command so that you can see which tec-
hniques suits your needs.

 I don't explain in depth every thing, but if you know basic shell
scripting, I hope you can understand easily.

 I extracted last five lines from my /etc/passwd file, and stored in a
file "file_passwd".

[root@www blog]# tail -5 /etc/passwd > file_passwd
[root@www blog]# cat file_passwd
venu:x:500:500:venu madhav:/home/venu:/bin/bash
padmin:x:501:501:Project Admin:/home/project:/bin/bash
king:x:502:503:king:/home/project:/bin/bash
user1:x:503:501::/home/project/:/bin/bash
user2:x:504:501::/home/project/:/bin/bash
I use this file whenever a sample file required.

Method 1: PIPED while-read loop

#!/bin/bash
# SCRIPT: method1.sh
# PURPOSE: Process a file line by line with PIPED while-read loop.
FILENAME=$1
count=0
cat $FILENAME | while read LINE
do
let count++
echo "$count $LINE"
done
echo -e "\nTotal $count Lines read"
With catting a file and piping the file output to a while read loop a single line of text is read into a variable named LINE on each loop iteration. This continuous loop will run until all of the lines in the file have been processed one at a time. Bash can sometimes start a subshell in a PIPED "while-read" loop. So the variable set within the loop will be lost (unset) outside of the loop. Therefore, $count would return 0, the initialized value outside the loop. Output:
[root@www blog]# sh method1.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash
Total 0 Lines read

Method 2: Redirected "while-read" loop

#!/bin/bash
#SCRIPT: method2.sh
#PURPOSE: Process a file line by line with redirected while-read loop.
FILENAME=$1
count=0
while read LINE
do
let count++
echo "$count $LINE"
done < $FILENAME
echo -e "\nTotal $count Lines read"
We still use the while read LINE syntax, but this time we feed the loop from the bottom (using file redirection) instead of using a pipe. You will find that this is one of the fastest ways to process each line of a file. The first time you see this it looks a little unusual, but it works very well. Unlike method 1, with method 2 you will get total number of lines out side of the loop. Output:
[root@www blog]# sh method2.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash
Total 5 Lines read
Note: In some older shell scripting languages, the redirected loop would also return as a subshell.

Method 3:while read LINE Using File Descriptors

A file descriptor is simply a number that the operating system assigns to an open file to keep track of it. Consider it a simplified version of a file pointer. It is analogous to a file handle in C. There are always three default "files" open, stdin (the keyboard), stdout (the screen), and stderr (error messages output to the screen). These, and any other open files, can be redirected. Redirection simply means capturing output from a file, command, program, script, or even code block within a script and sending it as input to another file, command, program, or script. Each open file gets assigned a file descriptor. The file descriptors for stdin,stdout, and stderr are 0,1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9 (may be vary depend- ing on OS). It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link. This simplifies restoration to normal after complex redirection and reshuffling . There are two steps in the method we are going to use. The first step is to close file descriptor 0 by redirecting everything to our new file descriptor 3. We use the following syntax for this step: exec 3<&0 Now all of the keyboard and mouse input is going to our new file des- criptor 3. The second step is to send our input file, specified by the variable $FILENAME, into file descriptor 0 (zero), which is standard input. This second step is done using the following syntax: exec 0<$FILENAME At this point any command requiring input will receive the input from the $FILENAME file. Now is a good time for an example.
#!/bin/bash
#SCRIPT: method3.sh
#PURPOSE: Process a file line by line with while read LINE Using
#File Descriptors
FILENAME=$1
count0=
exec 3<&0
exec 0< $FILENAME
while read LINE
do
let count++
echo "$count $LINE"
done
exec 0<&3
echo -e "\nTotal $count Lines read"
while loop reads one line of text at a time.But the beginning of this script does a little file descriptor redirection. The first exec comm- and redirects stdin to file descriptor 3. The second exec command red- irects the $FILENAME file into stdin, which is file descriptor 0. Now the while loop can just execute without our having to worry about how we assign a line of text to the LINE variable. When the while loop exits we redirect the previously reassigned stdin, which was sent to file descriptor 3, back to its original file descriptor 0. exec 0<&3 In other words we set it back to the system’s default value. Output: [root@www tempdir]# sh method3.sh file_passwd 1 venu:x:500:500:venu madhav:/home/venu:/bin/bash 2 padmin:x:501:501:Project Admin:/home/project:/bin/bash 3 king:x:502:503:king:/home/project:/bin/bash 4 user1:x:503:501::/home/project/:/bin/bash 5 user2:x:504:501::/home/project/:/bin/bash Total 5 Lines read

Method 4: Process file line by line using awk

awk is pattern scanning and text processing language. It is useful for manipulation of data files, text retrieval and processing. Good for manipulating and/or extracting fields (columns) in structured text files. Its name comes from the surnames of its authors: Alfred Aho, Peter Weinberger, and Brian Kernighan. I am not going to explain everything here.To know more about awk just Google it. At the command line, enter the following command: $ awk '{ print }' /etc/passwd You should see the contents of your /etc/passwd file appear before your eyes.Now, for an explanation of what awk did. When we called awk, we specified /etc/passwd as our input file. When we executed awk, it evaluated the print command for each line in /etc/passwd, in order.All output is sent to stdout, and we get a result identical to catting /etc/passwd. Now, for an explanation of the { print } code block. In awk, curly braces are used to group blocks of code together, similar to C. Inside our block of code,we have a single print command. In awk, when a print command appears by itself, the full contents of the curr- ent line are printed. Here is another awk example that does exactly the same thing: $ awk '{ print $0 }' /etc/passwd In awk, the $0 variable represents the entire current line, so print and print $0 do exactly the same thing. Now is a good time for an example.
#!/bin/bash
#SCRIPT: method4.sh
#PURPOSE: Process a file line by line with awk
FILENAME=$1
awk '{kount++;print kount, $0}
END{print "\nTotal " kount " lines read"}' $FILENAME
Output:
[root@www blog]# sh method4.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash
Total 5 lines read
Awk is really good at handling text that has been broken into multiple logical fields, and allows you to effortlessly reference each individ- ual field from inside your awk script. The following script will print out a list of all user accounts on your system: awk -F":" '{ print $1 "\t " $3 }' /etc/passwd Above, when we called awk, we use the -F option to specify ":" as the field separator. By default white space (blank line) act as filed sep- arator. You can set new filed separator with -F option. When awk proc- esses the print $1 "\t " $3 command, it will print out the first and third fields that appears on each line in the input file. "\t" is used to separate field with tab.

Method 5: Little tricky with head and tail commands

#!/bin/bash
#SCRIPT: method5.sh
#PURPOSE: Process a file line by line with head and tail commands
FILENAME=$1
Lines=`wc -l < $FILENAME`
count=0
while [ $count -lt $Lines ]
do
let count++
LINE=`head -n $count $FILENAME | tail -1`
echo "$count $LINE"
done
echo -e "\nTotal $count lines read"
On each iteration head command extracts top $count lines, then tail command extracts bottom line from that lines. A very stupid method, but some people still using it. Output:
[root@www blog]# sh method5.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash
Total 5 lines read

Time Comparison for the Five Methods

Now take a long breath, we are going test each technique. Before you get into test each method of parsing a file line by line create a large file that has the exact number of lines that you want to process. Use bigfile.sh script to create a large file. $ sh bigfile.sh 900000 bigfile.sh with 900000 lines as an argument,it has taken more than two hours to generate bigfile.4227. I don't know exactly how much time it has taken. This file is extremely large to parse a file line by line, but I needed a large file to get the timing data greater than zero.
[root@www blog]# du -h bigfile.4227
70M bigfile.4227
[root@www blog]# wc -l bigfile.4227
900000 bigfile.4227
[root@www blog]# time ./method1.sh bigfile.4227 >/dev/null
real 6m2.911s
user 2m58.207s
sys 2m58.811s
[root@www blog]# time ./method2.sh bigfile.4227 > /dev/null
real 2m48.394s
user 2m39.714s
sys 0m8.089s
[root@www blog]# time ./method3.sh bigfile.4227 > /dev/null
real 2m48.218s
user 2m39.322s
sys 0m8.161s
[root@www blog]# time ./method4.sh bigfile.4227 > /dev/null
real 0m2.054s
user 0m1.924s
sys 0m0.120s
[root@www blog]# time ./method5.sh bigfile.4227 > /dev/null
I waited more than half day, still i didn't get result, then I created
a 10000-line file to test this method.
[root@www tempdir]# time ./method5.sh file.10000 > /dev/null
real 2m25.739s
user 0m21.857s
sys 1m12.705s
Method 4 came in first place,it has taken very less time 2.05 seconds, but we can't compare Method 4 with other methods, because awk is not just a command, but a programming language too. Method 2 and method 3 are tied for second place, they produce mostly the same real execution time at 2 minutes and 48 seconds . Method 1 came in third at 6 minutes and 2.9 seconds. Method 5 has taken more than half a day. 2 minutes 25 seconds to pro- cess just a 10000 line file, how stupid it is.
Note: If file contain escape characters, use read -r instead of read,
then Backslash does not act as an escape character. The back-slash is
considered to be part of the line. In particular, a backslash-newline
pair may not be used as a line continuation.