Pages

Friday, May 14, 2010

How to Read a File Line by Line in a Shell Script


There are many ways to handle any task on a Unix platform, but some
techniques that are used to process a file waste a lot of CPU time.
Most of the wasted time is spent in unnecessary variable assignment and
continuously opening and closing the same file over and over. Using a
pipe also has a negative impact on the timing.

In this article I will explain various techniques for parsing a file
line by line. Some techniques are very fast and some make you wait for
half a day. The techniques used in this article are measurable, and I
tested each technique with time command so that you can see which tec-
hniques suits your needs.

I don't explain in depth every thing, but if you know basic shell
scripting, I hope you can understand easily.

I extracted last five lines from my /etc/passwd file, and stored in a
file "file_passwd".

[root@www blog]# tail -5 /etc/passwd > file_passwd
[root@www blog]# cat file_passwd
venu:x:500:500:venu madhav:/home/venu:/bin/bash
padmin:x:501:501:Project Admin:/home/project:/bin/bash
king:x:502:503:king:/home/project:/bin/bash
user1:x:503:501::/home/project/:/bin/bash
user2:x:504:501::/home/project/:/bin/bash

I use this file whenever a sample file required.

Method 1: PIPED while-read loop


#!/bin/bash
# SCRIPT: method1.sh
# PURPOSE: Process a file line by line with PIPED while-read loop.

FILENAME=$1
count=0
cat $FILENAME | while read LINE
do
let count++
echo "$count $LINE"
done

echo -e "\nTotal $count Lines read"

With catting a file and piping the file output to a while read loop a
single line of text is read into a variable named LINE on each loop
iteration. This continuous loop will run until all of the lines in the
file have been processed one at a time.

Bash can sometimes start a subshell in a PIPED "while-read" loop. So
the variable set within the loop will be lost (unset) outside of the
loop. Therefore, $count would return 0, the initialized value outside
the loop.

Output:

[root@www blog]# sh method1.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 0 Lines read

Method 2: Redirected "while-read" loop


#!/bin/bash
#SCRIPT: method2.sh
#PURPOSE: Process a file line by line with redirected while-read loop.

FILENAME=$1
count=0

while read LINE
do
let count++
echo "$count $LINE"

done < $FILENAME

echo -e "\nTotal $count Lines read"

We still use the while read LINE syntax, but this time we feed the
loop from the bottom (using file redirection) instead of using a pipe.
You will find that this is one of the fastest ways to process each
line of a file. The first time you see this it looks a little unusual,
but it works very well.

Unlike method 1, with method 2 you will get total number of lines out
side of the loop.

Output:

[root@www blog]# sh method2.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 Lines read

Note: In some older shell scripting languages, the redirected loop
would also return as a subshell.

Method 3:while read LINE Using File Descriptors


A file descriptor is simply a number that the operating system assigns
to an open file to keep track of it. Consider it a simplified version
of a file pointer. It is analogous to a file handle in C.

There are always three default "files" open, stdin (the keyboard),
stdout (the screen), and stderr (error messages output to the screen).
These, and any other open files, can be redirected. Redirection simply
means capturing output from a file, command, program, script, or even
code block within a script and sending it as input to another file,
command, program, or script.

Each open file gets assigned a file descriptor. The file descriptors
for stdin,stdout, and stderr are 0,1, and 2, respectively. For opening
additional files, there remain descriptors 3 to 9 (may be vary depend-
ing on OS). It is sometimes useful to assign one of these additional
file descriptors to stdin, stdout, or stderr as a temporary duplicate
link. This simplifies restoration to normal after complex redirection
and reshuffling .

There are two steps in the method we are going to use. The first step
is to close file descriptor 0 by redirecting everything to our new file
descriptor 3. We use the following syntax for this step:

exec 3<&0

Now all of the keyboard and mouse input is going to our new file des-
criptor 3. The second step is to send our input file, specified by the
variable $FILENAME, into file descriptor 0 (zero), which is standard
input. This second step is done using the following syntax:

exec 0<$FILENAME

At this point any command requiring input will receive the input from
the $FILENAME file. Now is a good time for an example.

#!/bin/bash
#SCRIPT: method3.sh
#PURPOSE: Process a file line by line with while read LINE Using
#File Descriptors

FILENAME=$1
count0=
exec 3<&0
exec 0< $FILENAME

while read LINE
do
let count++
echo "$count $LINE"
done

exec 0<&3
echo -e "\nTotal $count Lines read"

while loop reads one line of text at a time.But the beginning of this
script does a little file descriptor redirection. The first exec comm-
and redirects stdin to file descriptor 3. The second exec command red-
irects the $FILENAME file into stdin, which is file descriptor 0. Now
the while loop can just execute without our having to worry about how
we assign a line of text to the LINE variable. When the while loop
exits we redirect the previously reassigned stdin, which was sent to
file descriptor 3, back to its original file descriptor 0.

exec 0<&3

In other words we set it back to the system’s default value.

Output:

[root@www tempdir]# sh method3.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 Lines read

Method 4: Process file line by line using awk


awk is pattern scanning and text processing language. It is useful
for manipulation of data files, text retrieval and processing. Good
for manipulating and/or extracting fields (columns) in structured
text files.

Its name comes from the surnames of its authors: Alfred Aho, Peter
Weinberger, and Brian Kernighan.

I am not going to explain everything here.To know more about awk just
Google it.

At the command line, enter the following command:

$ awk '{ print }' /etc/passwd

You should see the contents of your /etc/passwd file appear before
your eyes.Now, for an explanation of what awk did. When we called awk,
we specified /etc/passwd as our input file. When we executed awk, it
evaluated the print command for each line in /etc/passwd, in order.All
output is sent to stdout, and we get a result identical to catting
/etc/passwd. Now, for an explanation of the { print } code block. In
awk, curly braces are used to group blocks of code together, similar
to C. Inside our block of code,we have a single print command. In awk,
when a print command appears by itself, the full contents of the curr-
ent line are printed.

Here is another awk example that does exactly the same thing:

$ awk '{ print $0 }' /etc/passwd

In awk, the $0 variable represents the entire current line, so print
and print $0 do exactly the same thing. Now is a good time for an
example.

#!/bin/bash
#SCRIPT: method4.sh
#PURPOSE: Process a file line by line with awk

FILENAME=$1

awk '{kount++;print kount, $0}
END{print "\nTotal " kount " lines read"}' $FILENAME


Output:

[root@www blog]# sh method4.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 lines read

Awk is really good at handling text that has been broken into multiple
logical fields, and allows you to effortlessly reference each individ-
ual field from inside your awk script. The following script will print
out a list of all user accounts on your system:

awk -F":" '{ print $1 "\t " $3 }' /etc/passwd

Above, when we called awk, we use the -F option to specify ":" as the
field separator. By default white space (blank line) act as filed sep-
arator. You can set new filed separator with -F option. When awk proc-
esses the print $1 "\t " $3 command, it will print out the first and
third fields that appears on each line in the input file. "\t" is used
to separate field with tab.

Method 5: Little tricky with head and tail
commands


#!/bin/bash
#SCRIPT: method5.sh
#PURPOSE: Process a file line by line with head and tail commands

FILENAME=$1
Lines=`wc -l < $FILENAME`

count=0

while [ $count -lt $Lines ]
do
let count++
LINE=`head -n $count $FILENAME | tail -1`
echo "$count $LINE"
done
echo -e "\nTotal $count lines read"

On each iteration head command extracts top $count lines, then tail
command extracts bottom line from that lines. A very stupid method,
but some people still using it.

Output:

[root@www blog]# sh method5.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 lines read


Time Comparison for the Five Methods


Now take a long breath, we are going test each technique. Before you
get into test each method of parsing a file line by line create a large
file that has the exact number of lines that you want to process.
Use bigfile.sh script to create a large file.

$ sh bigfile.sh 900000

bigfile.sh with 900000 lines as an argument,it has taken more than two
hours to generate bigfile.4227. I don't know exactly how much time it
has taken. This file is extremely large to parse a file line by line,
but I needed a large file to get the timing data greater than zero.

[root@www blog]# du -h bigfile.4227
70M bigfile.4227
[root@www blog]# wc -l bigfile.4227
900000 bigfile.4227


[root@www blog]# time ./method1.sh bigfile.4227 >/dev/null

real 6m2.911s
user 2m58.207s
sys 2m58.811s
[root@www blog]# time ./method2.sh bigfile.4227 > /dev/null

real 2m48.394s
user 2m39.714s
sys 0m8.089s
[root@www blog]# time ./method3.sh bigfile.4227 > /dev/null

real 2m48.218s
user 2m39.322s
sys 0m8.161s
[root@www blog]# time ./method4.sh bigfile.4227 > /dev/null

real 0m2.054s
user 0m1.924s
sys 0m0.120s
[root@www blog]# time ./method5.sh bigfile.4227 > /dev/null
I waited more than half day, still i didn't get result, then I created
a 10000-line file to test this method.
[root@www tempdir]# time ./method5.sh file.10000 > /dev/null

real 2m25.739s
user 0m21.857s
sys 1m12.705s

Method 4 came in first place,it has taken very less time 2.05 seconds,
but we can't compare Method 4 with other methods, because awk is not
just a command, but a programming language too.

Method 2 and method 3 are tied for second place, they produce mostly
the same real execution time at 2 minutes and 48 seconds . Method 1
came in third at 6 minutes and 2.9 seconds.

Method 5 has taken more than half a day. 2 minutes 25 seconds to pro-
cess just a 10000 line file, how stupid it is.

Note: If file contain escape characters, use read -r instead of read,
then Backslash does not act as an escape character. The back-slash is
considered to be part of the line. In particular, a backslash-newline
pair may not be used as a line continuation.

36 comments:

  1. If i did any mistake, even typo also please inform me.

    ReplyDelete
  2. Thanks for this post and this blog..It is really helpful.

    ReplyDelete
  3. Really good your study. Just let me give a hint about the comparison you made: the first method should take longer because the file is not in the buffers (RAM, HD, processor, etc.), so I advise you to do all the 5 tests and then return to the first and test the time again, this way the times should be more accurate.

    ReplyDelete
  4. Bash can sometimes start a subshell in a PIPED "while-read" loop, That's why Method1 taken more time.

    Check bellow output, I changed the order of execution. Still I get almost same result.

    [root@localhost procesfile]# time ./method2.sh bigfile.4227 > /dev/null

    real 2m46.832s
    user 2m37.890s
    sys 0m8.429s

    [root@localhost procesfile]# time ./method3.sh bigfile.4227 > /dev/null

    real 2m47.224s
    user 2m38.370s
    sys 0m8.369s

    [root@localhost procesfile]# time ./method1.sh bigfile.4227 > /dev/null

    real 6m10.158s
    user 3m2.095s
    sys 2m59.299s

    ReplyDelete
  5. how to read line by line by using system calls

    ReplyDelete
  6. Thnx Dude... Great Article..It helped me alot. :)

    ReplyDelete
  7. Simple Tutorial.

    Easy to understand.


    thanks
    salih

    ReplyDelete
  8. When the input to the loop comes from a process rather than from a file, things become even trickier.
    What I do in that case is to use a temporary file, but to speed things up I use a FIFO instead of a normal file.
    This uses the GNU coreutils package, which is pretty widespread and should work on any POSIX system (but I don't know about filesystem limitations).
    A non-complete example:

    # make up a temporary file name in system temp directory:
    TempFile="$(mktemp -t "$(basename "${0%.sh}")-${UID}-$$.XXXXXX")"

    # recreate that file as a FIFO instead:
    rm -f "$TempFile"
    mkfifo "$TempFile"

    # fill it with the output from MyCommand process (including error stream), in background:
    MyCommand >& "$TempFile" &

    # now process the input as above:
    while read -r LINE ; do
    # my process
    done < "$TempFile"

    # clean up
    rm -f "$TempFile"


    (note that I am writing this in the blog and it could have typos and mistakes -- in fact, most likely it has)
    It has always worked forme, but pay attention to both portability (try to ask FAT for a FIFO...) and for synchronization (the effectiveness /might/ depend on MyCommand output speed).

    ReplyDelete
  9. Great article mate, you have indeed covered topic in detail with examples. I have also blogged my experience as 10 examples of grep command in unix ,let me know how do you find it.

    Thanks
    Top 30 Unix Interview Questions

    ReplyDelete
  10. Very nice article that I found helpful. Thank you!

    ReplyDelete
  11. great Venu.. realy great. I like awk.

    ReplyDelete
  12. Hi!
    I am doing a programm which will read line by line command from a file text. I am doing so:
    cat $p1 | ( while read line;
    do
    ----execute----
    done )

    But i want that the programm doing a pause for press any key. I am doing so:
    cat $p1 | ( while read line;
    do
    ----execute----
    read -p "Press any key"
    done )

    But it dont work...
    Thanks

    ReplyDelete
  13. not sure what UNIX variant you're using, but in HPUX, the 'read' command has only 1 flag (-r, to not treat backslash char's as special) and 1 parameter, a variable name. For example, you would write the above like this:

    echo "Press any key"
    read mydata

    where mydata is the variable that will contain the character you enter from the keyboard.

    Give that a try...

    ReplyDelete
  14. NICE ;)
    Thanks for the effort

    ReplyDelete
  15. Thank you very much for your effort.
    Nice explanation and easy to understand.

    ReplyDelete
  16. Nice article bro, very useful one, keep them coming.

    Thank you.

    ReplyDelete
  17. Good 1...G8 effort

    ReplyDelete
  18. Method #5 is the only one that worked in my script. What I do is read file line-by-line, extract hostname, and do SSH to that host. For some reason, "while ; do ; done < $filename" only works till the first SSH, then the while loop exits.

    ReplyDelete
  19. Thanks for the post. Very informative and useful.

    ReplyDelete
  20. Very nice work indeed. Thank you!

    Amit Verma / IND

    ReplyDelete
  21. Can you give me an example for read/write by using bash?

    ReplyDelete
  22. thank u so much sir u done a great job

    ReplyDelete
  23. Hi All...
    I'm working on something where requirement is something like:
    1. One Shell Script - abc.sh
    2. One Text file - xyz.txt
    ===========================================
    xyz.txt contains values as below:
    [1st Section]
    a
    b
    c
    d

    [2nd Section]
    e
    f
    g
    h
    i

    [3rd Section]
    j
    k
    l
    m
    n
    o
    p

    ========================================

    Now I want print the values as per section vise using Menu driven script.
    whenevenr I person will execute abc.sh then it will ask a person which section
    values you want to see:
    ======
    Example:
    1. 1st Section
    2. 2nd Section
    3. 3rd Section
    Choose an option:
    (say) person has choosen value 2

    then it should print only
    e
    f
    g
    h
    i
    ======

    so I'm stuck @ how can I read a file from some particular line and till some particular point.

    ReplyDelete
    Replies
    1. n=$1

      case $n in
      1) n=1st ;;
      2) n=2nd ;;
      3) n=3rd ;;
      esac

      awk -v n="$n" '$1 ~ n,/^$/ {
      if ( $0 == "" || $1 ~ n ) next
      print
      }' "$file"

      Delete
  24. This comment has been removed by a blog administrator.

    ReplyDelete
  25. Hi,

    I have a site, that i need to monitor and read the content of site
    site ex : www.example.com and it has three component like below

    component1-true
    component2-true
    component3-true

    need to check three component "true" and get mails if not "true"

    i ams using curl command to read out but has some issues,

    can some one suggest any process.

    mail me on mpasha06@gmail.com

    ReplyDelete
  26. Hi!

    I am working on a script which should parse an input.txt file which contains more table definitions like this one:

    ------------------------------------------------
    -- DDL Statements for table "CMWSYS"."CMWD_TEC_SUIVI_TRT"
    ------------------------------------------------


    CREATE TABLE "CMWSYS"."CMWD_TEC_SUIVI_TRT" (
    "ORDER_ID" VARCHAR(20) ,
    "PERIODE_DATE" VARCHAR(50) ,
    "NOM_JOB" VARCHAR(100) ,
    "D_START" TIMESTAMP ,
    "D_END" TIMESTAMP ,
    "T_ELAPSED" TIME ,
    "STATUS" VARCHAR(50) )
    IN "CMW_DA16DN01" INDEX IN "CMW_IX16DN01" ;

    and then it should export to a file called dbname_tablename (in this case CMWSYS_CMWD_TEC_SUIVI_TRT) one by one all the table definitions.

    Do you have any idea how I can manage this task? At least a small hint.
    Thanks

    ReplyDelete
  27. Can anyone tell me about two statement in shell script
    exec 3<&0
    exec 0< $FILENAME # FILENAME=$1 is first argument as file name
    ?

    Please help..!

    ReplyDelete
  28. how to take input in variable form a file word by word??

    ReplyDelete