This article is all about how to read files in bash scripts using a while loop. Reading a file is a common operation in programming. You should be familiar with different methods and which method is more efficient to use. In bash, a single task can be achieved in many ways but there is always an optimal way to get the task done and we should follow it.
Before seeing how to read file contents using while loop, a quick primer on how while loop works. While loop evaluates a condition and iterates over a given set of codes when the condition is true.
while [ CONDITION ]
do
code block
done
Let’s break down while loop syntax.
- while loop should start with a while keyword followed by a condition.
- A condition should be enclosed within [ ] or [[ ]]. The condition should always return true for the loop to be executed.
- The actual block of code will be placed between do and done.
NUMBER=0
while [[ $NUMBER -le 10 ]]
do
echo " Welcome ${NUMBER} times "
(( NUMBER++ ))
done
This is a very simple example, where the loop executes until NUMBER is not greater than 10 and prints the echo statement.
Along with while we will use the read command to read the contents of a file line by line. Below is the syntax of how while and read commands are combined. Now there are different ways to pass the file as input and we will see them all.
# SYNTAX
while read VARIABLE
do
code
done
Piping in Linux
Normally we will use the cat command to view the contents of the file from the terminal. Also, we will pipe the output of the cat command to other commands like grep, sort, etc.
Similarly, we will use the cat command here to read the content of the file and pipe it to a while loop. For demonstration, I am using /etc/passwd file but it is not advisable to mess with this file so take a backup copy of this file and play with it if you desire so.
cat /etc/passwd | while read LREAD
do
echo ${LREAD}
done
Let’s break down what will happen when the above code is submitted.
- cat /etc/passwd will read the contents of the file and pass it as input through the pipe.
- read command reads each line passed as input from cat command and stores it in the LREAD variable.
- read command will read file contents until EOL is interpreted.
You can also use other commands like head, tail, and pipe it to while loop.
head -n 5 /etc/passwd | while read LREAD
do
echo ${LREAD}
done
Input Redirection in Linux
We can redirect the content of the file to while loop using the Input redirection operator (<).
while read LREAD
do
echo ${LREAD}
done < /etc/passwd | head -n 5
You can also store the file name to a variable and pass it through a redirection operator.
FILENAME="/etc/passwd"
while read LREAD
do
echo ${LREAD}
done < ${FILENAME}
You can also pass file names as an argument to your script.
while read LREAD
do
echo ${LREAD}
done < $1 | head -n 5
Internal Field Separator
You may work with different types of file formats (CSV, TXT, JSON) and you may want to split the contents of the file based on a custom delimiter. In this case, you can use “Internal field separator (IFS)” to split the content of the file and store it in variables.
Let me demonstrate how it works. Take a look at the /etc/passwd file which has a colon (:) as the delimiter. You can now split each word from a line and store it in a separate variable.
In the below example, I am splitting /etc/passwd file with a colon as my separator and storing each split into different variables.
while IFS=":" read A B C D E F G
do
echo ${A}
echo ${B}
echo ${C}
echo ${D}
echo ${E}
echo ${F}
echo ${G}
done < /etc/passwd
I displayed just one line split in the above screenshot considering screenshot size.
Empty Lines in Linux
Empty lines are not ignored when you loop through the file content. To demonstrate this I have created a sample file with the below content. There are 4 lines and few empty lines, leading whitespace, trailing white space, tab characters in line 2, and some escape characters (\n and \t).
while read LREAD
do
echo ${LREAD}
done < testfile
See the result, blank line is not ignored. Also, an interesting thing to note is how white spaces are trimmed by the read command. A simple way to ignore blank lines when reading the file content is to use the test operator with the -z flag which checks if the string length is zero. Now let’s repeat the same example but this time with a test operator.
while read LREAD
do
if [[ ! -z $LREAD ]]
then
echo ${LREAD}
fi
done < testfile
Now from the output, you can see empty lines are ignored.
Escape Characters
Escape characters like \n, \t, \c will not be printed when reading a file. To demonstrate this I am using the same sample file which has few escape characters.
while read LREAD
do
echo ${LREAD}
done < testfile
You can see from the output escape characters have lost their meaning and only n and t are printed instead of \n and \t. You can use -r to prevent backslash interpretation.
while read -r LREAD
do
echo ${LREAD}
done < testfile
That’s it for this article. We would love to hear back from you if there are any feedbacks or tips. Your feedback is what helps us to create better content. Keep reading and keep supporting.