This article provides a basic routine to read in large text files in Perl. It is not intended to be "authoritative" but rather to demonstrate the my preferred method of reading files in Perl.
File I/O and Perl
There are many books and articles on the subject of reading and writing to text
files in Perl. Some methods are "better" than others...or at least better
suited for certain situations.
One of the most common reasons for reading a file is because you need to search for and retrieve data. The first method I learned was to read each line of the file into an array element and then search through the array.
# Open file.
open( FH, 'data.txt') or die( 'INPUT ERROR: Error opening $in_file\n' );
# Read file into an array.
@data = <FH>;
# Close file.
close( FH );
# Print file contents.
foreach $line ( @data )
{
print $line;
}
It sounded pretty straight forward but I soon found when using this method on large text files (100 MB+), I encountered a bottle-neck. Before any processing could be performed, the entire file had to be read into memory. On machines that do not have much free memory available, this method is far from ideal. After experimenting with a few different methods, I found this method, by far the fastest method.
# Open file.
$in_file = 'data.txt';
open( in_file ) or die( 'INPUT ERROR: Error opening $in_file\n' );
# Print file contents.
while( defined( $line = <in_file> ))
{
print $line;
}
close( $in_file );
In cases where the text file is small or the script is running on a powerful machine, the difference may not be overly noticeable but the one time you have a large text to process, you will certainly be able to appreciate the difference.