Friday, September 7, 2012

Reading portion of a huge log file

If you have a several hundred MB log file, how do you quickly find a portion of it you're interested in? Your vi may not be able to open the file, complaining file too large, and/or take too long and use too much memory. Here's what you can do.

Let's say you want Mon Jul 15's data from July's access_log, here's the command
sed -n '/^Mon Jul 15/p; /^Tue Jul 16/q' access_log > /tmp/qq 
Without the quit command, sed would scan the file to the end or till you press ^C. Note the undocumented semicolon that allows you to put two sed commands on one line. (By the way, on Solaris creating a file under /tmp may be faster because tmpfs should be memory-based, unless you're short on RAM. But remember to delete big files under /tmp when you're done because they reduce available swap space.)