The Wisdom in Log Fies

by Vikas Kamat
First Online: April 15, 2001
Page Last Updated: February 17, 2024

Every web hosting plan (even the inexpensive plans) provides what are known as server logs that contain information on the user activity on your website. They can be a single file or separate files depending on the server configuration , but primarily contain three types of information: The Error Log, The Access Log, and the The Referer Log.

The Nosy Surfer See how people come to Kamat's Potpourri. and find out what they are searching for. You can implement a similar system to study  how surfers come to your site.

The Error Log

This error_log contains all the errors that were generated while processing users' request. Typical examples are missing files, CGI or script errors. As a webmaster, you should always watch this file, for signs of trouble in the website. Familiarize yourself with the HTTP error messages to diagnose the problems.

What is the favicon.ico?
Many times, in your error log,  you find an entry of the type:
[error] File does not exist: /usr/local/etc/httpd/htdocs/favicon.ico
favicon.ico is a 16x16 Windows® icon, the browser looks for so it can display it in the address toolbar of the browser. (If you are currently using Microsoft Internet Explorer (IE), it is the small graphic displayed  to the left of URL)
If you have numerous entries for a file favicon.ico in the error log, it is a good thing! It shows how many people using IE 4.0 and above have book marked your site. To remove reporting of this error, place an icon with that name in your root folder.

Most of the problems we encounter at Kamat's Potpourri are people linking to invalid URLs. Sometimes they make typographical mistakes, but by watching this log, you know of those mistakes and could provide aliases to the correct URLs.

The Access Log

This large file keeps track of all the resources (images, scripts, pages) that were accessed on the website. It also contains the time, and the location (IP Address) of the user.

What Pages are Popular?
By sorting the access log by number of hits you can find out which contents are popular among your readers. You will be surprised that they will also be your best sections; so highlight them!

Who are Your Patrons?
Sophisticated software systems available today (some are listed below), can mine the access log and tell you demographics information about your patrons. For example, if you had a website on medicine, the way a medical student browses your site is different than a practicing doctor. Of course, the access log does not tell you the names of the patrons, so you can assure the  patrons on protection of their privacy.

What Days or Times are Busy?
This is important so you can perform maintenance tasks. For example, we have determined that our usage was highest in the beginning part of the week, and hence we publish most contents for that audience. We perform updates and indexing when it is late at night in the USA, when our usage is the lowest.

The Referer Log

Along with every http request is sent a small piece of information called the HTTP_REFERER, which tells the name and location of the page that has a link to your website. This important information can be effectively used  to find out the effectiveness of Search engine listings, advertisement campaigns, or simply to find our how users come to your site (see our example, Adhik Prasangi).

How Do Users Find Your Site?
The Referer Log answers this important question, so you can optimize the site for

Build a Smarter Site
By carefully studying the search patterns, you can find out interesting information on general web usage. Here are some discoveries from our own logs:

• More and more people are searching for multiple words now, rather than just one word which was prevalent in earlier days of the web.
• Yahoo! and Google are leading search engines and by a huge margin. So, you can spend energy on getting a better rank on those.
• People make typing mistakes. By addressing the mistakes the users often make (for example, many Europeans spell Gandhi as Ghandi), you can get more traffic.

So do find out from your hosting company where the server log files are. They contain wisdom about your site that no one else can tell you about!

Internet Links

• Webtrends Log Analyzer -- an out of the box commercial solution 

• WebLog -- an inexpensive comprehensive access log analysis tool