The Wisdom in Log Fies
by Vikas Kamat
First Online: April 15, 2001
Page Last Updated: October 31, 2016
Every web hosting plan (even the
inexpensive plans) provides what are known as server logs that contain
information on the user activity on your website. They can be a single
file or separate files depending on the server
configuration , but
primarily contain three types of information: The Error Log,
The Access Log, and the The Referer Log.
This error_log contains all the errors that were generated
while processing users' request. Typical examples are missing files,
CGI or script errors. As a webmaster, you should always watch this
file, for signs of trouble in the website. Familiarize yourself with
the HTTP error messages to diagnose the problems.
See how people come to Kamat's Potpourri. and
find out what they are searching for. You can implement a similar
system to study how surfers come to your site.
What is the favicon.ico?
Many times, in your error log, you find an entry of the
[error] File does not exist: /usr/local/etc/httpd/htdocs/favicon.ico
favicon.ico is a 16x16
Windows® icon, the browser looks for so it
can display it in the address toolbar of the browser. (If you are
currently using Microsoft Internet Explorer (IE), it is the small graphic
displayed to the left of URL)
If you have numerous entries for a file favicon.ico in the error
log, it is a good thing! It shows how many people using IE 4.0 and
above have book marked your site. To remove reporting of this error,
place an icon with that name in your root folder.
Most of the problems we encounter at Kamat's Potpourri are people
linking to invalid URLs. Sometimes they make typographical mistakes,
but by watching this log, you know of those mistakes and could provide
aliases to the correct URLs.
This large file keeps track of all the resources
(images, scripts, pages) that were accessed on the website. It also
contains the time, and the location (IP Address) of the user.
What Pages are Popular?
By sorting the access log by number of hits you can find out which
contents are popular among your readers. You will be surprised that
they will also be your best sections; so highlight them!
Who are Your Patrons?
Sophisticated software systems available today (some are listed
below), can mine the access log and tell you demographics
information about your patrons. For example, if you had a website on
medicine, the way a medical student browses your site is different
than a practicing doctor. Of course, the access log does not tell
you the names of the patrons, so you can assure the patrons on
protection of their privacy.
What Days or Times are Busy?
Along with every http request is sent a small piece
of information called the HTTP_REFERER, which tells the name and
location of the page that has a link to your website. This important
information can be effectively used to find out the
effectiveness of Search engine listings, advertisement campaigns, or
simply to find our how users come to your site (see our example, Adhik
This is important so you can perform maintenance tasks. For
example, we have determined that our usage was highest in the beginning
part of the week, and hence we publish most contents for that
audience. We perform updates and indexing when it is late at night in
the USA, when our usage is the lowest.
How Do Users Find Your Site?
The Referer Log answers this important question, so you can optimize
the site for
Build a Smarter Site
By carefully studying the search patterns, you can find out
interesting information on general web usage. Here are some
discoveries from our own logs:
- More and more people are searching for multiple words now,
rather than just one word which was prevalent in earlier days of
- Yahoo! and Google are leading search engines and by a huge
margin. So, you can spend energy on getting a better rank on
- People make typing mistakes. By addressing the mistakes the
users often make (for example, many Europeans spell Gandhi
as Ghandi), you can get more traffic.
So do find out from your hosting company where the server
log files are. They contain wisdom about your site that no one else can
tell you about!