See the New Growth Report

Report by Matthew Gray

Measuring the Growth of the Web

June 1993 to June 1995

Goals and History
This report documents my attempts to realistically measure how fast the Web is growing, specifically in terms of growth in the number of Web servers through the use of an automated Web agent, the Wanderer.

The World Wide Web was started originally proposed in 1989 and the first implementation appeared in 1990. The Web however, did not gain any widespread popular use until NCSA Mosaic became available in early 1993.

In Spring of 1993, I wrote the Wanderer to systematically traverse the Web and collect sites. I was initially motivated primarily to discover new sites, as the Web was still a relatively small place. As the Web grew rapidly, the focus quickly changed to charting the growth of the Web. This report covers the period from June 1993 to June 1995.

The primary tool used to collect the data presented here was the World Wide Web Wanderer, the first automated Web agent or "spider". The Wanderer was first functional in spring of 1993 and performed regular traversals of the Web from June 1993 to June 1995.

Other valuable data sources include data from ZONE, a DNS walker and from the Merit Computer Network Backbone Statistics.

The Wanderer was run on a monthly basis from June 1993 to June 1995. While the Wanderer certainly did not reach every site, it was run with consistent methodology, hopefully yielding consistent data for the growth of the Web.

The Internet in general has grown quite fast. Any look at the growth of the Web necessarily examines the growth of the Internet at large. Doing so, particularly by looking at the data collected by ZONE and the Merit Computer Network, makes it all the clearer how fast the Web has grown, even compared to the explosive growth of the Internet in general.

The Web has grown fast. How fast? For the second half of 1993, the Web had a doubling period of under 3 months, and even today the doubling period is still nearly 5 months. Additionally, the percentage of Web sites that are commercial has increased dramatically. The numbers below show the percentage growth in the .com domain, which excludes foreign commercial sites (such as the domain, etc.).

The growth of the Web has been remarkable even compared to the Internet at large, as shown by the number of hosts per Web server. In June of 1995, even with the phenomenal growth of the Internet, the number of Web servers soared to a point where one in every 270 machines on the Internet is a Web server.

Results Summary
Month # of Web sites % .com sites Hosts per Web server
6/93 130 1.513,000
12/93 623 4.63,475
6/94 2,738 13.51,095
12/94 10,022 18.3451
6/95 23,500 31.3270
1/96 100,000 50.094

Copyright 1995 Matthew Gray of MIT
All Rights Reserved excepted by these stipulations.