Where it started…

The practice of Search Engine Optimisation (SEO) started in the late 1990s, when search engines such as Yahoo! and Google gained popularity for searching information on the internet.

As search engines became more sophisticated and the amount of content online grew, website owners realised they needed to optimise their sites to rank higher in search results and attract more traffic. Thus, the field of SEO was born.

A lot has changed since I started work in SEO, back in 2003. There have been ups and downs, twists and turns, and back then it was known as the ‘dark arts’ for a reason. SEO was forged in an underground community that most people had never heard of. People that worked (or dabbled) in the SEO industry ranged from the intrigued and curious, to the outright criminal. For all SEOs however, the term ‘ethical’ was an unknown concept. To understand why was the case, you need to understand the journey it’s taken.

This goes back almost as far as the World Wide Web itself, so the best place to start is August 1991.

1991: One website online

In 1991, the UK was gripped by a cultural phenomenon called ‘rave’. There were no mobile phones, no Internet, no social media, no Play Stations and no Xboxes. Video rental was booming and there were only four channels on TV. If we wanted to learn anything, we went to the library. If we wanted new music, we asked around the record shops or borrowed tape cassettes from friends. It was a time of complete naivety compared to today. We felt we were in the throes of a revolution that would change the world. But little did we know, far from the dingy warehouses and endless parties, on the other side of Europe, a real revolution was dawning.

On August 6th, 1991, the first Web page went live.

Before this page, the Internet could only be accessed by command-driven prompts and was closer to computer programming than anything we know today. It was an unwieldy machine used by small groups of people and the vast majority didn’t see any use in it at all. However, a computer scientist named Tim Berners-Lee, working at CERN in Geneva, created what must be one of the most world-changing inventions in human history. HTML.

HTML stands for ‘Hypertext Markup Language’. It looks simple now, but it caused an evolutionary shift greater than the wheel. It provided the ability to create a page of text that could be accessed instantly, anywhere in the world. It was initially created to share papers and speed up collaboration in the scientific community. All anyone needed was the address of a page and they could follow along and share updates. The real genius of HTML however, and the thing that led to Google becoming the world force it is, was the hypertext. You could link the address of a page to a word within the text. This allowed users to click from one page to another, seamlessly and encouraged others to create their own HTML pages. Scientists were quick to make use of this new technology and soon a spider’s web of pages developed and thus, the World Wide Web was born. For the first year or so, it was used mainly by the scientific community, but in January 1993 came another landscape shift. The first ever Web ‘browser’.

1993: 130 websites online

Mosaic was a desktop application that allowed anyone to browse the Web by visiting pages and clicking hyperlinks. It contained updates allowing for images and multimedia to make the whole experience more appealing. This greater accessibility and ease of use promoted a leap forward and soon more users were creating websites. It was around this time the topic of search came up, because following hyperlinks wasn’t the most efficient way of navigating the Web.

Search engines today use automated programs to follow ‘hyperlinks’ and report information back to a central database. This data can then be accessed by a search form and results delivered back to a user matching their query. These automatic programs are known as spiders (because they crawl the Web) or more often we call them robots or just bots.

It was a guy named Matthew Gray in June 1993 who developed the first ever spider and it was called the World Wide Web Wanderer. The purpose of the Wanderer was to measure the size of the Web by following every link and populating an index called ‘Wandex’. Unknowingly, Matthew had developed the perfect component for a search engine and in December 1993 a Scottish developer named Jonathon Fletcher furthered the concept by building the first recognisable search engine, JumpStation. This was announced on the Mosaic browser on the 21st of December 1993 and paved the way for 1994, the year the Web went commercial.

1994: 2,700 websites online

By January of 1994, the Web had grown exponentially, with a wide range of websites available. The introduction of search engines allowed for greater accessibility and it’s no surprise that early 1994 saw the first ever website provide ‘sponsored links’. The Global Network Navigator (GNN) was an online magazine created in 1993 by O’Reilly Media and by 1994 had enough visitors to start selling the first clickable adverts. Now there was money to be made, there would be no stopping this new innovation and word began to spread fast.

The World Wide Web’s user base continued to grow, and new websites appeared daily. Search engines were becoming more important but they still couldn’t provide the results we know today. Most only worked if you knew the exact name of the website you were searching for. This is why Web directories were the go to place for discovering new websites. Web directories provided a user-friendly page containing links categorised by topic, and it was far easier to discover new websites via a directory than it was through search.

The first search engine that could actually read web pages was called WebCrawler. It launched in April 1994 and had around 4,000 websites on its database. Within seven months it had delivered its millionth search query, the term “nuclear weapons design and research”. This goes to show that the Web’s user base was still largely scientific, but by the end of 1994 more commercial companies began launching websites. This was also the year that Microsoft entered the market, so it was inevitable that big changes were ahead.

1995: 23,500 websites online

1995 can be considered the year the Web hit the mass market. Netscape was the biggest browser, having struck a $5 million deal with the five biggest search engines, Yahoo!, Magellan, Lycos, Infoseek, and Excite. Not long after this, Bill Gates entered the browser market, introducing Internet Explorer on the new Microsoft Windows 95 operating system. This triggered the beginning of the infamous ‘browser wars’ which led to the death of Netscape in 2003. Already, just four years after the birth of the Web, big corporations were making a grab for power. Due to Internet Explorer being integrated into PCs, we would see a dramatic increase in Web users, which resulted in an even bigger increase in websites as we headed into 1996.

1996 – 257,601 websites online

By 1996, rave culture in Britain had all but fizzled out. Britpop ruled and a generation of disillusioned ravers wandered the streets not knowing what to do with themselves. Life was pretty much the same as it had been in 1991. There were still no mobile phones, however we did have a new game console called the Nintendo 64. I still had no knowledge of the Internet at this point and was only just starting to hear about the Web. Suspecting computers were the way forward, I enrolled into a computing course in 1996 and by 1997, I was starting university.

1997 – 1.1 million websites online

1997 saw the number of websites surpass a million. I was studying in Brighton at the University of Sussex and had just acquired my first email address. I also picked up my first mobile phone, a BT ‘brick’. I could now make phone calls for an extortionate fee, but being the first of my friends to own a mobile phone, I didn’t have anyone to text. I also had a new Pentium II computer with full access to the Web via the university’s network.

By this time, you could find websites on almost any subject and Yahoo! was the most dominant search engine. There were around 70 million Web users at this point and I suspect it was around this time that SEO started to become a thing. There was money to be made by having a website appear among the top results of a popular search engine. The vast majority of people would have no concept of SEO, so I imagine the first ‘SEOs’ having a field day. Much to the annoyance of search engine developers. The programmers behind search engines must have hated SEO, but to understand why, you need to put yourselves in the shoes of an SEO at the time. No one knows exactly when SEO started, but I imagine a typical scenario went something like this.

One afternoon in the summer of 1997, Richard Crenshaw, head of sales at Bell Computers walks into the IT department. He finds Brad, a 24 year old systems analyst sitting in front of a large CRT monitor.

“Hey, Brad, you know computers, right?”

“Uh, yeah sure,” says Brad.

Brad never had a member of the sales team speak to him, especially Crenshaw, the top salesman of the group.

“Do you know Lycos?”

“Sure,” says Brad. “It’s a search engine, why?”

“Can you open it up for me?”

Brad opens Netscape and browses to the Lycos home page.

“Right,” says Crenshaw. “Type in ‘LaserJet 500 Plus’.”

Brad does as he’s told and a list of websites appear on the screen. Crenshaw leans over.

“How do we get on that list?”

Brad studies the screen.

“The LaserJet 500 Plus is our best selling printer. I’m sure we would sell a whole lot more if we were on that list .”

Brad’s mind starts ticking.

“It must be an algorithm,” he says.

“Well, do what you’re paid for,” says Crenshaw. “Get us on that list!”

Brad, being an analyst, starts to wonder how search engines work. He’d heard of Web crawlers at university. He figured they must be delivering results based on an algorithm related to the search query. The search query is ‘LaserJet 500 Plus’ and he notices that all the results have the exact same phrase in the title of their web page.

He analyses the code of each page and deduces that the top result has the phrase ‘LaserJet 500 Plus’ mentioned more times than the results below. He counts up the phrase and it’s mentioned 11 times on the page. He navigates to their own website and they don’t even have a page for the LaserJet 500 Plus.

Within a week, Brad added a new LaserJet 500 Plus page to their website, with the phrase ‘LaserJet 500 Plus’ mentioned 20 times. A week or so after, the Bell Computers website is number one on Lycos for the search term ‘LaserJet 500 Plus’. Crenshaw is a happy salesman, and Brad has just been promoted to head of SEO at Bell Computers.

Brad isn’t the only person to make this discovery. As the years progressed, other Brad’s were put in the same situation. Computer ‘nerds’ tried things for fun. Before long, you would need the phrase ‘LaserJet 500 Plus’ on your web page over a hundred times for it to be in first position.

To get around this, many SEOs hid keywords by making them the same colour as the background, or making fonts so small they were invisible. Many of the high ranking web pages weren’t even selling LaserJet 500 Plus printers, SEOs just wanted to prove they could do it. Crenshaw and others like him, wanted to own the whole first page of results to prevent competitors from being there. Gradually, the search results for ‘LaserJet 500 Plus’ would be a mess and of little use to anyone who actually wanted to buy a printer.

This same scenario can be applied to a thousand different search terms, which made crawler based search engines like Lycos and Excite unpredictable at the time. This is why in 1997 and throughout 1998, Yahoo! remained the most popular search engine because its results came from a directory and not the Web itself. It wouldn’t be long however, before two innovations would change the face of search forever.

1998 – 2.4 million websites online

The World Wide Web continued to grow throughout 1998 and although crawler-based search wasn’t perfect, there was growing demand to navigate an ever increasing sea of websites. Search engines like Lycos and Excite were flourishing, but were thwarted by SEOs manipulating the results. The key to how easy it was to manipulate search engines back then, is the word ‘algorithm’. It was the job of an SEO to figure out how the search engine algorithm worked, then create web pages that filled the criteria to gain the top positions. In 1998, search algorithms were relatively simple and easy pickings for an SEO.

SEARCH ENGINE ALGORITHMS

In computer programming, an algorithm is a sequence of computer-implementable instructions. A Web crawling robot is programmed to follow hyperlinks and report information back to a database. The algorithm determines how this information is processed and presented to the user.

The first search engines only had the ability to search for a specific domain address. The first robots, like the World Wide Web Wanderer, would crawl the Web by following hyperlinks and sending addresses back to the database. This allowed for a search function that could return the link of an address the user had searched. The algorithm was simple and in plain English would have looked something like this:

A user searches the address: info.cern.ch
Algorithm: Find matches

Find all web pages with domain address containing: info.cern.ch
Display all web pages with domain address containing: info.cern.ch

This algorithm will deliver a list containing every web page containing ‘info.cern.ch’ in the domain address.

By 1998, search engine algorithms had evolved, but were still far less sophisticated than they are today. They mainly used metadata or text within the web page to determine if the page was relevant to a user search. Metadata was the first thing to be manipulated by SEOs, as this was the easiest component for the algorithm to use.

The metadata of a web page is placed within the head section of the HTML. It isn’t readable by users, but it is readable by robots and browsers. It’s the perfect place to instruct robots what the page is about, so they can categorise it and deliver it in search results.

This is the metadata taken from the Dell Computers home page in January 1998:

<title>Welcome to Dell.com</title>

<meta name="KEYWORDS" content="Dell, Dell Computer Corporation, server, home office, small business, direct model, software, peripherals, printer, latitude, dimension, OptiPlex, desktop, notebook, PowerEdge, laptop, Inspiron, hardware, corporate, workstation">

<meta name="DESCRIPTION" content=" Dell Computer Corporation, the world's leading direct computer systems company, offers an online store where customers can electronically design, price and purchase computer systems, software and peripherals and obtain online service and support.">

<meta name="author" content=" Dell Online, Round Rock, Texas">

You will notice there are four components to the above ‘metadata’. These are title, keywords, description and author. The title, description and author contain information describing the page itself, the same as you would have for a book, or a movie. The “keywords” part is an instruction for search engines, to help them determine which words match a user’s search. Search engine algorithms relied heavily on the “keywords” metadata, but consistent ‘keyword stuffing’ by SEOs resulted in it being phased out by the year 2000.

A simple algorithm for a search engine in 1998 may have looked like this:

A user searches the word: laserjet printer
Algorithm: Find matches

Find all web pages with domain address containing: laserjet printer
Find all web pages with meta name=”KEYWORDS” containing: laserjet printer
Find all web pages with meta name=”TITLE” containing: laserjet printer
Find all web pages with meta name=”DESCRIPTION” containing: laserjet printer
Find all web pages that contain text: laserjet printer

The above algorithm will compile a list of every web page containing the phrase ‘laserjet printer’. It isn’t necessary for a web page to contain the phrase in all of the criteria, but it needs to be in at least one. For example, a web page may only have the phrase mentioned in the meta “keywords”, or the meta “title”. However, if it is mentioned anywhere within the criteria, it will be added to the list.

A different algorithm is then required to determine which order to present the results to a user. A random list would be no help, so it wants to deliver the most relevant result first and then order the rest by relevance. This could be achieved by the following logic:

Algorithm: Order by relevance

Web pages with laserjet printer in the domain address (i.e laserjetprinter.com)
Web pages with laserjet printer in meta name=”KEYWORDS”
Web pages with laserjet printer in meta name=”TITLE”
Web pages with laserjet printer in meta name=”DESCRIPTION”
Web pages with most mentions of laserjet printer

According to the above algorithm, every page with the phrase ‘laserjet printer’ in the domain name will be at the top of the list. This makes sense because the actual address of the website matches the search, so it must be the most relevant. This would work better with exact match domain names such as laserjetprinter.com, but a close variation such as bestlasterjetprinters.com would also be relevant. After this, it would class the meta name=”keywords” as the most important, because the owner of the web page has specifically instructed that this page is relevant to that phrase.

The next order would be the pages that have the phrase in the meta “title”. This makes the page more relevant because the phrase is in both places. After that, every page that also has it in the meta “description” will be sent to the top. Then, every page containing the phrase within the content on the page itself will be sent to the top. So, out of however many thousands of web pages that mention the phrase, the pages that have it in all five places will be at the top of the list.

Sorting these would then just be a case of ordering by quantity. Logic would say a page is more relevant the more times it mentions the phrase. So, the page that fulfils all of the criteria the most amount of times will be at the top of the search results. The rest will be ordered from highest to lowest.

This is a logical algorithm designed to deliver the most relevant search results to a user. The problem is that programmers didn’t allow for human nature and more specifically, SEOs. For SEOs in 1998, it was simple. You buy a domain name and add every phrase you want to be found for on every part of the page, as many times as possible. There may have only been around 2.4 million websites in 1998, but with over 188 million Web users, there was plenty of incentive to be top of the search results. This must have been extremely frustrating for search engine developers wanting to display the best results, because the best results depended on who was doing the most aggressive SEO. This was a conundrum, but two major innovations were about to change this and make crawler-based search the dominant model on the Web.

In 1998, Yahoo! was the most popular search engine, but it wasn’t a crawler-based search engine, it directly accessed the Yahoo! Directory. Though crawler-based search offered more potential, there hadn’t been one developed that was as user-friendly or delivered results as good as Yahoo!

However, in February of that year a company called GoTo, owners of a search engine called the World Wide Web Worm, came up with an advertising model that would change the landscape forever. This new innovation was the birth of ‘Pay Per Click’ advertising and GoTo became the pioneer of Paid Search. Rather than take on SEOs, GoTo offered a bidding system, where advertisers could bid on how much they would be willing to pay to appear at the top of the search results. The bid was paid every time a searcher clicked a link to their website. Within five months advertisers were paying anything up to a dollar per click. Everyone could see how profitable this new Pay Per Click model could be and all that was needed was a search engine that could rival the results offered by Yahoo! Two students from Stanford University came up with the answer.

Two years earlier in 1996, while studying for a PhD in computer science, Larry Page completed a dissertation exploring the mathematical properties of the World Wide Web, and more specifically, its link structure. He focused on how and why web pages linked to one another, and considered how those links provide information about the pages they link to. It may have been a mathematical exercise, but it wasn’t a giant leap to see how this could be used to improve the process of search. Over the next two years, Larry Page and Sergey Brin developed a web crawler and algorithm based on links to web pages rather than metadata and text like existing search engines. Right away they discovered this provided far superior results and it opened the door to a new search engine that would eclipse all others.

In 1998 they authored a paper titled “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, which documented how their new search engine worked.

“We chose our system name, Google, because it is a common spelling of googol, or 10100 and fits well with our goal of building very large-scale search engines.”

Larry Page

With the introduction of Pay Per Click advertising, the timing couldn’t have been more perfect. On September 7th, 1998, Google was officially born.

1999 – 3.2 million websites online

As we entered 1999, the Y2K bug was putting fear into businesses worldwide. The World Wide Web continued to grow and Google set about their plan for world domination. In June, they received $25 million in funding, including a $250,000 cheque from Jeff Bezos. With Pay Per Click advertising and highly accurate search results delivered by the new ‘PageRank’ algorithm, Google would soon develop a licence to print money. As long as the world didn’t crash due to Y2K, it looked as though the 21st century belonged to Google. But, what about SEO? The landscape had changed, but SEO was now big business and people were employed just for the purpose of SEO. They had to find a way around it.

GOOGLE PAGERANK

The algorithm pioneered by Google was named PageRank. It must have been fate or a happy coincidence that Larry Page’s surname is Page because he called the algorithm he devised to rank pages, PageRank. Either way, it was relatively simple and highly effective.

It worked by counting and reading the links it followed and assigning each page a rank depending on how many links were pointing at it. It didn’t just count the number of links however, it also looked at the relevancy. For example, if a web page about printers links to another web page about printers, the other page would be awarded an extra point for relevance. This thwarted SEOs who were trying to rank pages that weren’t relevant because other websites wouldn’t link to it if it wasn’t useful. PageRank determined that the more relevant links a web page had, the more useful it would be.

This logic was then scaled to weigh the importance of specific links. If a web page had the most links related to a specific subject, for example, printers, then a link from that page would be more powerful than a page that didn’t have any links at all. This is where PageRank came in and it was a score between 0 and 10. The most powerful web pages in the world had a PageRank 10, because they had thousands or even millions of other pages linking to them. A new page would have a PageRank 0 because it had no other pages linking to it. This allowed PageRank to flow and be passed between links, so a PageRank 10 linking to a PageRank 0 would pass on a percentage of its PageRank and the 0 may become a 4. The logic of all this meant that the higher the PageRank, the higher the page would rank in the search results, so it became the number one focus for SEOs.

There were now over 3 million websites online and over 280 million Internet users, but with the introduction of broadband, the year 2000 would see an explosion.

2000 – 17.1 million websites online

The new century had arrived and it was nine years since the first website went live. There was no stopping the World Wide Web and with nearly half a billion worldwide users, it was here to stay. Sitting on top of all of this was the most accurate search engine to date and SEOs had their sights set firmly on Google’s PageRank algorithm. This had implications that stretched the length and breadth of the Web and would give SEOs the bad name they still live with today.

SPAM

Spam has a number of meanings aside from the tinned meat. The Oxford English dictionary defines it in terms of email spam, but there’s a specific type of spam when it comes to SEO. Spam within SEO refers to the unethical measures that try to influence the position of a website within the search results. Before Google, we saw it in the number of keywords within the content and metadata. This was the bane of search engines such as Lycos and Excite, but aside from search engines, the only websites affected were those managed by SEOs. Now that SEOs had PageRank to contend with, they needed to gain links from other websites, and this would impact the entire Web itself.

GOOGLE’S ALGORITHM

Links were now a major part of SEO and the aim was to get as many as possible. Links were the major difference between Google’s algorithm and previous search engines and basing the algorithm on links produced better results because it was harder to manipulate. Google had decided to ignore the meta name “keywords” completely at this point, so in 1998, their algorithm would have looked something like this:

A user searches the word: laserjet printer
Algorithm: Find matches

Find all web pages with domain address containing: laserjet printer
Find all web pages with meta name=”TITLE” containing: laserjet printer
Find all web pages with meta name=”DESCRIPTION” containing: laserjet printer
Find all web pages that contain: laserjet printer
Find all links with anchor text containing: laserjet printer
Find all links pointing at each web page that fulfils one or more of the above

The above algorithm would be similar to other search engines, but now they also returned data based on links. The major difference would then be how Google ordered their results.

Algorithm: Order by importance

Web pages with laserjet printer in the domain name (i.e laserjetprinter.com)
Web pages with laserjet printer in meta name=”TITLE”
Web pages with laserjet printer in meta name=”DESCRIPTION”
Web pages with laserjet printer in the headings of the page
Web pages with most mentions of laserjet printer
Web pages with most links from other websites that contain ‘laserjet printer’ in the hypertext.
Web pages with most links from other websites
Web pages with the highest Page Rank.

The first part of the algorithm is similar to other search engines, but after the results got to the point where the pages with the most mentions were at the top, it came down to links. The Google robot (Googlebot) could count how many links were pointing at each web page and it could count how many of those links contained the words ‘laserjet printer’ in the anchor text.

The anchor text of a hyperlink is the word or words containing the link. So, if a web page contains the phrase ‘laserjet printer’ and this is hyperlinked to a page selling laserjet printers, then Google would consider it an important ranking signal.

So now, the results would be sorted again and the web pages with the most links containing the anchor text ‘laserjet printer’ would go to the top. After this, the web page with the highest PageRank would go top. The ultimate goal for an SEO who wanted to rank first on Google for the keyword ‘laserjet printer’ would be:

• Register the domain name ‘laserjetprinter.com’ or something close.
• Include ‘laserjet printer’ in the meta name=”TITLE”
• Include ‘laserjet printer’ in the meta name=”DESCRIPTION”
• Include ‘laserjet printer’ in all the headings on the page
• Mention the phrase ‘laserjet printer’ as many times as possible on the page
• Gain as many links as possible from other websites with the anchor text ‘laserjet printer’
• Gain as high a PageRank as possible by gaining links on other high PageRank web pages.

Completing the above criteria was relatively easy for an SEO at this time, especially as the competition was nowhere near as fierce as it is today. Having an exact match domain such as ‘laserjetprinter.com’ was a huge advantage, as this would usually trump most other requirements. This caused a whole new industry to form called domain brokerage. Fortunes could be made by registering an exact match domain name in a competitive industry. For example, creditcards.com sold for $2.75 million in 2004 and porn.com went for $9.5 million in 2007. Links themselves were a little more difficult to come by and this would also become an industry. Link building, as it was now called, would become Google’s biggest threat and caused them to implement a massive algorithm change in 2012. In 2003 however, link-building was a major part of the process.

2003 – 40.9 million websites online

By 2003, the Web had a user base of over a quarter of a billion people and there were over 40 million websites competing to be found on search engines. I had completed my HND by now and was still undecided where to take it. A chance encounter with a guy selling furniture put SEO on the map for me. His website ranked in first position on Google for keywords such as furniture, mirrors, oak tables and more, and he was making a fortune. He kept his SEO process a secret, but it sparked a curiosity that remains with me to this day.

The biggest search engine was still Yahoo! with 29.5% of the market, but Google was closing fast with 28.9% and Microsoft’s MSN completed the big three with 27.6%. The following year Google would go public and cement its position as the dominant search engine, which it still remains today.
With 40 million websites to compete with and a third of search traffic going through Google, link building was now an industry. There were lots of ways to gain links from other websites and in 2003 these were the most popular:

BUYING LINKS

The go-to place for an SEO between the years 2003 and 2012 was an SEO forum. SEO was an underground community between these years and forums were the place to find all the tricks that worked and those that didn’t. The top forums were Webmaster World, SEOChat and Aaron Wall’s SEO Book, but I joined most and all I could find. Forums also happened to be the best place to buy and sell links. It was all about PageRank and you could see the PageRank of a page by adding a toolbar to the browser. You would often find people selling a PR5 link for around $50 and a few of these were usually enough to gain a PR4 for your own website.

WEB DIRECTORIES

Web directories were a booming market now that links were required to rank on Google. Yahoo! and DMOZ were the two biggest on the Web, but hundreds sprang up and you could usually gain a link from a directory in exchange for a small fee or a link back from your own website.

ARTICLE MARKETING

People tend to be lazy by nature and this would be taken to the extreme in later years, but in 2003 article marketing was booming. Websites needed content and rather than hire a writer, it was easier to browse through an article directory and use as much content as you wanted for free. The only caveat would be that any content you used would contain a link or two added by the writer. For an SEO who knew how to write, this was a great way to gain links on multiple websites. You could write an article on printers for example, add a link to your client’s website and publish it on an article marketing website such as ezine.com. This would then be used by multiple websites and gain valuable links back to your website.

FORUMS

Forums were an easy way to gain links in the 2000s. It was easy to create an account on a forum, make a post and add a link in your signature. This way every time you made a post you would gain an additional link. I used to frequent the Statcounter forum, which being a PR10 meant it was visited multiple times each day by search engines. If I wanted a search engine to find an update on any of my pages, I would add a link on the Statcounter forum and it would usually visit within hours.

BLOG COMMENTS/ GUEST PAGES

WordPress is the number one blogging platform today and in fact, WordPress websites make up almost 40% of the entire Web. In 2003, however, the blogosphere was wide open with a range of different platforms. Links could be gained by commenting on blogs and guest pages and this became a staple part of the link-building process.

LINK EXCHANGES

Link exchanges were common because every website needed links. It would be a simple case of sending emails and requesting a link in exchange for a link back. At the time it was a win/win for both parties and an accepted part of SEO.

All of the above link-building methods were highly effective between 2003 and 2012 and an essential part of SEO. However, the World Wide Web continued to grow and this meant more SEOs and more companies wanting SEO. This led to more competition and a higher number of links required to rank on Google. This led to automation on an unprecedented scale.

2010 – 206.9 million websites online

By 2010, there were over 2 billion Web users and well over 200 million websites. Google owned 65% of the search market with Yahoo! and MSN competing for a mere 13% each. Google was adept at acquisitions by this point and bought companies that were either a threat or could be used to further its own dominance. By 2010, Google had purchased multiple advertising platforms (later to become AdSense), mobile phone technology (later to become Android), mapping technology (later to become Google Maps) and YouTube among others.

I had been in SEO for some time by now and was getting nervous about where it was all heading. The practice of unethical SEO was (and still is) known as ‘black hat SEO’. There are degrees of black hat SEO I would never resort to, such as malware, hacking, or cloaking pages. There were, however, certain aspects of black hat SEO that were essential for ranking on Google at the time. There were also lots of grey areas. For example, I discovered a trick that involved adding a hundred or more words into the meta “title” of a page, which seemed to trick the algorithm and resulted in multiple first place listings. I wouldn’t have considered this black hat, because it was something I was doing to my own websites to achieve results. As an SEO it was almost impossible to take a client’s money and rank high on Google using ‘white hat’ techniques alone. Every SEO was using tactics to beat the algorithm and they worked.

Ranking on Google became a game of having to do what your neighbour did, whether you agreed with it or not. I did, however, start to question my ethics when it came to link-building. There was now automated software for link building that delivered links at a frightening rate. Link exchanges were virtually dead at this point, because if you needed a thousand links, then making a thousand link exchange requests would take forever. SEOs were building robots of their own and the World Wide Web was paying the price.

WEB DIRECTORY AUTOMATION

There were thousands of Web directories by 2010 because they were an easy way to gain links or earn money by charging a fee in return for a listing. It takes around twenty minutes to create a listing on a directory, so when you’re competing with websites that have hundreds of directory links, it can take a long time to catch up. For this reason, automated submission software was created that could submit a website to a thousand or more directories at the click of a button. This filled up low quality directories with low quality listings that multiplied year on year. When you think that a Web directory can have a hundred thousand pages or more, and there are thousands of Web directories, this means there were millions of pages built purely for gaining links to help websites rank on Google.

ARTICLE MARKETING AUTOMATION

Article marketing had also changed a lot since 2003. Google implemented a duplicate content update in 2005. This meant that pages with duplicate content were now devalued and wouldn’t be able to rank. The days of multiple websites using the same article were gone, but article marketing was still big business. Much like directories, there were thousands of article marketing websites whose sole purpose was to sell or exchange links. To get around the duplicate content issue, some bright SEO developed a process called ‘Spintax’. This is where you could write one article, but provide multiple synonyms for words throughout, and then ‘spin’ the article to produce multiple copies that were (in the eyes of Google) unique content.

For example here is a sentence that includes Spintax:

The {best|greatest|most amazing} {SEO|marketing} {agency|company|firm} in the {country|world|universe}

On ‘spinning’ this sentence we can produce multiple other sentences that would pass a duplicate content check. For example:

• The best SEO company in the world.
• The greatest SEO agency in the country
• The most amazing marketing firm in the universe.

Software such as Article Marketing Robot could generate a thousand articles out of one article spun a thousand times. It would then submit those articles to a thousand article marketing websites and gain a thousand backlinks at the click of a button. The problem with this is 99% of the articles would read terribly and any human could tell they were autogenerated. Similar to directories, thousands and thousands of these articles were hitting the Web daily and their only intention was to gain links for ranking positions on Google.

FORUM AUTOMATION

Forums were still a great way to gain links in 2010, but again, manually creating accounts took time. There was an automated solution for this too with software called XRumer. This would crawl the Web looking for forums and automatically create accounts and add links to the user profiles. XRumer could easily gain thousands of forum links a day and was the bane of a forum owners’ life. Yet again, this would create millions of pages that were only there for the purpose of gaining links for Google.

BLOG COMMENT AUTOMATION

WordPress had become the number one platform by 2010 and blogging had never been easier. There were millions of WordPress websites online and links within blog comments still counted as links. Not surprisingly, there was automated software for this too. Scrapebox was a Web crawler application that could be given instructions to carry out pretty much anything. An easy win was to ask it to find WordPress websites and add some auto generated text and a link in the comment sections. You could easily gain five thousand blog comment links each day using this tactic. For anyone maintaining a blog, this was a nightmare. There were, however, thousands of blogs that weren’t maintained regularly, and these gradually filled up with spam comments containing links that were only there because links were needed to rank on Google.

PRIVATE BLOG NETWORKS

With the ease of building WordPress websites came the rise of PBNs or Private Blog Networks. This was a great way of gaining powerful links and involved buying expired domains that had PageRank already assigned. After purchasing a domain name, you could attach it to a WordPress website and start adding or selling links. People would build networks of hundreds, or thousands of blogs and sell link packages to SEOs. This again filled the Web with websites and pages that were purely for the benefit of SEO.

When you put all this together, you can see how many millions of pages were added to the Web only to serve one purpose. To provide links for the benefit of Google ranking positions. In 2010, these tactics worked and it was easy to rank on Google for big money-making keywords. The ease of these tactics also kept the price of SEO low. I remember one client on a budget of £500 per month ranking first for the keyword ‘lawyers’. He was paying Google over £18,000 to be one place higher in the Google Ads position.

The practice of SEO was being seriously demonised around this time, led by the head of web spam at Google, Matt Cutts. There was good cause to blame SEOs, but they were trying to make a living and if you wanted to rank on Google, you had to do it. There were arguments to be had that all this was Google’s fault because it was their algorithm that relied on links. Something had to give and I knew it couldn’t continue the way it was going. It came sooner than I expected however and caused much more devastation.

2011 – 346 million websites online

The wind of change was in the air, and in February 2011, Google introduced the first of what would be two major algorithm updates. The first of these targeted websites with low quality content and ‘content farms’. This was mostly aimed at websites with spun content, such as article marketing sites, but many others got penalised too.

PANDA

The name “Panda” comes from the Google engineer, Navneet Panda, who helped develop the algorithm. Considering Google were increasing their own advertising space around this time, it’s ironic that this update had a major impact on web pages containing large amounts of advertising. No matter how adversarial Google had become with SEO by this point, I had to admit this was an update that was needed. Some of the content being produced by automated software was laughable.

When the Panda algorithm hit, it had a massive impact on the Web and affected around 12% of search results. Initially the quality of search went down and Google had to complete several revisions to get the effects they were after. This caused huge disruption, as it was the first time an algorithm update had impacted so many websites. In general, most SEOs had no complaints as it was clearly needed to reduce the amount of spam being released. Shortly after the update, Google made an announcement offering guidance on what makes a high quality web page.

• Would you trust the information presented in this article?
• Is this article written by an expert or enthusiast who knows the topic well, or is it more shallow in nature?
• Does the site have duplicate, overlapping, or redundant articles on the same or similar topics with slightly different keyword variations?
• Would you be comfortable giving your credit card information to this site?
• Does this article have spelling, stylistic, or factual errors?
• Are the topics driven by genuine interests of readers of the site, or does the site generate content by attempting to guess what might rank well in search engines?
• Does the article provide original content or information, original reporting, original research, or original analysis?
• Does the page provide substantial value when compared to other pages in search results?
• How much quality control is done on content?
• Does the article describe both sides of a story?
• Is the site a recognized authority on its topic?
• Is the content mass-produced by or outsourced to a large number of creators, or spread across a large network of sites, so that individual pages or sites don’t get as much attention or care?
• Was the article edited well, or does it appear sloppy or hastily produced?
• For a health related query, would you trust information from this site?
• Would you recognise this site as an authoritative source when mentioned by name?
• Does this article provide a complete or comprehensive description of the topic?
• Does this article contain insightful analysis or interesting information that is beyond obvious?
• Is this the sort of page you’d want to bookmark, share with a friend, or recommend?
• Does this article have an excessive amount of ads that distract from or interfere with the main content?
• Would you expect to see this article in a printed magazine, encyclopaedia or book?
• Are the articles short, unsubstantial, or otherwise lacking in helpful specifics?
• Are the pages produced with great care and attention to detail vs. less attention to detail?
• Would users complain when they see pages from this site?

The most significant part of the Panda algorithm is it affected entire websites rather than just pages. This meant that good housekeeping of a website became more important than ever. Some website platforms, especially eCommerce like Magento, can create multiple duplicates of pages which aren’t intended to be part of the main website. The Panda algorithm meant that if this went unchecked, then the entire website could fall victim to Panda and struggle to rank. Initially, Panda would have a refresh every few months, but in 2013 it was integrated into the main algorithm and became part of standard SEO practice. On the whole, I agreed with the Panda update, as did most SEOs and the wider Web community in general.
Google’s head of Web spam, Matt Cutts later said that Google had made a monetary loss due to Panda, saying in 2016:

Google took a big enough revenue hit via some partners that Google actually needed to disclose Panda as a material impact on an earnings call. But I believe it was the right decision to launch Panda, both for the long-term trust of our users and for a better ecosystem for publishers.

Matt Cutts

However, this is at odds with what Larry Page said in 2012:

Google had a really strong quarter ending a great year. Full year revenue was up 29%, and our quarterly revenue blew past the $10 billion mark for the first time.

Larry Page

So things were going well for Google and Panda did a great job of cleaning up the Web. Search results were at a higher standard and SEOs had a set of guidelines on how to produce high quality content. Life was sweet. Which made what happened next, even harder to believe.

2012 – 697.1 million websites online

In the year 2012, Google owned over 70% of the search engine market. Their dominance was complete. There were 2.5 billion Internet users and over 697 million websites online. Google generated over $50 billion in revenue in 2012 and was a global powerhouse. Even though Google had become a phenomenal success, it seemed the more money they made, the hungrier they became. Google’s own advertising space was taking over the search results, with adverts at the top, side and bottom of the pages. The board at Google must have been aware that SEOs were taking money away from advertising and they hated it. They were now a global corporation with shareholders who wanted more profit. It was for this reason that many SEOs believe the actions taken in 2012 were not for the benefit of the Web, but were more about destroying the competition. This was a cutthroat business and the actions Google took in March 2012 were completely without mercy.

It was 9am, 26th March, 2012 when I received the first telephone call. I had around thirty SEO clients at the time and at 9am a confused client phoned to say she couldn’t find her website on Google. I searched her company name and I couldn’t find it either. I said I would call her back while I investigated. It was true, her website had disappeared. I looked for my own website and that was missing too. The phone rang again and it was another client saying they couldn’t find their website either. Confused, I headed over to the SEO forum, and the boards were on fire. Every two minutes another post or thread appeared announcing the disappearance of websites. Out of my thirty plus clients, around twenty had been removed from Google, including my own, which was ranking first for ‘SEO company’ the day before. It turned out Google had made another algorithm update. A big one.

PENGUIN

The previous update, Panda, was named after the engineer, Navneet Panda, but no engineer would want their name attached to this next update. Instead, Google picked the name of another animal whose name begins with P, Penguin. It would have been more fitting to call it Piranha or Python, but even that wouldn’t have done it justice. We waited a month before we had an official announcement made by Matt Cutts on 24th April:

We’re launching an important algorithm change targeted at webspam. The change will decrease rankings for sites that we believe are violating Google’s existing quality guidelines. We’ve always targeted webspam in our rankings, and this algorithm represents another improvement in our efforts to reduce webspam and promote high quality content. While we can’t divulge specific signals because we don’t want to give people a way to game our search results and worsen the experience for users, our advice for webmasters is to focus on creating high quality sites that create a good user experience and employ ethical SEO methods instead of engaging in aggressive webspam tactics.

Matt Cutts

The irony of this statement is that Google’s algorithm allowed these tactics to work for years. An SEO had no choice but to use these tactics if they wanted to rank on Google. There were a number of ways they could have approached this entire thing, but this is what they did.

A manual penalty was issued to any website involved in ‘unnatural’ link building. A manual penalty resulted in the removal of the offending website from Google. A reconsideration request could be submitted to Google and if you could remove all unnatural links then they may let your website back on its search engine.

According to Google, unnatural link building included:

• A high percentage of keywords within the anchor text of links
• Links from websites known for selling or exchanging links
• Too many links gained in a short space of time

Google estimated that Penguin affected approximately 3.1% of search queries. This sounds like a low figure, but if you think about it, every Google search returns around 30 pages of results. This is around 300 results for each keyword searched. 3% of 300 is 9, which is almost the entire first page of results. So, Penguin effectively wiped out the front page of results for most searches and it was usually the websites ranking on page one that were conducting SEO. If they wanted to take out the SEO community, they did it with the effectiveness of a nuclear bomb.

The unfortunate collateral damage of Penguin was that thousands of companies that relied on Google for business went bust. These were businesses that didn’t know anything about how Google or the Web worked. All they knew was they needed to be on Google, they paid for SEO and now, out of nowhere, their website was banned. This wasn’t a case of SEO damaging search results. The Web was highly established now and companies paying for SEO had well-maintained, highly relevant websites. In most cases, the search results weren’t being skewed at all and were providing excellent results which helped Google achieve the dominant position it enjoyed.

A more ethical approach from Google would have been announcing Penguin before they released it, giving SEOs a chance to clean up their backlinks and find a new process. Instead, they launched a surprise attack in the middle of the night, bankrupting the businesses of thousands of innocent people. Another option, which would have reduced the devastation for genuine businesses, would have been to penalise the link farms themselves rather than the websites they were linking to. This may have resulted in a drop in ranking positions for a business owner’s website, but not a permanent ban, which is how it turned out for most people. With highly intellectual programmers and tacticians, it is impossible to think that Google didn’t consider these options. Instead, they chose the most devastating, which would have two major benefits.

First, it would destroy the existing process of SEO and end any shortcuts that led to high ranking positions. This would dramatically increase the price of SEO and make the whole process take much longer than before. The obvious benefit of this is that it would be harder for SEO agencies to gain customers. The second major benefit for Google is that it destroyed the reputation of SEOs and made the whole industry appear unethical and dangerous. Who would want to pay for SEO only to see their website banned after a few months or a year? Although it was ruthless, it was a masterstroke for Google and sent a clear message to businesses. Don’t pay for SEO, pay for Google advertising, it’s safer.

Penguin would go on to have further updates and was eventually integrated into the core algorithm. The initial wave of a hundred thousand or so penalties caused massive damage and like a ruthless predator, Google followed up by applying constant pressure to the neck. We now have algorithmic penalties to deal with, and these are more insidious because there is no way to appeal or even be certain you’re in one. Google is notoriously vague on the subject of algorithmic penalties and with good reason. Divide and conquer is a Machiavellian strategy detailed in the book, The Art of War. It states “a Captain should endeavour with every act to divide the forces of the enemy”.

NEGATIVE SEO

Given human nature and the history of SEO, it wasn’t rocket science to work out what could happen if a website could be demoted or banned for having unnatural backlinks. Open warfare among SEOs. If you want to knock a competitor off the front page of Google, just start building automated links to their website. Matt Cutts had to play this down within months of Penguin being introduced in 2012, and his predecessor John Mueller was still playing it down in 2020. The fact is though, it exists. Luckily I haven’t encountered it too much, but it works and is difficult to come back from.
I knew it was time for a change. I lost my love for SEO. Google was doing their best to destroy it and all it cared about was profit. SEO as we knew it was dead. I knew it was over, but it would take some SEOs longer to realise than others.

2013 – 672.9 million websites online

In 2013, during the aftermath of Penguin, most SEOs were at a loss. Even mainstream media were announcing the death of SEO. I had a business partner at the time, who couldn’t accept it. He continued the same tactics, knowing full well it could, and did cause penalties for clients. I may have questioned my ethics when it came to automated link building, but I wasn’t prepared to play Russian Roulette with other people’s businesses. I walked away from the company, giving him everything.

Over the next three years, I worked from home, concentrating on creative writing. I always preferred the creative side of SEO, such as building websites and writing copy. SEO was all too formulaic with the rise of automated link building and It was obvious it couldn’t last. Many SEOs continued, however, and more innocent businesses got penalised. Gone were the days of £500 monthly campaigns. Google had effectively wiped out small businesses that couldn’t afford the high prices involved. Rather than continue under the old model, I quit the game. I couldn’t, however, shake off my fascination and continued to analyse the algorithm.

I noticed more established brands taking over the search results. There had been a relatively minor update in 2009 called Vince, which appeared to favour brands, but a good SEO could still get a small company dominating search results nationwide. It wasn’t how much money a business had, it was how clever its SEO was. In the aftermath of Panda and Penguin, however, the big brands were taking over. Now it would take a big investment to compete. The more I analysed the search results the more I saw the algorithm favoured brands. It was an intriguing divergence because the more brands I analysed, the more commonalities I saw between them. There appeared to be brand signals and I spent more time analysing this than I did the novel I was supposed to be writing.

2015 – 863 million websites online

2015 was the year Alphabet Inc was formed, the new holding company of Google and parent to numerous acquisitions and subsidiaries. Google now owned 90% of the search market and ‘Google’ and ‘Googling’ had become verbs. Mobilegeddon was the latest algorithm update, and this promised to reward websites that were mobile-friendly and/or penalise those that weren’t. The algorithm itself was almost beyond analysis by this point and contained thousands of directives. For me, it made more sense to analyse high ranking brands than it did to analyse the Google algorithm.

2016 – 1.04 billion websites online

There were over a billion websites in 2016. The landscape was completely different to when I started in 2003. Smartphones and mobile apps were the norm. Social media dominated the lives of most people. Google owned over 90% of the search market and if you owned a business, it was accepted you needed to be on Google. Google’s annual revenue was over $90 billion.

I knew the whole model of SEO needed to change. Dominating search results on Google couldn’t be achieved by one person anymore. Now, it was more about building a brand, and that required multiple skills. SEO had always been billed monthly and was usually based on a fixed ‘package’, from small, medium to large. The only way I could see a successful SEO campaign running now was to charge by the hour, and each client would need different things.

When Google introduced IP based search results in 2014, small businesses could still compete on a local level. However, brand building was where it was at nationally, and this was expensive. For an SEO agency to compete on Google, it would require more work and resources than I had the heart for. I didn’t need the money and my attention was on writing screenplays, novels and other creative endeavours. However, one day I received an email from a startup in Brighton. I had always loved Brighton. It was by the sea, had a great vibe and was only 20 minutes from where I lived. On a whim, I decided to meet.

I went to Brighton and met the two owners. They were both successful at selling PPC campaigns. Pay Per Click advertising had its own issues with ethics and many companies that sold PPC were taking margins from their clients budgets. This meant that their clients thought they were spending on Google advertising, but in reality, a large portion was going to the PPC management company. They were uncomfortable selling this, so they created their company as an ethical agency that didn’t take margins and were completely transparent with their fees. This model proved successful and they started gaining PPC clients fast. An obvious partner to PPC is SEO and often their paid media clients would request SEO. They hired various SEOs to set up that side of the business, but were encountering the type of SEOs still stuck in the model that died in 2013. By now, SEO had a bad name because many, or most, SEO agencies were still using tactics that either didn’t work or that caused Google penalties. They had just fired their latest head of SEO because he admitted he couldn’t do it anymore. “SEO is dead!” he announced before leaving.

This meant the first discussion I had with this new startup, in June 2016, was ‘is SEO dead?’ SEO isn’t dead, I explained, but it’s expensive now. The type of SEO their previous hires conducted is dead. SEO is about building a brand, and that requires investment. If a company has the ambition to build a brand and a big enough budget to invest, then SEO is more important now than ever. The practice of SEO will never be dead because the Web is forever expanding. As we move through the 2020s we have more than two billion websites online. We need sophisticated search engines to navigate all this, and where there are search engines, there will always be search engine optimisation.

In many ways, SEO is the same as it was when it first started. The same as it was when our fictional Brad was asked to analyse the results back in 1997. We just need to analyse the websites performing the best and replicate what they’ve done, but do more of it. Trying to conduct SEO based purely on the Google algorithm is next to impossible now.