Okay, enough of that. Today I’m going to talk about crawlability and web design. I got to browsing the Unique ID blog and read Zack’s post “Straight From Google, The Four Biggest Search Rank Factors,” in which “crawlability” is listed as the very top, highest priority, most important search engine ranking ractor for a web site. This week I’ve also been following a discussion at the High Rankings forums about whether web designers have any SEO responsibility when designing a web site.

The discussion at High Rankings opened with the story of a businessman who hired someone to design a web site for his business. The site was built in Flash, and, as might be expected, the businessman’s web site didn’t do so very well in the search engines. When he sought professional SEO help, he was flabbergasted to learn that an all-Flash site is likely to rank poorly, if at all, in the search engines.

He asked the SEO pro, “Why did the designer use Flash when he knew I wanted search engine visibility?”

A better question would be, why do designers design “search-engine hostile” web sites when they know clients want search engine visibility?

As things stand in the world of web design, anyone with some elementary graphic design skills can get themselves a copy of Dreamweaver or FrontPage and hang out their “Web Designer” shingle, offering their services for a fee to all comers.

Some of these designers do indeed have a lot of artistic talent with respect to creating pretty, aesthetically pleasing, visually attractive web sites.

What these designers lack is a fundamental understanding of the underlying code and structure of web pages, and a fundamental understanding of how search engines crawl and index web pages, and a fundamental understanding of how a web site needs to be structured in order to have a chance of getting search engine traffic.

So these web designers make a “pretty design” in Photoshop or Fireworks or Flash, and use the built-in export features from those programs to auto-generate the code or the Flash file. The client ends up with a very pretty site that hasn’t a snowball’s chance in hell of doing well in organic searches.

The web is still relatively new as a commercial medium, and there is still some level of technical knowledge required in order to build a crawlable web site. Daily we see self-labeled “professional” web designers creating all-Flash sites, or using fancy javascript-based rollover images for global navigation, or relying on other artsy-fartsy features that doom a site to search engine purgatory — a site that is uncrawlable by search engine spiders, and generally invisible in the search engines.

The client doesn’t understand why his beautiful site gets little or no search engine traffic. The client eventually discovers, if he’s lucky or persistent, that he now has to pay for his site all over again, this time to have someone else tear apart his beautiful artsy-fartsy site and re-build it using underlying code and techniques that the search engines can crawl.

Does it have to be this way?

Should it be this way?

I say no, it shouldn’t. Some people might argue that a web designer’s responsibility is to design pretty stuff, not to perform search engine optimization. That’s true up to a point — I wouldn’t posit that it’s the web designer’s responsibility to do link building or write linkbait articles or do keyword research, unless those activities are explicitly included in the agreement.

But I do argue that anyone who holds himself out as a “professional web designer” should have a broad and fundamental understanding of the technology of the medium and the factors that are required for success in that medium. I do argue that the “professional web designer” is holding himself out as an expert, and the client is relying on the expert’s knowledge and experience.

The client shouldn’t have to have a specialist’s knowledge of the medium — that’s why the client hires a professional. When I hire a contractor to build a house, I shouldn’t have to become an expert on building houses, and I shouldn’t have to give the contractor explicit detailed instructions about how to run the wiring so it doesn’t burn the house down. I expect the contractor — ”the professional” — to have the knowledge and expertise to do that himself, even if the contract doesn’t explicitly state that the contractor will run the wiring so that it doesn’t burn the house down.

In an ideal world, building contractors would always run the wiring so that it doesn’t burn the house down, and in that same ideal world, web designers would always build crawlable web sites.

The only exception I would make to this general rule is when a client specifically requests features that will cause crawlability problems, and, after being educated by the web designer about the consequences of his request, the client insists that his aesthetic vision is more important than search engine visibility. The client is paying for the site, after all. But even then there are usually steps the web designer can take to mitigate and overcome the problems — including text links in the footer to complement the pretty Flash buttons at the top of the page, for example.

Meanwhile, we live in an imperfect “buyer beware” world where the web designers who understand the medium are competing against the web designers who don’t. Clients have to educate themselves sufficiently, and ask lots of questions of potential designers, in order to be sure they end up with a crawlable web site.

Developing a new web site — or re-developing an old one, for that matter — typically involves consulting with the client to determine the site’s target audience and primary objective, creating an attractive and functional design, turning the design into properly coded, valid html, building out the pages of content, and writing the server-side programming to perform whatever dynamic features are needed. But there are several steps that are frequently overlooked before a site “goes live.”

A custom “404 not found” page should be created. At minimum, it should incorporate the site’s overall design and navigation links, and might also include a search form, links to the most popular sections of the site, and/or a way to contact the site owner or webmaster for assistance. And make sure that requests for non-existent pages actually get a “404 page not found” server response. If any other server response is returned — particularly a “200 OK” response — the site could easily become persona non grata in the search engines, among other problems.
A robots.txt file should be created to tell the search engine spiders what pages or parts of the site should not be spidered. Even if you want every page to be spidered, a robots.txt file should be placed in the document root of the site, so as to avoid filling up the site’s error logs with “not found” entries for a non-existent robots.txt. This makes it much easier to spot errors resulting from actual bad links when you examine the error logs.

A canonical URL redirect should be implemented to send all site traffic to your desired canonical URL — either www.example.com, or just example.com (without the www). Whichever you prefer, your should make sure that all traffic to the other form is redirected via a proper 301 redirect.

Test all forms and other interactive features. Submit every form. Attempt to submit forms without required information, or with invalid information. Try to break them. Try really really hard to break them. And make sure that whatever is supposed to be done after the form is submitted actually happens. If an e-mail is supposed to be sent to the site owner, test it, and make sure the owner gets that e-mail with all the appropriate information. If there’s a search engine, search for some things. If the site relies on cron jobs, set them up ahead of time, and make sure they’re running as scheduled and performing as expected.

Test all redirects and rewrites. If the site uses Apache’s mod_rewrite module to present search-engine-friendly URLs, test them all, and test the non-search-engine-friendly versions, to make sure that every bit of content can only be reached by one unique URL.

Check subdirectories for directory listings. Make sure that directory indexes are turned off, and/or for any subdirectory that doesn’t have an index page, plop one in there.

Test all internal links and all outgoing links. Make absolutely sure that every single link leads to the right place. We don’t need to stinkin’ dead links!

There’s simply no excuse for ever launching a site without having completed each of these steps.

Web sites need to contain good content. It doesn’t matter how many people “ooh” and “aah” over how pretty your site is. If visitors don’t find what they’re looking for, they’ll go elsewhere.

Web sites need to be functional and usable. If your site contains exactly what your visitors are looking for, but they can’t find it because of a random, incomplete, or incoherent navigational structure, they’ll go elsewhere. If visitors find what they’re looking for, but when they fill out your contact form and hit “submit” they get an incomprehensible error message, they’ll go elsewhere.

Web sites need to be spiderable by search engine spiders, so that they can be found. If your site is extremely functional and usable, and contains great content, but your potential clients/customers can’t find you when they search for your product or service, they’ll go elsewhere.

Each of these factors can — and do — have entire books written about them. And there’s much more. In this brief post I’ve barely touched on the essential components of an effective web site. The web site owner shouldn’t need to become an expert in all of these things. But the web site developer should absolutely have more than a passing familiarity with all of the aspects of effective web site development, or should work with people who do.

Every day I see web sites developed by so-called “professionals,” for paying clients, that don’t even come close. I see sites in which every single page is a giant sliced-up graphic exported in toto out of Photoshop. I see sites that use Flash for all internal site navigation. I see sites that use frames and iframes for all or most of the site’s primary content. I see sites that have the same title and description for every single page. I see sites that return a “200 OK” header instead of a “404 Not Found” for non-existent pages. I see sites with plenty of pretty pages but virtually no content. Sometimes I see all of these things in a single site!

Attention, web site owners, and those in need of web sites: Tropical Web Works is not the only web site development firm that can develop an effective, functional site for your business. But please bear these points in mind and ensure that whoever you do hire understands these basic concepts.