First, ugly sites are often sites that were originally built back in the 1990s by some business owner who had no web design sense or technical knowledge but wanted to promote his business online. It was much more difficult back then to actually sell online (not many shopping cart scripts existed, and ones that did exist were very expensive). So the business owner didn’t just install a cart and sit back and wait for orders to roll in. Rather, he put valuable and unique content on his site. Over the years, he added and added and added more and more content, and garnered lots of link love, and now his site is very well aged and very well linked.

MacGregor has a content-rich, incredibly useful, but butt-ugly site.

The MacGregor Sailboatsite is a good example of this. It was built by Roger MacGregor himself, who designed the MacGregor sailboat and who owns MacGregor Yachts. It’s a horrible horrible site from the perspective of design, usability and navigation. It does all of that completely wrong. But it’s packed chockful of everything you ever wanted to know about the MacGregor, and MacGregor owners love that site.

Second, ugly sites are often sites that were built by hobbyists — people who are simply passionate about a topic, and started their site without any intention of making money from it. Before eBay, before CJ, before AdSense, these hobbyists were out there creating volumes of good content about their topic of interest.

They, like the business owner above, garnered lots of link love, and today have well-aged and well-linked sites. Perhaps they’ve added AdSense or affiliate links on their site, and now they realize some hefty income from their hobby site. It’s icing on the cake for them; they didn’t start the site to make money, and if the money dried up they would continue to maintain the site. They didn’t know or care about design, and they still don’t. But they offer some of the best resources for information about their hobby. Non-profit organizations often fall into this category too. A Page About Freemasonry is a good example of this type of site. It was created by a guy who is a Mason and who loves Masonry, and wanted to share information about it. The site is nothing special, design-wise, but it contains scads of good info about Masonry.

Third, ugly sites that are ecommerce sites are often built by small business owners who can’t or won’t pay a professional to build their site. They use whatever free software they can get their hands on, and they do it themselves. They care deeply about their business, but they have no clue about web site development.

These sites can do well because the owner’s personality and passion very often shine through loud and clear. People get a sense for how much the owner cares about his product and business, and so they trust the site enough to buy from it despite its lack of professionalism and design aesthetic. These sites are often created by business owners who are very active within the community of people who use their products — so they’re known within the industry, and people feel comfortable buying from them.

AmmoMan is a trusted source of ammunition in the gun community

AmmoMan is a good example of this type of site. My word! I would never deliberately design an ecommerce site to look like that! But lots of people who shoot know AmmoMan is a good place to buy ammunition, and lots of shooters have met the owner at various shoots and gun shows over the years.

So…. yes, ugly sites can do very well. But in my experience, whenever I’ve taken on a client with an ugly site and redesigned it, it did better than before. My redesign will typically also improve search engine crawlability, usability, navigation, etc., so it’s not an apples/apples comparison. But it provides evidence that ugliness, by itself, is neither required nor sufficient for a well-performing site. When ugly sites perform well, it’s typically not because of their ugliness but in spite of it. People are very forgiving of ugliness when the site gives them the information they couldn’t find anywhere else, or sells them the product they need at a better price than they can get anywhere else.

I’m currently participating in a discussion on the SitePoint forums in which the topic of discussion is a web site developer who offers “SEO friendly CMS” (content management system) — but this developer builds his clients’ sites in frames. Not only does he use frames, but the framed content is actually hosted on his domain, not his clients.

There are many, many reasons this is a terrible idea. I won’t go into those here.

What I want to talk about is the statement by one forum member who wrote, “When the pages are indexed, yes they’ll show his domain but unless the client has specifically asked for that not to happen it’s not unethical.”

That’s completely wrongheaded thinking, in my opinion. Clients should not have to know enough about the website building process, the architecture of html pages and framesets, and the inner workings of search engines to request that the developer avoid certain practices that would be bad for the client’s site.

The client hires the web developer because the web developer allegedly has expertise in this area. Any monkey can peck at keys on a keyboard; it’s not the web developer’s time-on-keyboard the client is in need of. It’s the web developer’s expertise.

When I hire professionals or specialists to perform a service for me, I rely on their expertise. I don’t think I should have to study up so that I can tell my mechanic how much torque to use when tightening the tires on my car, or tell my plumber what not to do in order to avoid damaging my plumbing system, or tell my doctor which medicines are contraindicated, or provide my lawyer with the exact wording necessary for a business contract. I hire those people because they supposedly know what they’re doing. They’re supposed to know how to do the things they’re hired to do, and they’re supposed to know what techniques and processes to avoid due to potential damage.

In a similar vein, I believe that anyone who holds him or herself out as a professional website developer — and particularly one who claims “SEO friendly” anything — has a responsibility to perform the promised service using industry-accepted “best practices” without regard to whether the client has sufficient knowledge to request those best practices specifically.

I believe, in fact, that it is unethical for a website developer to engage in the types of practices discussed in the SitePoint forum thread, and in many other practices as well.

But unfortunately, from the client’s point of view, I don’t have a solution to offer. The client, who is usually not an expert in these things, doesn’t even know the right questions to ask, much less have any way of evaluating the answers. It’s all too easy for the unethical developers to give impressive-sounding, baffle-’em-with-bullshit, high-falutin’ answers. It’s easy to talk the talk; but how does the client assess whether the developer walks the walk?

A lot of the things the client should be assessing aren’t obvious from looking at the developer’s portfolio, unless the client himself becomes an expert on web technology. It’s not rocket science, but in my experience, most clients just don’t have the time or the inclination to do that. They’re too busy running their businesses.

I generally recommend that they use WordPress and host it on their own site — usually in a subdirectory on their site’s domain, e.g., example.com/blog, or perhaps on a subdomain, e.g., blog.example.com.

There are several reasons I lean toward hosting your own blog on your own site:

1. Reliability and Safety: I dislike, in general, the idea of having your material dependent on a third-party service where the security and reliability of your blog is at the mercy of that third party. A few years ago one of the major hosted blogging services closed up shop — unexpectedly and unnanounced — and stole away in the night. All of the bloggers who had been using that service found their blogs gone with no backups available and no way to re-create their work elsewhere. With your own copy of WordPress running on your own site, your blog is totally portable — you can host it anywhere you wish, and you can easily set it up elsewhere if your hosting disapppear That’s provided you have a backup, of course — when my clients host with me, there’s always a backup available, because I run a cron job that backs up all the databases nightly, and my hosting provider runs daily backups, as well. So none of my clients will ever find their site, or their blog, gone with no backup. All of the remotely hosted blogging platforms have a terms of service (TOS) that disclaims any responsibility for your material.
2. Link Juice: Assuming you do a good job writing your blog, and it attracts inbound links from other sites or blogs, links to your blog at example.com/blog are likely to help the site as whole rank well in the search engines, whereas a link to yourblog.blogspot.com isn’t going to have the same effect.
3. Branding: When your blog is hosted on your own domain, people who visit and read your blog can’t help but be aware of the rest of your site. But when your blog is at yourblog.blogspot.com, visitors might have no idea that you even have a site of your own, even if you add links to your site in the blog sidebar.
4. Flexibility: WordPress has a lot of functionality “out of the box,” and much more available through a variety of free plug-ins. With the remotely hosted services, you can only do what they allow.

With that said, I’ve recently been exploring Blogger, the blogging platform now owned by Google, and I have to say I’m pretty impressed. There’s more flexibility and functionality available than I was aware of, and the set-up is even faster and easier than WordPress, which is famous for its 5-minute installation. The WYSIWYG editor in Blogger for writing posts seems to be a little easier to use than WordPress’s. And Google loves it own blog service — A blog that I set up there had its first post found and crawled by GoogleBot the very same day I posted it.

Nevertheless, I shall continue recommending to clients that they host their own blog on their site. I think the advantages of self-hosting far outweigh the advantages of hosting on Blogger or any other blogging service.

Okay, enough of that. Today I’m going to talk about crawlability and web design. I got to browsing the Unique ID blog and read Zack’s post “Straight From Google, The Four Biggest Search Rank Factors,” in which “crawlability” is listed as the very top, highest priority, most important search engine ranking ractor for a web site. This week I’ve also been following a discussion at the High Rankings forums about whether web designers have any SEO responsibility when designing a web site.

The discussion at High Rankings opened with the story of a businessman who hired someone to design a web site for his business. The site was built in Flash, and, as might be expected, the businessman’s web site didn’t do so very well in the search engines. When he sought professional SEO help, he was flabbergasted to learn that an all-Flash site is likely to rank poorly, if at all, in the search engines.

He asked the SEO pro, “Why did the designer use Flash when he knew I wanted search engine visibility?”

A better question would be, why do designers design “search-engine hostile” web sites when they know clients want search engine visibility?

As things stand in the world of web design, anyone with some elementary graphic design skills can get themselves a copy of Dreamweaver or FrontPage and hang out their “Web Designer” shingle, offering their services for a fee to all comers.

Some of these designers do indeed have a lot of artistic talent with respect to creating pretty, aesthetically pleasing, visually attractive web sites.

What these designers lack is a fundamental understanding of the underlying code and structure of web pages, and a fundamental understanding of how search engines crawl and index web pages, and a fundamental understanding of how a web site needs to be structured in order to have a chance of getting search engine traffic.

So these web designers make a “pretty design” in Photoshop or Fireworks or Flash, and use the built-in export features from those programs to auto-generate the code or the Flash file. The client ends up with a very pretty site that hasn’t a snowball’s chance in hell of doing well in organic searches.

The web is still relatively new as a commercial medium, and there is still some level of technical knowledge required in order to build a crawlable web site. Daily we see self-labeled “professional” web designers creating all-Flash sites, or using fancy javascript-based rollover images for global navigation, or relying on other artsy-fartsy features that doom a site to search engine purgatory — a site that is uncrawlable by search engine spiders, and generally invisible in the search engines.

The client doesn’t understand why his beautiful site gets little or no search engine traffic. The client eventually discovers, if he’s lucky or persistent, that he now has to pay for his site all over again, this time to have someone else tear apart his beautiful artsy-fartsy site and re-build it using underlying code and techniques that the search engines can crawl.

Does it have to be this way?

Should it be this way?

I say no, it shouldn’t. Some people might argue that a web designer’s responsibility is to design pretty stuff, not to perform search engine optimization. That’s true up to a point — I wouldn’t posit that it’s the web designer’s responsibility to do link building or write linkbait articles or do keyword research, unless those activities are explicitly included in the agreement.

But I do argue that anyone who holds himself out as a “professional web designer” should have a broad and fundamental understanding of the technology of the medium and the factors that are required for success in that medium. I do argue that the “professional web designer” is holding himself out as an expert, and the client is relying on the expert’s knowledge and experience.

The client shouldn’t have to have a specialist’s knowledge of the medium — that’s why the client hires a professional. When I hire a contractor to build a house, I shouldn’t have to become an expert on building houses, and I shouldn’t have to give the contractor explicit detailed instructions about how to run the wiring so it doesn’t burn the house down. I expect the contractor — ”the professional” — to have the knowledge and expertise to do that himself, even if the contract doesn’t explicitly state that the contractor will run the wiring so that it doesn’t burn the house down.

In an ideal world, building contractors would always run the wiring so that it doesn’t burn the house down, and in that same ideal world, web designers would always build crawlable web sites.

The only exception I would make to this general rule is when a client specifically requests features that will cause crawlability problems, and, after being educated by the web designer about the consequences of his request, the client insists that his aesthetic vision is more important than search engine visibility. The client is paying for the site, after all. But even then there are usually steps the web designer can take to mitigate and overcome the problems — including text links in the footer to complement the pretty Flash buttons at the top of the page, for example.

Meanwhile, we live in an imperfect “buyer beware” world where the web designers who understand the medium are competing against the web designers who don’t. Clients have to educate themselves sufficiently, and ask lots of questions of potential designers, in order to be sure they end up with a crawlable web site.

Developing a new web site — or re-developing an old one, for that matter — typically involves consulting with the client to determine the site’s target audience and primary objective, creating an attractive and functional design, turning the design into properly coded, valid html, building out the pages of content, and writing the server-side programming to perform whatever dynamic features are needed. But there are several steps that are frequently overlooked before a site “goes live.”

A custom “404 not found” page should be created. At minimum, it should incorporate the site’s overall design and navigation links, and might also include a search form, links to the most popular sections of the site, and/or a way to contact the site owner or webmaster for assistance. And make sure that requests for non-existent pages actually get a “404 page not found” server response. If any other server response is returned — particularly a “200 OK” response — the site could easily become persona non grata in the search engines, among other problems.
A robots.txt file should be created to tell the search engine spiders what pages or parts of the site should not be spidered. Even if you want every page to be spidered, a robots.txt file should be placed in the document root of the site, so as to avoid filling up the site’s error logs with “not found” entries for a non-existent robots.txt. This makes it much easier to spot errors resulting from actual bad links when you examine the error logs.

A canonical URL redirect should be implemented to send all site traffic to your desired canonical URL — either www.example.com, or just example.com (without the www). Whichever you prefer, your should make sure that all traffic to the other form is redirected via a proper 301 redirect.

Test all forms and other interactive features. Submit every form. Attempt to submit forms without required information, or with invalid information. Try to break them. Try really really hard to break them. And make sure that whatever is supposed to be done after the form is submitted actually happens. If an e-mail is supposed to be sent to the site owner, test it, and make sure the owner gets that e-mail with all the appropriate information. If there’s a search engine, search for some things. If the site relies on cron jobs, set them up ahead of time, and make sure they’re running as scheduled and performing as expected.

Test all redirects and rewrites. If the site uses Apache’s mod_rewrite module to present search-engine-friendly URLs, test them all, and test the non-search-engine-friendly versions, to make sure that every bit of content can only be reached by one unique URL.

Check subdirectories for directory listings. Make sure that directory indexes are turned off, and/or for any subdirectory that doesn’t have an index page, plop one in there.

Test all internal links and all outgoing links. Make absolutely sure that every single link leads to the right place. We don’t need to stinkin’ dead links!

There’s simply no excuse for ever launching a site without having completed each of these steps.

Web sites need to contain good content. It doesn’t matter how many people “ooh” and “aah” over how pretty your site is. If visitors don’t find what they’re looking for, they’ll go elsewhere.

Web sites need to be functional and usable. If your site contains exactly what your visitors are looking for, but they can’t find it because of a random, incomplete, or incoherent navigational structure, they’ll go elsewhere. If visitors find what they’re looking for, but when they fill out your contact form and hit “submit” they get an incomprehensible error message, they’ll go elsewhere.

Web sites need to be spiderable by search engine spiders, so that they can be found. If your site is extremely functional and usable, and contains great content, but your potential clients/customers can’t find you when they search for your product or service, they’ll go elsewhere.

Each of these factors can — and do — have entire books written about them. And there’s much more. In this brief post I’ve barely touched on the essential components of an effective web site. The web site owner shouldn’t need to become an expert in all of these things. But the web site developer should absolutely have more than a passing familiarity with all of the aspects of effective web site development, or should work with people who do.

Every day I see web sites developed by so-called “professionals,” for paying clients, that don’t even come close. I see sites in which every single page is a giant sliced-up graphic exported in toto out of Photoshop. I see sites that use Flash for all internal site navigation. I see sites that use frames and iframes for all or most of the site’s primary content. I see sites that have the same title and description for every single page. I see sites that return a “200 OK” header instead of a “404 Not Found” for non-existent pages. I see sites with plenty of pretty pages but virtually no content. Sometimes I see all of these things in a single site!

Attention, web site owners, and those in need of web sites: Tropical Web Works is not the only web site development firm that can develop an effective, functional site for your business. But please bear these points in mind and ensure that whoever you do hire understands these basic concepts.

But sometimes that’s simply not possible. Particularly when the potential client is looking for improved search engine performance or better usability.

Sometimes a site can be improved dramatically by minor tweaks: Add unique, custom title tags to each page, add alt text to images where needed, add text-based site navigation, beef up the content, and a few other improvements. Bang, I’m done, and the client can look for improved performance whenever the search engines see fit to recognize the changes to the site. (That might take days, it might be weeks, it might be months — I try to make sure the client is aware that I have no control over what the search engines do in that regard.)

But all too often the site is constructed so badly that nothing but a total redeveloment will do:

• Frames — frames-based sites are not only less usable for human visitors, but they still throw roadblocks in the path of the search engines trying to index the site.
• Frames Part Two — Even worse is sites that frame content from other sites. The framing site gets no credit in the SEs for the framed off-site content.
• All-Flash Sites — The search engines are working on their ability to index Flash sites, but your all-Flash site is unlikely to be the breakthrough. Flash elements should be used sparingly to add to the visitor’s experience, but Flash should not be the site.
• Search-engine-hostile Dynamic URLs– There are so many ways to go wrong here that it’s hard to list them all. Session IDs in URLs. Meaningless long numbers and section ids and category ids. URLs that display all the site’s content on the same page (using formats like “index.php?page=thispage”).It’s so easy to use the magic of server-side technology to write user-friendly and search-engine-friendly URLs, and to keep session IDs out of URLs. Which type of URL do you like better:
  • http://www.example.com/products/red-widgets.html, OR
  • http://www.example.com/index.php?cat=52 &section=355&prodid=125
&_trksid=p0.m570.l1313&hash=2394087lasmn8&sess_id=5348725987ofdj30487590fjglkae098t87q34085uofilajg

  Yeah, I thought so. Me too. The search engines like the first one better too.

• Lack of content — if there’s little or nothing for the search engines to spider, there’s little or no likelihood of any of the pages turning up in searches.
• Sites that requires the user to submit a form before seeing the content — Hey, search engine spiders don’t submit forms; they’ll never see all that great content on your site.
• Invalid tag soup — Badly coded sites with such badly formed html that it’s darn near impossible to work with the code. When I’m optimizing a site, I need to get down-and-dirty in the code, and pages with invalidly nested tables, invalidly nested divs, incomprehensible, invalid code everywhere — well, I just can’t work with it. It’s hard to even touch the code in a site like that, because you just don’t know what will happen.
• Rigid, inflexible design and code — This usually results from so-called “designers” who design a pretty image in Photoshop or Fireworks and then slice it up and export the entire thing from their image-editing program. The code for these sites is so rigid that you can’t make a single change without breaking the entire thing.
• Template sites based on some badly designed content management system — There are too many big companies out there who have built half-baked content management systems that allow anyone to “build their web site” with a few clicks of their mouse. Nice concept, but usually badly implemented. These sites are usually required to be hosted with the “big company” and runs off their database on their servers. These template sites often don’t allow custom title tags or custom descriptions for each page. They also don’t allow any access to the underlying code. This means I can’t make search-engine friendly URLs, and I can’t set up a 301 redirect, and I can’t eliminate the appearance of duplicate content, and I can’t do any of the easy tweaks that are needed.

All too often, I see sites that suffer from most or all of the above problems. I may be able to work around one or two of the above items, but when a site presents numerous serious technical problems, there’s just no point in attempting to patch the old wineskin.

It’s a shame when someone has paid good money for a site that is so badly constructed that it can’t be improved. But there are times that it makes more sense to throw out the old and build anew.

If this were not so, I would have started this blog long before now. Work on client sites always comes first. I finally decided to just do it.

Tropical Blogging will focus primarily on web site design and development, search engine optimization, technology, html and css standards, hosting, and other Internet-related topics. Most of the discussion will be aimed at the type of people who comprise the primary clientele of Tropical Web Works — smart businesspeople who are not necessarily steeped in the arcana of these topics. We might stray, from time to time, into politics, pop culture, current events, or other off-topic topics. But that’s okay. This is my blog, and I can write about whatever I want.

I’ve always tended to be more of a lurker than a talker, both online and off. But I frequently have invigorating and animated conversations in my head. I’m hoping to use this space to pour out some of that thinking here — expose it to the world, for your amusement or edification. Either way, it’s time to step out of the sidelines and join the blogosphere, and this space is where I’ll put my thoughts and ideas out there, front and center. Do with it what you will.