Unbeknownst to me, my precinct moved since the last time I voted. I went to where I used to vote, but there was nothing there. Then I drove around aimlessly for a bit until I spotted the blue “Precinct 30″ sign.

Here in Punta Gorda, we use old-fashioned pieces of paper and have to color in little ovals with a pen. I hate that — I never could color inside the lines. Also, I officially have middle-aged eyes, and I forgot to bring my reading glasses with me, so I had to squint painfully at the printing on the ballot to read it.

The election workers were, as always, pleasant and helpful. Around here, they’re always old, too. When I get old maybe I’ll have time to work elections.

I checked the DNS settings on the server. I did traceroutes from my computer. I went to websites in other parts of the country and did traceroutes. I pinged. I double-checked and triple-checked the nameservers. Everything was exactly right, and I was tearing my hair out trying to figure out why the client couldn’t reach his own site.

Finally, it struck me: The client was searching for the site, using the search box on Yahoo and Comcast.

Once I told him that his site not indexed yet (because it’s brand new), and that he needed to enter the URL directly into the browser’s address bar — voila! Problem solved.

This is one to file away to deal with future “can’t reach my site” issues.

Note to website designers and developers: If you ever have clients who can’t reach their site, check to see if they’re searching for it.

PayPal Impersonators

Anyway, a client sent me a copy of an e-mail they had received, allegedly from PayPal, which contained “confirmation” of a purchase by the client using their PayPal account. The e-mail included a prominent link to “Dispute Transaction,” and the surrounding text instructed the recipient, “If you haven’t authorized this charge, click the link below to cancel the payment and get a full refund.”

Now, I understand that PayPal is good about chargebacks for fraudulant transactions, but I’ve never heard of them so openly inviting people to dispute a transaction. So that should have been a clue. Fortunately, when the client clicked the link to dispute the transaction, their antivirus program popped up with a warning message about it being a scam. The client then promptly contacted me to ask what they should do.

A PayPal Phishing Attempt

Undoubtedly, the link would have taken the client to a site that looked exactly like the PayPal site. There would have been instructions to log in to dispute the transaction. The client would have entered his PayPal account name and password, with the intention of disputing a fraudulent charge. Bingo! The scammer would have just got hold of the client’s PayPal login information — and there’s no telling what havoc would have been wreaked. Disaster averted — thank heaven the client had a security program installed and running on his computer.

Microsoft’s False Positive Phishing Warnings

And then there’s Microsoft. Internet Explorer 7, to be precise. With it’s much-vaunted “anti-phishing filter.” Bah, I say!

I recently started using an RSS feed from eBay to display live listings from eBay on an informational site. When I was testing the site in IE7, IE was giving me security warnings that this was a “suspicious site” and might be a “phishing site.” I know darn well it’s not — the site is clearly not eBay, it doesn’t pretend to be eBay, it’s clearly a separate site that doesn’t look anything like eBay — it merely displayed auction listings from eBay, with affiliate links to those listings on eBay. (For the record, it was my Dog Breeds Site, but I’m using the eBay feed on several sites.)

IE7 offered me the option to submit a report to Microsoft, stating that I was the site’s owner and could verify that it wasn’t a phishing site. I did this, and the next day I got an e-mail from Microsoft that they had inspected the submitted URL, verified that it was not “phishing,” and removed the warning.

Then I discovered they had only removed the warning from one page of the site. In order to remove the warning from every page that used these listings, I would have to report every page individually.

This was first of all, much too time consuming, and second, far too annoying. I would have to submit that report for every single page on every single site where I wanted to use the eBay feed. So I set about looking for what the code might contain that caused IE to pop up the phishing warning.

Here’s what I learned:

• Text links to the eBay listings didn’t trigger the warning.
• The images, pulled in directly from eBay and displayed on my site, didn’t trigger the warning.
• But the images, when linked to the eBay listings, did trigger the warning.

Hmmm ….. I didn’t want to remove the links from the images. People are naturally inclined to click on the images. After some trial and error, I discovered that if I sent those links through a redirection script, it stopped the phishing warnings cold.

Yay me!

Okay, enough of that. Today I’m going to talk about crawlability and web design. I got to browsing the Unique ID blog and read Zack’s post “Straight From Google, The Four Biggest Search Rank Factors,” in which “crawlability” is listed as the very top, highest priority, most important search engine ranking ractor for a web site. This week I’ve also been following a discussion at the High Rankings forums about whether web designers have any SEO responsibility when designing a web site.

The discussion at High Rankings opened with the story of a businessman who hired someone to design a web site for his business. The site was built in Flash, and, as might be expected, the businessman’s web site didn’t do so very well in the search engines. When he sought professional SEO help, he was flabbergasted to learn that an all-Flash site is likely to rank poorly, if at all, in the search engines.

He asked the SEO pro, “Why did the designer use Flash when he knew I wanted search engine visibility?”

A better question would be, why do designers design “search-engine hostile” web sites when they know clients want search engine visibility?

As things stand in the world of web design, anyone with some elementary graphic design skills can get themselves a copy of Dreamweaver or FrontPage and hang out their “Web Designer” shingle, offering their services for a fee to all comers.

Some of these designers do indeed have a lot of artistic talent with respect to creating pretty, aesthetically pleasing, visually attractive web sites.

What these designers lack is a fundamental understanding of the underlying code and structure of web pages, and a fundamental understanding of how search engines crawl and index web pages, and a fundamental understanding of how a web site needs to be structured in order to have a chance of getting search engine traffic.

So these web designers make a “pretty design” in Photoshop or Fireworks or Flash, and use the built-in export features from those programs to auto-generate the code or the Flash file. The client ends up with a very pretty site that hasn’t a snowball’s chance in hell of doing well in organic searches.

The web is still relatively new as a commercial medium, and there is still some level of technical knowledge required in order to build a crawlable web site. Daily we see self-labeled “professional” web designers creating all-Flash sites, or using fancy javascript-based rollover images for global navigation, or relying on other artsy-fartsy features that doom a site to search engine purgatory — a site that is uncrawlable by search engine spiders, and generally invisible in the search engines.

The client doesn’t understand why his beautiful site gets little or no search engine traffic. The client eventually discovers, if he’s lucky or persistent, that he now has to pay for his site all over again, this time to have someone else tear apart his beautiful artsy-fartsy site and re-build it using underlying code and techniques that the search engines can crawl.

Does it have to be this way?

Should it be this way?

I say no, it shouldn’t. Some people might argue that a web designer’s responsibility is to design pretty stuff, not to perform search engine optimization. That’s true up to a point — I wouldn’t posit that it’s the web designer’s responsibility to do link building or write linkbait articles or do keyword research, unless those activities are explicitly included in the agreement.

But I do argue that anyone who holds himself out as a “professional web designer” should have a broad and fundamental understanding of the technology of the medium and the factors that are required for success in that medium. I do argue that the “professional web designer” is holding himself out as an expert, and the client is relying on the expert’s knowledge and experience.

The client shouldn’t have to have a specialist’s knowledge of the medium — that’s why the client hires a professional. When I hire a contractor to build a house, I shouldn’t have to become an expert on building houses, and I shouldn’t have to give the contractor explicit detailed instructions about how to run the wiring so it doesn’t burn the house down. I expect the contractor — ”the professional” — to have the knowledge and expertise to do that himself, even if the contract doesn’t explicitly state that the contractor will run the wiring so that it doesn’t burn the house down.

In an ideal world, building contractors would always run the wiring so that it doesn’t burn the house down, and in that same ideal world, web designers would always build crawlable web sites.

The only exception I would make to this general rule is when a client specifically requests features that will cause crawlability problems, and, after being educated by the web designer about the consequences of his request, the client insists that his aesthetic vision is more important than search engine visibility. The client is paying for the site, after all. But even then there are usually steps the web designer can take to mitigate and overcome the problems — including text links in the footer to complement the pretty Flash buttons at the top of the page, for example.

Meanwhile, we live in an imperfect “buyer beware” world where the web designers who understand the medium are competing against the web designers who don’t. Clients have to educate themselves sufficiently, and ask lots of questions of potential designers, in order to be sure they end up with a crawlable web site.

But sometimes that’s simply not possible. Particularly when the potential client is looking for improved search engine performance or better usability.

Sometimes a site can be improved dramatically by minor tweaks: Add unique, custom title tags to each page, add alt text to images where needed, add text-based site navigation, beef up the content, and a few other improvements. Bang, I’m done, and the client can look for improved performance whenever the search engines see fit to recognize the changes to the site. (That might take days, it might be weeks, it might be months — I try to make sure the client is aware that I have no control over what the search engines do in that regard.)

But all too often the site is constructed so badly that nothing but a total redeveloment will do:

• Frames — frames-based sites are not only less usable for human visitors, but they still throw roadblocks in the path of the search engines trying to index the site.
• Frames Part Two — Even worse is sites that frame content from other sites. The framing site gets no credit in the SEs for the framed off-site content.
• All-Flash Sites — The search engines are working on their ability to index Flash sites, but your all-Flash site is unlikely to be the breakthrough. Flash elements should be used sparingly to add to the visitor’s experience, but Flash should not be the site.
• Search-engine-hostile Dynamic URLs– There are so many ways to go wrong here that it’s hard to list them all. Session IDs in URLs. Meaningless long numbers and section ids and category ids. URLs that display all the site’s content on the same page (using formats like “index.php?page=thispage”).It’s so easy to use the magic of server-side technology to write user-friendly and search-engine-friendly URLs, and to keep session IDs out of URLs. Which type of URL do you like better:
  • http://www.example.com/products/red-widgets.html, OR
  • http://www.example.com/index.php?cat=52 &section=355&prodid=125
&_trksid=p0.m570.l1313&hash=2394087lasmn8&sess_id=5348725987ofdj30487590fjglkae098t87q34085uofilajg

  Yeah, I thought so. Me too. The search engines like the first one better too.

• Lack of content — if there’s little or nothing for the search engines to spider, there’s little or no likelihood of any of the pages turning up in searches.
• Sites that requires the user to submit a form before seeing the content — Hey, search engine spiders don’t submit forms; they’ll never see all that great content on your site.
• Invalid tag soup — Badly coded sites with such badly formed html that it’s darn near impossible to work with the code. When I’m optimizing a site, I need to get down-and-dirty in the code, and pages with invalidly nested tables, invalidly nested divs, incomprehensible, invalid code everywhere — well, I just can’t work with it. It’s hard to even touch the code in a site like that, because you just don’t know what will happen.
• Rigid, inflexible design and code — This usually results from so-called “designers” who design a pretty image in Photoshop or Fireworks and then slice it up and export the entire thing from their image-editing program. The code for these sites is so rigid that you can’t make a single change without breaking the entire thing.
• Template sites based on some badly designed content management system — There are too many big companies out there who have built half-baked content management systems that allow anyone to “build their web site” with a few clicks of their mouse. Nice concept, but usually badly implemented. These sites are usually required to be hosted with the “big company” and runs off their database on their servers. These template sites often don’t allow custom title tags or custom descriptions for each page. They also don’t allow any access to the underlying code. This means I can’t make search-engine friendly URLs, and I can’t set up a 301 redirect, and I can’t eliminate the appearance of duplicate content, and I can’t do any of the easy tweaks that are needed.

All too often, I see sites that suffer from most or all of the above problems. I may be able to work around one or two of the above items, but when a site presents numerous serious technical problems, there’s just no point in attempting to patch the old wineskin.

It’s a shame when someone has paid good money for a site that is so badly constructed that it can’t be improved. But there are times that it makes more sense to throw out the old and build anew.

URL (pronounced you-are-ell, or sometimes “earl” as in Duke of) stands for Uniform Resource Locator. It’s the technical name for the address of a particular web page. For example, the URL of this site’s home page is http://www.tropicalwebworks.org, and the URL of this page is http://www.tropicalwebworks.org/2007/01/18/infamous-canonical-url/.

It’s common that any particular web page may be reached at multiple URLs. If this site were not configured optimally, the home page might be reachable at both http://www.tropicalwebworks.org and http://tropicalwebworks.org (notice the missing “www.”). Normal people would logically think that this would be desirable: After all, you don’t want people to get a “server not found” error if they try to get to your site without including the www part.

But Google sees these as two completely separate URLs that just happen to contain exactly the same content. There are two problems with such a situation:

1. First, the “strength” of that page, and its ability to turn up in the search engine results, is diluted. Some of the page’s strength is allotted to one version, and some to the other, and neither “page” performs as well as it would if all the strength were concentrated in one page.
2. And second, Google attempts to filter out pages containing duplicate content, based on the reasonable logic that people don’t want to see multiple results in their searches for the exact same thing. Thus, since both of these “pages” contain the exact same content, one of them will suffer in searches due to the dupe content filter.

It’s a double whammy. It’s not that your site actually has duplicate content. No, we could possibly call this situation “virtual duplicate content.” But it’s all the same to Google: It’s duplicate content, period.

And if that’s not bad enough, many people link to their home page like this: http://www.example.com/index.html. Now Google sees yet another instance of duplicate content: http://www.example.com and http://www.example.com/index.html. So ultimately what Google sees is four “duplicate content” pages:

• http://www.example.com
• http://example.com
• http://www.example.com/index.html
• http://example.com/index.html

And all this before we’ve even gotten past the home page of your site!

It’s easy-peasy to configure the server to do what’s called a “301 permanent redirect” from the non-www version to the www version of your site. This technique, which is recommended by Google, tells Google that the two are indeed the same and keeps the poor Googlebot from deciding that you have duplicate content and splitting your page’s strength among more than one version. “301″ refers to the status code that’s returned by the web server to the browser (or the spider, in this case), and it says, in effect, “Hey, the correct, permanent URL for the page you’re requesting is actually over there. Don’t index it at this URL.”

It’s likewise easy-peasy to link to your home page without the “index.html” (or other directory index name, such as home.htm or default.asp). For index pages in subdirectories, you simply link to the directory: http://www.example.com/subdirectory/, again leaving out the actual filename index.html.

I apply an appropriate 301 permanent redirect to the www version of every web site I develop. It’s not something I charge extra for, or something that I tout to my clients as being anything special. It’s about a 20-second task to set up the 301 properly. And I never link to directory index pages by filename. I don’t know why some of the big companies aren’t aware of this issue, or, if they are aware, why they don’t care enough to do it properly. It raises the question, if they’re so ignorant, or uncaring, about a thing that is so simple to do right, in how many other areas are they incompetent?

It used to be, for the average website owner, there was no easy, non-copyright-infringing way to display a map on a web site. Typically, the easiest solution was to link out to MapQuest or Yahoo, sending visitors away from the web site. Unauthorized use of map screenshots was rampant.

But Google Maps changed all that. On a site currently in development for an operator of whale-watching tours, I’m displaying a hybrid satellite/map of the Silver Bank of the Dominican Republic. Site visitors can pan and zoom the map themselves. Little pushpin markers show where the Silver Bank is and indicate the departure point for the tours. Google recently added an option to display the little overview map in the bottom right corner, too, so that you can have a close-up view while still giving the perspective of a larger-scale map. All this, without leaving the web site in question.

Norton was at it again. People download and install this software that is supposed to protect their computer from harm, but unbeknownst to them it filters out images whose filename contains the word “ad” or “banner” (or any of several dozen other words). It also filters out images inside a directory whose name contains those words. And it filters out images whose dimensions match any of the common “banner ad” sizes.

I’ve never used Norton – I work primarily on my Mac, so I don’t need third-party software to protect me from the rampant malware that’s out there — but it seems obvious me that the good folks at Norton do not make it clear to their users just what their software is doing. I’ve seen many many web site designers and developers get caught out by this program, and these people are by definition likely to be more computer-savvy than your average bear.

What’s both scary and frustrating is that people (like me) who don’t use Norton may never be aware that sites we develop are not displaying to people as intended, and the site visitors who have Norton installed don’t even know that they’re missing anything.

These factors aren’t important so much for their ability to rank a site highly, as they are for avoiding problems that can harm a site’s ranking.

Using standard href links rather than javascript links, and avoiding canonical URL problems by implementing a 301 redirect from non-www to www, won’t actually help your site rank better for any given search. But using all javascript links and having canonical URL problems can harm your site’s ability to rank well.

These technological factors fall under the “Do no harm” umbrella. Don’t throw obstacles in the path of high rankings. Removing the obstacles won’t cause you to win that race — but it will make winning possible.

Adobe’s 800-lb. gorilla, Photoshop, is of course the mother of all graphics programs. Vector graphics, raster images, photos, scanning, touching up, resizing, cropping, retouching, batch processing. Every version of Photoshop is better than the last.

But everyone knows about Photoshop, whether they use it or not. What about all those little programs and utilities that not everyone knows about?

Transmit, from the good folks at Panic, is my FTP program of choice. I was a long-time Fetch user, and never thought I’d see any reason to move away from Fetch, but Transmit hooked me a couple of years ago, and it’s done nothing but improve since then. Transmit has almost reached the point where the only way it could get more convenient to use would be if I could merely think a file to the remote server.

When I’m working in my text editor of choice (TextMate), I hit cmd-S for save, then cmd-+ to upload the file. (That’s my own custom keyboard shortcut, by the way. I didn’t find Transmit’s default keyboard shortcut convenient, so I changed it.) Anyway, cmd-+ and off the file goes. TextMate hooks into Transmit via some technology that I don’t understand, and Transmit logs into the appropriate remote server, cd’s into the appropriate directory, and uploads the file. I don’t have to switch programs. I don’t have to drill down through the directory path. Likewise, I can drag a file from the Finder onto Transmit’s icon in my dock, or I can drag the titlebar icon from any open document onto the Transmit dock icon, and it gets sent off to the right place. Folks, FTP’ing can’t get any easier than this.

More on TextMate another time. I love this little program, too.

Lately I’ve been using OmniOutliner to keep track of things. I keep one OO file that serves as my master to-do list, subcategorized by Client and Project, with deadlines and completion dates noted, and notes and files attached where needed. Another OO file serves as my master changelog, where I make a quick note whenever I make any change to any site. Many is the time I’ve needed to know, exactly when did we launch that site? or, When did I install that new script? or, When did I make those changes the client requested? OO is helping me track all of that.

And don’t get me started on StickyBrain. I think the Chronos people have discontinued development, or are planning to, but as long as my current version of StickyBrain keeps working, I’ll keep on using it for my dozens (nay, hundreds) of random notes and bits of data. It’s Sticky Notes on steroids, is what it is. I must have more than 3 dozen categories of stickies: snippets of php code, samples of complex MySQL queries, unix commands that I have trouble remembering, notes about my dedicated server, bits of css and html, boilerplate paragraphs for letters, e-mails and contracts — it’s in there. It’s searchable (and better yet, findable). It backs itself up. It keeps track of all those little things that my RAM-starved brain can’t.

I have plenty of software designed to do things. The software that helps me organize and keep track of the things I do, and makes doing those things easier and faster, is what wins my heart in the software wars.

SEO includes everything from the technical configuration of the server hosting a web site, to the site’s coding and structure, to the visible content on the pages, to the development of incoming links to the site from other sites.

The server needs to be set up so that it properly serves 404 page-not-found HTTP responses for invalid requests. If Virtual Hosting is used, the Virtual Hosts must be defined and configured properly. Any redirects in the .htaccess or httpd.conf file need to serve the appropriate “file moved” response — usually a 301 permanent redirect. Other directives in the configuration file must likewise be set up so as not to cause problems for the search engine spiders, or bots.

The underlying code on the page must present spiderable content to the bots. Occasionally when a client comes to me for help with their site, I discover that every single internal link on the site is a javascript-generated link. These links look just like standard href links to the client, but for the bots, they might as well not exist.

Keyword research needs be to done, to explore the range of possible keywords and phrases and their relative value for search targeting, taking into account the estimated number of searches, the level of difficulty in attempting to rank well for those searches, and the probable success of those searches in bringing targeted users to the site who are likely to turn into customers.
Titles must be custom-written for each page. The title must be optimized for the primary keywords the page is focused on, and must also be written with human marketing in mind, since the title is what appears as the bolded, clickable link in search engine results pages.

Visible content on the pages must be optimized for the search engine bots and for human visitors. Content must also be worthy of linking to. Link development is often viewed as “off-site optimization,” but what’s on the site is critical for this off-site optimization. You want links? Why should anyone link to you? Because of what your site offers, that’s why.

Headlines and subheads are best presented as text, despite the aesthetic limitations inherent in HTML. Using CSS to style headlines helps there. Many sites do well using graphical images as headlines and including appropriate alt text for the image, or using technical methods to show the pretty graphical headline while feeding the text to search engine bots, but it’s my opinion that nothing beats POT (plain old text) for all headings and subheadings on a page.

A good SEO professional understands all of these issues, and many more. Canonicalization issues. Duplicate content, and the appearance of duplicate content. Internal linking structures and practices that let the spiders know what the most important pages are on the site. Semantically correct code that uses h1, h2, and h3 tags appropriately for headlines.

One site may lend itself to SEO improvement by simply adjusting some of the underlying code, or the visible text on the page. Another site may require a complete overhaul to fix deeply entrenched problems. SEO is not an aftermarket add-on that you bolt on to your site after it’s complete. Good on-page SEO is built into the site from the ground up.

Sure, I can batch process the images in Photoshop to create the thumbnails and larger images, but the tools available to create the actual galleries leave something to be desired with respect to the time required and the code generated.

So I wrote a little php script that loops through the thumbnails directory and creates a grid of thumbnails, with each thumbnail linking to a popup window of the larger image.

Then I turned the thumbnail-display code into a function, and the popup window with the larger image another function.

Now, I can create new photo gallery pages by simply calling the first function, passing it the name of the directories that contain the thumbnail and large image files. The entire photo gallery is created with one line of php code in my file.

Bada-bing bada-boom — all the photo galleries the client wants, costing only a few minutes of my time for each one. My client will be happy, because they’ll save money. I’ll be happy, because I can get my work done faster, and provide faster service for more clients.

I love php.