This is a two-edged sword. First, they may do something inadvertently to obscure a site they didn’t want to obscure. Second, they don’t know how to help a client achieve that rare status of being on the web but only visible to people “in the know.” Therefore, there are reasons on both sides of this divide to know about securing and obscuring your site.
In no particular order, here are five ways of securing and obscuring your site when you want a bit of privacy:
Pull-down lists:
These are a good way to indicate to the visitors who know where your site is just what it contains, without giving up the goods to search engines. You can make links to the other important pages but the search engine spiders can’t see that text in the pull-down list. In addition, they cannot follow the links and index anything else you are surreptitiously pointing to behind the menus.
If your site does not have pull-down menus, you can add them fairly quickly without having to tear down and rebuild your entire site. This is a quick fix but perhaps not so comprehensive, so you might combine it with other maneuvers.
Text in images or media
Until recently, search engines could not read the text embedded in Flash animations and video clips. They weren’t friendly to web crawlers, but they didn’t play nice with your internal search, either. Although Adobe (which owns Flash) and other companies are now adding textual metadata into the motion graphics—something Silverlight has been doing since it was introduced—this method of obscuring/securing may be going the way of the dodo.
Of course, if you are not using the latest version of Flash, the spiders won’t find the hidden content. How long you will be able to do so is unclear, since ongoing upgrades may force your hand here. Remember, too, that HTML has always been able to associate image and text with the "ALT" tag. If you want to skulk around the web, don’t add them, although you will run afoul of the Americans with Disabilities Act that “encourages” you to help vision-impaired people using web audio readers.
Custom-configured robots.txt
The robots.txt files is a de facto web standard that gives you a way to “uninvite” at least the “polite and participating” spiders and crawlers from your site. Of course, an incorrectly configured one could also make you miss things in your own internal search indexing.
Also, robots.txt is not really a globally accepted standard, and deferring to its requests is optional for each spider. The ones from Google and Yahoo! are polite, but less etiquette-oriented ones can simply ignore your requests. To resolve the internal/external problem, you can change robots.txt to assert different rules on the different spiders. With a little work you can allow your internal search engine access wherever you want, while blocking the external ones.
Dynamic URLs, JavaScript
JavaScript and ASP are widespread technologies that enable local processing and they are not, in themselves, problematic. Developers can use them in certain ways to achieve obscurity/security, though.
You could have some local code, the function of which is to create a URL for redirecting user requests. You could make it so the URL to follow is not visible as a static link, in which case spiders will not be likely to find the referenced page.
There are other methods that may work for the purpose of obscuring or securing your content, such as forms-based secure logins, etc. If you are creating a new site and do not plan ahead for this purpose, the landing page you create in most WYSIWYG (What You See Is What You Get) website applications will, by default, set up spider-friendly pages.
You must identify the content you wish to keep “behind the scenes” and employ special methods for doing so, both by coding and cookie-cutter steps, to make sure the goods are safe from prying eyes. You should do a lot of testing, too, since you cannot improve what you cannot test and measure. Also, be certain of what you are doing, because you may face the prospect of a complete site redesign if you change your mind.