As SEO practitioners, there’s often a good reason to exclude a certain search engine or site spider from looking at your pages. For example, some people who do not want their site archived over time by Archive.org will take action to block them from doing so. That’s where the robots.txt kicks in; set one up on your server, and most websites will adhere to your commands as far as what they are allowed to spider on your site. If you wish to prevent a specific site spider from indexing your website’s content, then configure your robots.txt file this way:

User-agent: Name_of_Robot
Disallow: /

Here are some of the most common site robot names that you might want to exclude:

Archive.org: ia_archiver
Google: googlebot
Yahoo: Slurp

 Regardless of what you’re trying to do with robots.txt, the most important thing is to make sure you’re doing it right. A small typo in your config file can cause disastrous results. So make sure to use the following resources to set yours up properly.

For thorough lists of robots files you can exclude, check these sites:

http://www.hostsun.com/gr/bots_index.php

http://www.robotstxt.org/wc/active.html

For thorough documentation as to how robots.txt files work, and how to properly set them up, check out these sites: 

http://www.robotstxt.org/wc/faq.html

http://en.wikipedia.org/wiki/Robots.txt 

Marc Rasmussen’s fascinating blog post on his recent real estate SEO struggles with Google is a must-read for anyone doing real estate website optimization. It shows how you can really get caught with your pants down if you engage in linking schemes. And it also demonstrates how difficult it is to get your clients back in the SERPs if they are taken down.

Matt Cutts’ blog has some fiesty banter going on about Google’s real estate “shot across the bow,” specifically who it targeted. What Marc and many others are suggesting are that Google’s policy enforcement team are not penalizing all real estate websites equally for reciprocal linking.

What we will say regarding all of this is that Google did not penalize our real estate websites at all, simply for the fact that we never engaged in linking schemes because they are set against improving the overall user experience. Unfortunately people think they can get ahead and never get penalized for it. Google has shown in the past they won’t tolerate black hat SEO tactics that go against their webmaster guidelines, and this is simply another case of policy enforcement.

If you’re saying, “what’s wrong with having state pages on a Florida real estate website for out of state referrals?” Unless your site is a national vacation home referral website, there’s no reason to have state pages on it. The more relevant website will always be a website pertaining to a specific state. And why should a website that has more quality information on real estate in Florida be lower in the SERPs than a Florida site with less quality information on Florida and a bunch of information pages on other states with spammy reciprocal links?

Our ultimate hope is that Google doesn’t cave into these people asking for full re-inclusion because how is it fair to the people who play the game the way it’s supposed to be played if offenders can just find a sink, wash their hands, and then get back in where they started?

If you aren’t already using social bookmarking sites like Digg to advertise your great blog postings, then you might as well be writing your entries on a barn wall in rural Iowa with a red Sharpie because no one will read it. No matter how good a writer you are, most stuff will lose its freshness long before the search engines pick it up naturally. Social bookmarking sites step in and allow readers to review posts when they are recommended. People who are good at picking relevant, interesting/helpful articles end up getting other readers to subscribe to their choices. So if you’re writing about website SEO, then you want some good SEO experts to start recommending your work. It’ll lead to a whole lot of other people reading your posts. That’s why we’ll give your posts a good square kick in the marketing rear by getting you in the habit of submitting your posts to a couple of social bookmarking sites whenever you blog.

The good news is that it only takes a few minutes to register an account. We recommend picking a login name that contains keywords because an SEO article recommended by someone who has the name SEOMarketing looks like, at least on first glace, that it has more promise than one “dugg” by JoeBlow.  When you recommend articles, the title and description don’t have to match the exact blog post. So you can use that opportunity to throw a good key phrase in there or two, and really make the post pop out any way you can. The goal with social bookmarking is to pique the curiosity of someone so they read your description, then travel to the blog, and also “digg” the article so that the hype machine continues to run. As with any kind of promotional campaign, you could lose people at any step of that process; so you’ll ultimately want to optimize each step to maximize your chances of getting more favorable reviews of your blog.

Finally, one thing you’ll want to do is some research. What kinds of articles are getting “dugg” more frequently in your industry. Of course you don’t want to retread, but it never hurts to steal some topical thunder and riff off someone else’s success story. Whatever you do, make sure all the posts that you suggest to the bookmarking site are relevant to each other, and also make sure to recommend other articles you find that are helpful; it makes your account seem less biased.

If you aren’t using Archive.org while performing your domain name SEO research, then you certainly should be… or maybe not for long? It looks like some clever chap has found a way through robots.txt to undermine the Internet Archive Wayback Machine so that the site history cannot be viewed. I won’t go into too much detail because it is a site that we are considering purchasing, however there are some interesting entries in the site’s robots.txt file:

Disallow:/cgi-bin
Disallow:/cgi-bin/link.cgi
Disallow:/cgi-bin/links.cgi
Disallow:/cgi-bin/Links.cgi
Disallow:/cgi-bin/Count.cgi
Disallow:/cgi-bin/ls1.pl
Disallow:/servlet/UserPrice
Disallow:/servlet/PrivateSale
Disallow:/servlet/buyDomDLS
Disallow:/ticker.jsp
Disallow:/searchAdvanced.jsp
Disallow:/searchPrice.jsp
Disallow:/searchPremiums.jsp
Disallow:/searchRegister.jsp
Disallow:/jsp/search_advanced.jsp
Disallow:/jsp/search_advanced_text.jsp
Disallow:/jsp/search_category.jsp
Disallow:/jsp/search_expired.jsp
Disallow:/jsp/search_keyword.jsp
Disallow:/jsp/search_letter.jsp
Disallow:/jsp/search_namefind.jsp
Disallow:/jsp/search_pricerange.jsp
Disallow:/jsp/search_similar_domains.jsp

It looks like a lot of these entries are referring to apps that crawl the web looking for expired domains, or domains for sale. At any rate, for the price this particular site owner is asking for the site in question, this use of robots.txt to prevent prospective buyers from seeing the history of a site seems a little shady. We’re still thinking about buying the domain because it’s a great name, but it’s more difficult to gauge the value of a site we don’t know the history of. And just the fact that this owner would go to such lengths to prevent people from seeing the history of the site does not reassure BigSERP’s confidence in a possible transaction.

What this means is that people parking domains, buying sites for resale, or vulturing for deleted sites now have a powerful tool in their arsenal to potentially deceive honest people looking for a good name. Who knew, it turns out robots.txt is more valuable than just a piece of code that most sites ignore!

How do I write effective HTML title tags?

In less competitive industries/markets, sometimes all that’s needed for first-page placement on key terms is a simple website with plenty of SEO content and… correctly labeled title tags! When done right, your page title tags (along with the META keywords and description tags) assist the search engines in properly indexing your content. On the other hand, if handled poorly or without thought, title tags can be one of your worst SEO enemies, causing your pages to either be misinterpreted or worse yet be thrown into duplicate content filters. Both of these things result in the search engines disregarding your pages when they consider the most relevant sites for the key phrases that are most important to your business. With so much riding on this 63 character HTML element, let’s make sure and get you off on the right foot with configuring your title tags. Here are some helpful SEO tips that will optimize those tags for you:

1. Be wary of the length of your title tags. On the Search Engine Results Pages (SERP) of most of the major search engines, the title tag is used as the first line to identify your website. This line is typically in bold face, and hyperlinks to your site. Now all the search engines have different line character limits before they begin to truncate your title. That’s why it’s important to get the most vital keywords into the first 63 characters. There’s a good chance that everything after that will be replaced by ellipses.

2. Don’t put your corporate branding first. There are exceptions to many rules, and there is one in this case, but unless your company name is already extremely popular and competitive as a key phrase, there’s no reason to put your company branding first and foremost in your title tag. Best practice is to put the company name at the end of the title, after a dash. This way, your site has the best chances of being indexed for the actual key phrases that people are most likely to type into the search engines. And your pages will still get indexed for your company name, especially if it’s a unique one.

3. Prioritize your key phrases. For each page of content, which should be written to emphasize a main key phrase, and several secondary phrases, you should configure the tags to reinforce the importance of those phrases. Always place the most important key phrase, in the exact order that it is typed into the search engine, in the very front of your title tags. Then use commas to separate out the remaining key phrases that are relevant to this page of content, ordering them from most important to least.
(Note: Something that’s at least fun to consider with this is checking out your competition for specific key phrases. If you type in a phrase, and notice that your competition is showing up for that phrase, and all of their titles begin exactly the same, you may want to try and at least switch up the first key word so that when your page displays differently from the crowd. What usually works best is to add original content pages of all kinds of different variations of the phrases you want to show up for, so that increases your chances of having your page appear differently than the others in the SERP.)

4. Use synonyms and stems. The search engines are pretty smart these days. Most of them can equate “websites” with “site,” “sites,” “website,” “web pages,” and so on. So don’t be afraid to use stems of words or synonyms in the second half of your title tags. For example, you could use “car website” early on in your title tag, and then use “car site” later on. Definitely do a little research on your competition and see what synonyms and stems they are having success using, and then do some of your own experimentation. If you need help determining what search engines consider synonyms, Google actually has a cool feature built into it. If you place a tilde (~) in front of a search term, the SERP will return a page full of relevant sites with the synonyms highlighted.

5. Eliminate unnecessary words. Connectors like “the,” “for,” and “a” are disregarded by many search engines, so there’s no need to waste precious real estate in your title tags for them. Eliminate them wherever it makes sense to.

6. Don’t completely ignore formatting. In the end, the title tag is what searchers are going to see as the representation of your page. So when you’re finished considering all the previous tips, give your title a final look and see if there aren’t a couple of small formatting touches you can make to polish the look of the title up. Adding a colon or a connector here and there can make a comma-separated list of keywords look more like an ad, and will increase the chances that once you actually get on the first page, that someone will clickthrough to you.

If you utilize these tips to the fullest, and get plenty of practice, your content will be indexed properly, and will receive the best consideration by the search engines for placement in the SERP. Don’t forget to check out the other free SEO tips provided on BigSERP.Com, and don’t hesitate to contact us if you need professional assistance with optimizing your entire site’s title and META page tags.

This tip is for those of you running an online business that has managed to achieve the holy grail of internet marketing: first page natural placement for terms that are highly relevant to your products and services. Some frustrated site owners, out of concern for a lackluster website design, or a re-branding of some sort, will make the worst mistake possible… they will mess with success! Anyone who has learned this harsh lesson first-hand will tell you that it doesn’t take much for the search engines to re-index your site. And once your site goes into that re-indexing purgatory, who knows how long it’ll take to get back. If it gets anywhere near where it was before. Trust us, it’s not worth risking first page placement to rebrand your site, or tweak with your tags.

Here are just a few things that can cause your site to be re-indexed:

1. Changing page tags, including the page title and META tags.

2. Completely changing the content on your home page, especially if that content contains links.

3. Changing your homepage layout, especially if it reorders your links and content on the page.

So be ultra careful whenever you change any of these things, because once you lose that prime positioning, it is very hard to get back. This is especially true for sites that do not have a ton of inbound links established.

Our recommendations are for you to consult with a professional. Often times, the best solution involves developing another site and using your old one to boost the new one, or taking position reinforcing measures before making major changes to your site. But the best course of action is always found after carefully considering your business and its marketing goals. Please let the SEO experts here at BigSERP.Com assist you with a free consultation. We can turnaround your current campaign, or custom design a successful solution from scratch for you.

In our line of work we must get asked about it hundreds of times a year, so it makes sense to dedicate a post to the topic: getting your site 24/7 coverage on the search engines. Most want to know what it takes to get to that point. We’ll run you through a common mistake, and then the best strategy.

Many of our PPC clients initially don’t opt into any natural strategies (despite our warnings) because most of them have a large up-front cost associated with them. These same clients find themselves wondering why their ads aren’t serving all the time at top positions in the SERP. It’s hard for many to accept, but quite often ad budgets simply will not allow for day and night coverage for those highly-relevant, coveted terms. What’s even worse about this is that the reaction of some of these clients is to then throw more money at PPC, forcing their way to the top against our recommendations. In the end, they end up with a relatively small amount of high-quality traffic that isn’t cost-effective, and has no long-term value to the site. Don’t make the same mistake they quite often do: let us implement for you a blended online marketing strategy that gets your site ranked on the first page twice!

Of course, we’ll still set up a PPC account for you. But instead of going for position #1, we’ll do some “position surfing” to get you quality traffic at a price that won’t break your bank each month. Then we’ll take an initial budget and dedicate it to two things that your marketing strategy should not be without: highly relevant SEO content that targets specific key phrases that you want to show up for in the SERP, and link building campaigns that will generate valuable inbound links to your site from high ranking, relevant sources. And then we’ll re-allocate part of your PPC budget to both content and link maintenance, meaning we’ll continue to build on the robust content and rich inbound links coming into your site on a monthly basis.

The end result, as we have seen for so many of our clients, is first page placement not only on the PPC results, but also in the natural SERP. And when you have both of those, it more than doubles the chances you’ll get a click, and also keeps a competitor out of one of those slots!

So definitely utilize this combined marketing strategy to the fullest. And don’t hesitate to contact us if you’d like a quote on having BigSERP professionally set it all up for you… we’re here to help you achieve your online marketing goals.

Ask different SEO professionals about the relevance, importance, and impact that URLs have on natural rankings and you will most definitely get differing answers, opinions, and suggestions. Until the major search engines post their inner-workings online for all SEOers to read, then the question should not be whether having relevant keywords in your domains is actually doing anything, but rather what are the benefits of having relevant keywords in your domains.

One school of thought is that by having relevant keywords in the domain, the search engines will see this as a more relevant and appropriate search result for search queries. As time and algorithms change, so follows this common misconception. It is now much less important for a domain to have relevant keywords in it for search engine rankings. In addition, with the evolution of dynamic websites, implementing highly relevant domains is becoming more and more difficult. However, before you give up on your domain buying efforts and start selling your existing domains with relevant keywords keep in mind that creating URLs with relevant keywords is NOT obsolete or completely unnecessary.

Here are some reasons to feel a little better about the 100 plus keyword stuffed domains you might be sitting on:

1) Having relevant keywords in your domain still has some beneficial effects on natural rankings in and of itself as the search engines have not completely denounced these efforts yet.

2) Consider the ad copy that appears in the SERPs: the domain name occupies just about 20% of the very valuable real estate in each SERP result. No matter how you see it, one cannot deny its function as a call to action and the increase it provides to the searcher’s perceived relevance.

3) The simple fact that keyword based URLs are much easier to remember than long, irrelevant URLs is one reason that you should snatch up any domains that contain your company most relevant keywords.

4) Since search engines give credit to anchor text with keywords in them and anchor text usually cites the domain, having keyword based domains and anchor text with these keyword embedded domains will provide some search engine ranking help.

5) The psychological effects of seeing what you are looking for in a display url in a SERP result in a way acts as a call to action. Think about it, if you are searching for “los angeles real estate” and you get results with these domains:

www.realtypros.com
www.mrla.com
www.losangelesrealestate.com

Which one are you going to click on?

Exactly!