Google API’s
Lately, I’ve been getting a few questions about the Google API’s. Although mostly intended for Web developers, search engine implementations or Internet applications that query the Internet using the Google database, Google’s API could be of some help for you, depending on your general level of knowledge or expertise in Web technology and your programming skills.
This article is meant only as an introduction to Google’s API’s. If the need warrants it, I would be pleased to write a more detailed and in-depth article on them at a later date.
What Is A Google API?
The Google API stands for ‘Application Programmable Interface’. As it’s name implies, it is an interface that queries the Google database to help programmers in the development of their applications.
At this point, it is important to remember that all of Google’s APIs are only available in beta version, which means they are mostly still in their initial trial release and that there could still be a few adjustments required to some of them, although I must honestly say that I am quite pleased with what I saw so far.
The Google API’s consist basically of specialized Web programs and specialized scripts that enable Internet application developers to better find and process information on the Web. In essence, Google APIs can be used as an added resource in their applications.
How Can Google API’s Be Used Effectively?
In the real world, application programmers, developers and integrators write software programs that can connect remotely to the Google API’s. All data communications are executed via the ‘Simple Object Access Protocol’ (SOAP), which is a Web services standard as defined by the industry. The SOAP protocol is an XML-based technology meant to easily exchange information entered into a Web application.
Google’s API can better assist developers in easily accessing Google’s web search database, empowering them in developing software that can query billions of Web documents, constantly refreshed by Google’s automated crawlers. Programmers can initiate search queries to Google’s vast index of more than three billion pages and have results delivered to them as structured data that is simple to analyse and work with.
Additionally, Google API’s can seamlessly access data in the Google cache, while at the same time provide checking in the spelling of words. Google APIs will in fact implement the standardized search syntax used on many of Google’s search properties.
Use Of Google’s API’s In The Real World
Here I will give a few examples of real-life Web applications that could effectively use Google API’s. I will only give a few examples, but obviously, there can be many more.
‘ Querying the Web, using non-HTML interfaces, such as complex industrial software or command-line interfaces used in certain Unix applications
‘ Processing specialized market information, research and analysis of discrepancies in certain data types used in various industries
‘ Initiating regularly scheduled search requests to track the Internet for new and updated information on a specific subject
Currently, Google issues each programmer or developer who registers to use the APIs a set limit of one thousand queries per day. Some think that number could increase in the future, but for now, that is the imposed limit at the ‘Googleplex’.
Google’s API is an experimental program that is provided free to anybody that agrees to its terms. As of now, the available resources to fully implement and support the program are rather limited, which explains why there is a 1,000 queries a day limit on all accounts.
Registering For Your Google API
In order to be able to use and implement any Google API’s you must first agree to the terms and promise to abide by Google’s rules concerning it’s API’s. You will also have to create a Google ‘API account’. Once you have done all that, Google will email you a personal license key to use with your APIs.
Remember that when you build an application, you must integrate your API account number in your code. That way, every time your Web application makes a request or queries Google’s database, your license key is part of the query string.
More On The Google API Account
Creating an API account with Google is simple. All you need to do is to go to http://www.google.com/apis/ and simply follow the easy instructions on your screen. You will be required to provide your email address and a password. Currently, Google rules and regulations forbid you from having more than one account.
One word of caution about Google API’s: remember that you cannot create or develop any industrial application or commercial querying program or software using Google’s API technology, without first obtaining a valid and written consent from Google.
Reference: Google Inc., http://www.google.com/apis/
Author:
Serge Thibodeau of Rank For Sales
Search Engine Robots
Automated search engine robots, sometimes called “spiders” or “crawlers”, are the seekers of web pages. How do they work? What is it they really do? Why are they important?
You’d think with all the fuss about indexing web pages to add to search engine databases, that robots would be great and powerful beings. Wrong. Search engine robots have only basic functionality like that of early browsers in terms of what they can understand in a web page. Like early browsers, robots just can’t do certain things. Robots don’t understand frames, Flash movies, images or JavaScript. They can’t enter password protected areas and they can’t click all those buttons you have on your website. They can be stopped cold while indexing a dynamically generated URL and slowed to a stop with JavaScript navigation.
How Do Search Engine Robots Work?
Think of search engine robots as automated data retrieval programs, traveling the web to find information and links.
When you submit a web page to a search engine at the “Submit a URL” page, the new URL is added to the robot’s queue of websites to visit on its next foray out onto the web. Even if you don’t directly submit a page, many robots will find your site because of links from other sites that point back to yours. This is one of the reasons why it is important to build your link popularity and to get links from other topical sites back to yours.
When arriving at your website, the automated robots first check to see if you have a robots.txt file. This file is used to tell robots which areas of your site are off-limits to them. Typically these may be directories containing only binaries or other files the robot doesn’t need to concern itself with.
Robots collect links from each page they visit, and later follow those links through to other pages. In this way, they essentially follow the links from one page to another. The entire World Wide Web is made up of links, the original idea being that you could follow links from one place to another. This is how robots get around.
The “smarts” about indexing pages online comes from the search engine engineers, who devise the methods used to evaluate the information the search engine robots retrieve. When introduced into the search engine database, the information is available for searchers querying the search engine. When a search engine user enters their query into the search engine, there are a number of quick calculations done to make sure that the search engine presents just the right set of results to give their visitor the most relevant response to their query.
You can see which pages on your site the search engine robots have visited by looking at your server logs or the results from your log statistics program. Identifying the robots will show you when they visited your website, which pages they visited and how often they visit. Some robots are readily identifiable by their user agent names, like Google’s “Googlebot”; others are bit more obscure, like Inktomi’s “Slurp”. Still other robots may be listed in your logs that you cannot readily identify; some of them may even appear to be human-powered browsers.
Along with identifying individual robots and counting the number of their visits, the statistics can also show you aggressive bandwidth-grabbing robots or robots you may not want visiting your website. In the resources section of the end of this article, you will find sites that list names and IP addresses of search engine robots to help you identify them.
How Do They Read The Pages On Your Website?
When the search engine robot visits your page, it looks at the visible text on the page, the content of the various tags in your page’s source code (title tag, meta tags, etc.), and the hyperlinks on your page. From the words and the links that the robot finds, the search engine decides what your page is about. There are many factors used to figure out what “matters” and each search engine has its own algorithm in order to evaluate and process the information. Depending on how the robot is set up through the search engine, the information is indexed and then delivered to the search engine’s database.
The information delivered to the databases then becomes part of the search engine and directory ranking process. When the search engine visitor submits their query, the search engine digs through its database to give the final listing that is displayed on the results page.
The search engine databases update at varying times. Once you are in the search engine databases, the robots keep visiting you periodically, to pick up any changes to your pages, and to make sure they have the latest info. The number of times you are visited depends on how the search engine sets up its visits, which can vary per search engine.
Sometimes visiting robots are unable to access the website they are visiting. If your site is down, or you are experiencing huge amounts of traffic, the robot may not be able to access your site. When this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website. In most cases, robots that cannot access your pages will try again later, hoping that your site will be accessible then.
Many search engine robots are unidentifiable when you look through your web logs, they may be visiting you but the logs states it is someone using a Microsoft browser, etc. Some robots identify themselves by the actual search engine’s name (googlebot) or a variation of it (Scooter = AltaVista).
Depending on how the robot is set up through the search engine company, the information is indexed and then delivered to the databases.
The information delivered to the databases then becomes part of the search engine and directory ranking process. This is when the mathematical calculations are used for a final database result which is what you see when you search in the search engines.
The search engine databases update at varying times. Even directories who have secondary search results use the robot results as content for their website.
Once you are in the search engine databases, the robots keep visiting you. The amount of times you are visited depends on how the search engine sets up its visits, which can vary per search engine.
In fact, robots aren’t just used by the search engines. There are robots that check for new content for databases, re-visit old content for databases, check to see if links have changed, download entire websites for viewing, and more.
Sometimes visiting robots are unable to access the website they are visiting, when this happens, the website may not be re-indexed, depending on the frequency of the robot visits to your website.
For this reason, reading log files (either raw logs from your server or through your log statistics program) and tracking your search engine listings helps you keep track of the indexing of your website pages.
Resources
SpiderSpotting – Search Engine Watch
Robotstxt.org
Protocols for setting up a robots.txt file.
Spider-Food
Tutorials, forums and articles about Search Engine spiders and Search Engine Marketing.
SpiderHunter.com
Articles and resources about tracking Search Engine spiders.
Sims Search Engine Simulator
Search Engine World has a spider that simulates what the Search Engine robots read from your website.
Author Bio:
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing (http://www.searchinnovation.com), a Search Engine Promotion company serving small businesses. Besides running her own company, Daria is an associate of WebMama.com, an Internet web marketing strategies company. She has specialized in search engine optimization since 1998, including three years as the Search Engine Specialist for O’Reilly & Associates, a technical book publishing company.
Organic SEO: Patience For Long Term Ranking Results
When does long term SEO show ranking results? It takes time for optimization to produce targeted traffic to your website. Organic SEO requires time to take effect, just as it takes time for your web pages to start showing up in the search engine results.
Clients regularly ask me about the timing of a search engine optimization campaign and when those results will be seen in the search engine listings. A long term marketing campaign based on search engine optimization takes time: patience is the name of the game.
Optimization Timeframe
SEO’s timeframe depends on a number of factors. Part of this involves the accuracy of keyword phrase choices: is the keyword phrase one your visitors would use to find your product or website? If your keyword phrases are targeted to your audience, you will gain optimum results. Did you use Paid Inclusion and/or PPC services? The best combination for success involves using a combination of SEO, Paid Inclusion and PPC services. If you do not use Paid Inclusion or PPC, using organic SEO only, it takes more time to achieve results.
Paid Results
When you use Paid Inclusion or PPC (Pay-Per-Click) bidding, your results show up sooner than traditional SEO. Paid Inclusion submissions state the time-frame in which your page will be indexed by the search engine robots when you sign up for services. PPC bidding results show up as soon as searchers start clicking on your PPC ads. This type of search engine marketing requires an annual budget to renew Paid Inclusion submissions and payment per month for PPC click-through costs. If you are paying too much for your PPC services, organic SEO combined with PPC often helps to keep the prices down for the paid service. By generating additional targeted traffic on those costly terms you may be able to bring the bidding prices down in your PPC campaign or even eliminate some keyword bidding.
The timeline given for paid submissions means the search engines are generating income through this process. Paying for results also gives you a guarantee the listings will be relatively stable in the database.
Paid Inclusion submissions will always take precedence over free submissions because the company makes money from Paid Inclusion. For this reason most search engines will implement free search engine submissions over a longer period of time than paid submissions. When using SEO without the paid submission choices, the process is the same but the optimized pages take longer to be processed into the search engine databases.
Organic SEO
Organic SEO works differently. The best reason to use organic SEO is that it is a low-cost method to promote your website. It can take up to three to six months to see the full results of optimizing your website, especially if you are only using organic optimization. The plus to an organic approach is that once you optimize your pages, the main part of the work is done. You may tweak your keywords and text here and there, but unless you completely re-design your pages, you have what you need in place to begin drawing in targeted traffic. Continue checking your ranking status and reading your log statistics, especially for new keywords visitors are using to find your website.
When using free submissions, expect a three to six month wait before seeing most of the long term results showing in the search engine listings. If you build on a link popularity program and have links pointing back to your website, the search engine robots will find your website through the links, eliminating the need for free submissions. Look at it this way: you pay once for basic optimization and over time the results improve to optimum level. You don’t have to keep paying for this service because, unless search engine databases drops your free submission pages (which is not often these days), you will be visible and present to the search engine users when they search on your targeted keyword phrases. Over time you should see a progression in your ranking, depending on how competitive your keyword phrases are.
Budget SEO
What if you can’t afford Paid Inclusion or PPC services? Organic SEO is a great way to increase targeted traffic to your website over time. If you do not have a budget for Paid Inclusion submissions and PPC programs, organic SEO will give you good results if you are willing to wait instead of gaining immediate results. Combine organic SEO with plenty of good content and a solid link building program for optimum results. Remember, the search engine listings may entice visitors to come to your website, but you must give them a reason to stay once they arrive. Build your content to keep your new visitors at your website.
Patience Pays Off
Organic SEO is “common sense” promotion. Not a lot of fancy bells and whistles, and it takes time. The addition of good navigation, good content with your keyword phrases throughout the pages and topical sites pointing links back to your website equals long term success. Practice patience when going organic for your SEO campaign. It may take time but it will be worth the long term results you reap.
Author Bio:
Daria Goetsch is the founder and Search Engine Marketing Consultant for Search Innovation Marketing (http://www.searchinnovation.com), a Search Engine Promotion company serving small businesses. Besides running her own company, Daria is an associate of WebMama.com, an Internet web marketing strategies company. She has specialized in search engine optimization since 1998, including three years as the Search Engine Specialist for O’Reilly & Associates, a technical book publishing company.
Server Log Files
You have invested considerable amounts of money in search engine optimization for your website. You might even participate in a PPC (Pay-per-Click) campaign or other forms of advertising. Yet, you would like to know just who, when, how and where did they go while visiting your site. Maybe you are also asking yourself the eternal question: Did they buy from me? If not, why did they leave without buying anything?
After being in this business for more than 6 years, I still can’t understand why it is that so few site owners, business administrators and even many Webmasters don’t look at their server logs more often and, most importantly, more closely.
Few seem to realize that there is a ‘diamond mine’ of information residing in those log files. All it takes is to retrieve them and analyze them, at least once a week, at a minimum. If you can look at them everyday, it’s even better.
A Primer On Server Log Files
First, let’s rewind back to 1994-1995. In the mid-90’s, when the Internet was still relatively young, commercially that is, most site owners and Webmasters were mostly interested in the number of ‘hits’ their site got, from one day to the next. Even today, I still get asked: “How many hits did I get today?” By talking with some of these clients, I realize that they need a little primer article, such as this one, in helping them better understand the importance of professional analytics and measurements of Web traffic, site demographics, geo-location data, statistics, etc.
Today, modern traffic analysis tools and complex analytical software applications can yield far beyond this rudimentary, basic information. The more information a business administrator can have at his or her fingertips, the more he or she can use it to their best advantage over their competitors. Plus, there are many other benefits to be had as an added bonus. If some executives within your organization would like to know exactly how some visitors behave once they are inside your site, they can now get access to this kind of information, with a good analytical package.
Additionally, one of the most important features of most of the good packages I have seen is the ‘link referrer’ information. As its name implies, the link referrer information will clearly identify the search engine that brought you that visitor, or if it’s a link from one of your affiliate’s sites or a site that links back to you. What’s more, when properly configured, the search engine referrer log will even tell you which specific keywords or key phrases they entered to find your site.
The Importance Of Error Logs
Another often-overlooked and neglected area is the error log. It’s an excellent way to discover problems with your website before a visitor mentions it to you. A good example is when you find ‘error 404’ or ‘file not found’ in your error logs. When added with the site referrer information, you are also able to discover which page caused the error. This feature alone makes it a lot simpler to correct this problem.
With today’s most popular Web hosting packages offered on the market, many of them provide some kind of server log files that one can analyze and evaluate. You should ask your Web hosting provider for that information.
Conclusion
Frequent and regular server log file analysis and monitoring can be a very valuable source of important website information in deed, if done correctly. It can mean all the difference between discovering that a site has little or no ROI and one that truly achieves the best overall value for any size business. There are very simple and inexpensive server log analytical programs and software available today. Some are even free. As with anything, you can also purchase expensive software suites that are more suited for a larger enterprise. It all depends on your needs and budget.
Author:
Serge Thibodeau of Rank For Sales
Making Sure Your Site Is Crawler-friendly
I couldn’t find any “meaty” questions for this week’s newsletter, so I thought I’d just talk generally about what makes a site “crawler-friendly.” I used to call this “search-engine-friendly” but my friend Mike Grehan convinced me that the more accurate phrase was “crawler-friendly” because it’s the search engine crawlers (or spiders) that your site needs to buddy-up to, as opposed to the search engine itself.
So, how do you make sure your site is on good terms with the crawlers? Well, it always helps to first buy it a few drinks. But, since that’s not usually possible, your next-best bet is to design your site with the crawlers in mind. The search engine spiders are primitive beings, and although they are constantly being improved, for best results you should always choose simplicity over complexity.
What this means is that cutting-edge designs are generally not the best way to go. Interestingly enough, your site visitors may agree. Even though we SEO geeks have cable modems and DSL, our site visitors probably don’t. Slow-loading Flash sites, for example, may stop the search engine spiders right in their tracks. There’s nothing of interest on the average Flash site to a search engine spider anyway, so they’re certainly not going to wait for it to download!
Besides Flash, there are a number of “helpful” features being thrown into site designs these days that can sadly be the kiss of death to its overall spiderability. For instance, sites that require a session ID to track visitors may never receive any visitors to begin with — at least not from the search engines. If your site or shopping cart requires session IDs, check Google right now to see if your pages are indexed. (Do an allinurl:yourdomainhere.com in Google’s search box and see what shows up.) If you see that Google only has one or two pages indexed, your session IDs may be the culprit. There are workarounds for this, as I have seen many sites that use session IDs get indexed; however, the average programmer/designer may not even know this is a problem.
Another source of grief towards getting your pages thoroughly crawled is the use of the exact same Title tags on every page of your site. This sometimes happens because of Webmaster laziness, but often it’s done because a default Title tag is automatically pulled up through a content management system (CMS). If you have this problem it’s well worth taking the time to fix it.
Most CMS’s have workarounds where you can add a unique Title tag as opposed to pulling up the same one for each page. Usually the programmers simply never realized it was important, so it was never done. The cool thing is that with dynamically generated pages you can often set your templates to pull a particular sentence from each page and plug it into your Title field. A nice little “trick” is to make sure each page has a headline at the top of the page that is utilizing your most important keyword phrases. Once you’ve got that, you can set your CMS to pull it out and use it for your Titles also.
Another reason I’ve seen for pages not being crawled is because they are set to require a cookie when a visitor gets to the page. Well guess what, folks? Spiders don’t eat cookies! (Sure, they like beer, but they hate cookies!) No, you don’t have to remove your cookies to get crawled. Just don’t force-feed them to anyone and everyone. As long as they’re not required, your pages should be crawled just fine.
What about the use of JavaScript? We’ve often heard that JavaScript is unfriendly to the crawlers. This is partly true, and partly false. Nearly every site I look at these days uses some sort of JavaScript within the code. It’s certainly not bad in and of itself. As a rule of thumb, if you’re using JavaScript for mouseover effects and that sort of thing, just check to make sure that the HTML code for the links also uses the traditional <a> tag. As long as that’s there, you’ll most likely be fine. For extra insurance, you can place any JavaScript links into the tag, put text links at the bottom of your pages, and create a visible link to a sitemap page which contains links to all your other important pages. It’s definitely not overkill to do *all* of those things!
There are plenty more things you can worry about where your site’s crawlability is concerned, but those are the main ones I’ve been seeing lately. One day, I’m sure that any type of page under the sun will be crawler-friendly, but for now, we’ve still gotta give our little arachnid friends some help.
One tool I use to help me view any potential crawler problems is the Lynx browser tool. Generally, if your pages can be viewed and clicked through in a Lynx browser (which came before our graphical browsers of today), then a search engine spider should also be able to make its way around. That isn’t written in stone, but it’s at least one way of discovering potential problems that you may be having. It’s not foolproof, however. I just checked my forum in the Lynx browser and it shows a blank page, yet the forum gets spidered and indexed by the search engines without a problem.
This is a good time to remind you that when you think your site isn’t getting spidered completely, check out lots of things before jumping to any conclusions.
Jill
Author Bio:
Jill Whalen of High Rankings is an internationally recognized search engine marketing consultant and editor of the free weekly email newsletter, the High Rankings Advisor.
She specializes in search engine optimization, SEO consultations and seminars. Jill’s handbook, “The Nitty-gritty of Writing for the Search Engines” teaches business owners how and where to place relevant keyword phrases on their Web sites so that they make sense to users and gain high rankings in the major search engines.
Search Engine To Improve
Wouldn’t it be nice if the search engines could comprehend our impressions of search results and adjust their databases accordingly? Properly optimized web pages would show up well in contextual searches and be rewarded with favorable reviews and listings. Pages which were spam or which had content that did not properly match the query would get negative responses and be pushed down in the search results.
Well, this reality is much closer than you might think.
To date, most webmasters and search engine marketers have ignored or overlooked the importance of traffic as part of a search engine algorithm, and thus, not taken it into consideration as part of their search engine optimization strategy. However, that might soon change as search engines explore new methods to improve their search result offerings. Teoma and Alexa already employ traffic as a factor in the presentation of their search results. Teoma incorporated the technology used by Direct Hit, the first engine to use click through tracking and stickiness measurement as part of their ranking algorithm. More about Alexa below.
How Can Traffic Be A Factor?
Click popularity sorting algorithms track how many users click on a link and stickiness measurement calculates how long they stay at a website. Properly used and combined, this data can make it possible for users, via passive feedback, to help search engines organize and present relevant search results.
Click popularity is calculated by measuring the number of clicks each web site receives from a search engine’s results page. The theory is that the more often the search result is clicked, the more popular the web site must be. For many engines the click through calculation ends there. But for the search engines that have enabled toolbars, the possibilities are enormous.
Stickiness measurement is a really great idea in theory, the premise being that a user will click the first result, and either spend time reading a relevant web page, or will click on the back button, and look at the next result. The longer a user spends on each page, the more relevant it must be. This measurement does go a long way to fixing the problem with “spoofing” click popularity results. A great example of a search engine that uses this type of data in their algorithms is Alexa.
Alexa’s algorithm is different from the other search engines. Their click popularity algorithm collects traffic pattern data from their own site, partner sites, and also from their own toolbar. Alexa combines three distinct concepts: link popularity, click popularity and click depth. Its directory ranks related links based on popularity, so if your web site is popular, your site will be well placed in Alexa.
The Alexa toolbar doesn’t just allow searches, it also reports on people’s Internet navigation patterns. It records where people who use the Alexa toolbar go. For example, their technology is able to build a profile of which web sites are popular in the context of which search topic, and display the results sorted according to overall popularity on the Internet.
For example a user clicks a link to a “financial planner”, but the web site content is an “online casino”. They curse for a moment, sigh, and click back to get back to the search results, and look at the next result; the web site gets a low score. The next result is on topic, and they read 4 or 5 pages of content. This pattern is clearly identifiable and used by Alexa to help them sort results by popularity. The theory is that the more page views a web page has, the more useful a resource it must be. For example, follow this link today –
http://www.alexa.com/data/details/traffic_details?q=&url=http://www.metamend.com/
– look at the traffic details chart, and then click the “Go to site now” button. Repeat the procedure again tomorrow and you should see a spike in user traffic. This shows how Alexa ranks a web site for a single day.
What Can I Do To Score Higher With Click Popularity Algorithms?
Since the scores that generate search engine rankings are based on numerous factors, there’s no magic formula to improve your site’s placement. It’s a combination of things. Optimizing your content, structure and meta tags, and increasing keyword density won’t directly change how your site performs in click-tracking systems, but optimizing them will help your web site’s stickiness measurement by ensuring that the content is relevant to the search query. This relevance will help it move up the rankings and thus improve its click popularity score.
Search Engines Can Use The Click Through Strategy To Improve Results
Search engines need to keep an eye to new technologies and innovative techniques to improve the quality of their search results. Their business model is based on providing highly relevant results to a query quickly and efficiently. If they deliver inaccurate results too often, searchers will go elsewhere to find a more reliable information resource. The proper and carefully balanced application of usage data, such as that collected by Alexa, combined with a comprehensive ranking algorithm could be employed to improve the quality of search results for web searchers.
Such a ranking formula would certainly cause some waves within the search engine community and with good reason. It would turn existing search engine results on their head by demonstrating that search results need not be passive. Public feedback to previous search results could be factored into improving future search results.
Is any search engine employing such a ranking formula? The answer is yes. Exactseek recently announced it had implemented such a system, making it the first search engine to integrate direct customer feedback into its results. Exactseek still places an emphasis on content and quality of optimization, so a well optimized web site, which meets their guidelines, will perform well. What this customer feedback system will do is validate the entire process, automatically letting the search engine know how well received a search result is. Popular results will get extended views, whereas unpopular results will be pushed down in ranking.
Exactseek has recently entered into a variety of technology alliances, including the creation of an Exactseek Meta Tag awarded solely to web sites that meet their quality of optimization standards. Cumulatively, their alliances combine to dramatically improve their search results.
ExactSeek’s innovative approach to ranking search results could be the beginning of a trend among search engines to incorporate traffic data into their ranking algorithms. The searching public will likely have the last word, but webmasters and search engine marketers should take notice that the winds of change are once again blowing on the search engine playing field.
Author Bio:
Richard Zwicky is a founder and the CEO of Metamend Software, a Victoria, B.C. based firm whose cutting edge Search Engine Optimization software has been recognized around the world as a leader in its field. Employing a staff of 10, the firm’s business comes from around the world, with clients from every continent. Most recently the company was recognized for their geo-locational, or GIS technology, which correlates online businesses with their physical locations, as well as their cutting edge advances in contextual search algorithms.
Google Search Engine Related Question
Hello Jill,
First off, thank you for your informative and straightforward newsletter.
I see Google in a different light than you. [Last week you said] “As long as Google is still around and still focused on relevancy…”
Google is more concerned with inbound links than relevancy and they are ripe for misrepresentation if you have the money. If you search for “gas scooters” on Google you will see that the same company owns 9 of the top 10 spots. The same company pays for their text links at a very large, PageRank 6 site then interlinks all of the sites. If this sounds like sour grapes, it’s because it is. You can’t compete against money and brute force if you are small.
Thanks again,
Mike
Jill’s Response
When I first read Mike’s email, I basically thought of it as just another complaint from someone looking for a scapegoat because he couldn’t get high rankings himself. However, I received several similar emails from other subscribers over the past week. They were all concerned about their own Web sites’ ability to be ranked highly when there were companies dominating the search engine results pages (SERPs) through apparently deceptive means.
So, are the bad guys really winning at Google? Unfortunately, for many keyword phrases, it appears that they are.
I looked at the results for the gas scooter phrase Mike mentioned and yes indeed, I was aghast at the tricks being used on many of the top sites. I found every trick in the book, in fact. After doing a bit of digging, I even found some subtle clues that makes me think that I know which “SEO firm” is helping them.
One of the things I found were high-PageRanked sites, cross-linked with other high-PageRanked sites of the same genre, with numerous keyword phrases in the hyperlinks. Checking backlinks on these sites was also interesting because I found pages that were cloaking, pages that were stuffing image alt attribute tags and other things that I thought were pass’ in Google. Apparently they’re not at all pass’.
Surfing the sites with a Lynx browser was very revealing because JavaScript doesn’t work with Lynx, so I saw lots of stuff I wasn’t supposed to see.
The bigger question that I have to ask, however, is how does the person looking for gas scooters feel about these results at Google? Are they getting what they want? Are they happy with the results? Are they relevant to their query?
At first glance it appears that they are.
This may certainly be all that Google cares about. I don’t know — I’m not Google. Perhaps Google really does only care that the results are relevant and the means used to place them there are of no consequence. We know that their first priority is the user. As long as they’re happy, Google is happy. I guess they are happy…?
The thing is, I’ve always kind of thought that they did care. Maybe I’m wrong. Or maybe they’re just a little bit mixed up right now. Or maybe there’s just no way for Google to spot these techniques and we should all just use them. Okay, I wouldn’t actually go that far! Many of the techniques being used on these sites have been penalized in the past. I’ve seen that sort of cross-linking thing get sites PR0’d (penalized) many times. Perhaps Google’s PR0 penalty thingamabob is just broken?
Come to think of it, I remember that I wrote something similar way back in April of this year regarding the SERP for the phrase “email marketing consultant.” If it was just a bug, surely it would be fixed by now — over 6 months later. A quick check shows that nope, Google is still full of sites using deceptive optimization techniques.
So what’s up, Google? Tell me it isn’t so. Tell me that you still believe in good vs. evil and that the ends don’t justify the means. Cuz right now, you’re telling me that it does, and that makes me sad.
As to Mike and the others who have to compete in this space, all I can say is keep at it. You can keep making your sites better and better. You can continue to build up high-quality backlinks to your site. I took a quick look at Mike’s site, and Google is not showing any backlinks. He does have links, but they’re not yet considered high-enough quality to count for much. I didn’t notice any high-quality directory links pointing to Mike’s site, which might make a difference. Keep at it. Build up a great resource site all about your gas scooters. Invite others to write articles about their gas-scootering experiences, and whatever else you can think of.
We have no control over Google’s rankings. If they choose to let deceptive sites win, eventually the overall quality of Google will deteriorate. I’m sure they don’t want that to happen. They have a very tough job having to fight spammers every single day. The thing is, when they err on the side of penalizing too many sites, then everyone is up in arms because they got caught up in spam filters by mistake.
I’m confident that eventually Google will find a happy middle ground and someday be able to automatically tell the difference between sites that got there because they are truly the most relevant, and sites that deceptively appear to be the most relevant. Could be a long wait though.
If you believe that another site is abusing Google’s quality guidelines, feel free to report it to them here. Don’t expect any miracles, however.
Author Bio:
Jill Whalen of High Rankings is an internationally recognized search engine marketing consultant and editor of the free weekly email newsletter, the High Rankings Advisor.
She specializes in search engine optimization, SEO consultations and seminars. Jill’s handbook, “The Nitty-gritty of Writing for the Search Engines” teaches business owners how and where to place relevant keyword phrases on their Web sites so that they make sense to users and gain high rankings in the major search engines.
Robots Exclusion Protocol
Back in the spring of 2003, I wrote an article on the Robots exclusion protocol that generated a lot of emails and many questions. I am still getting them, so it seems that a more extended article is warranted. Often referred to as the ‘Robots.txt file’, the Robots exclusion protocol can be a very important part of your search engine optimization program, but it needs to be carefully implemented to be successful.
If used incorrectly, this small and ‘innocent looking’ text file can cause a lot of problems. It can even cause your site from being excluded from the search engine’s databases, it it’s not written correctly. In this extended article, I will show you how to correctly write it and make the Robots exclusion protocol an important part of your SEO efforts in attaining good visibility in the major search engines.
How The Robots Exclusion Protocol Works
Some of you may ask what is it and why do we need it? In a nutshell, as it’s name implies, the Robots exclusion protocol is used by Webmasters and site owners to prevent search engine crawlers (or spiders) from indexing certain parts of their Web sites. It could be for a number of reasons, such as sensitive corporate information, semi-confidential data, information that needs to stay private, or to prevent certain programs or scripts from being indexed, etc.
A search engine crawler or spider is a Web ‘robot’ and will normally follow the robots.txt file (Robots exclusion protocol) if it is present in the root directory of a Website. The robots.txt exclusion protocol was developed at the end of 1993 and still today remains the Internet’s standard for controlling how search engine spiders access a particular website.
If the robots.txt file can be used to prevent access to certain parts of a web site, if not correctly implemented, it can also prevent access to the whole site! On more than one occasion, I have found the robots exclusion protocol (Robots.txt file) to be the main culprit of why a site wasn’t listed in certain search engines. If it isn’t written correctly, it can cause all kinds of problems and, the worst part is, you will probably never find out about it just by looking at your actual HTML code.
When a client asks me to analyse a website that has been online for about a year and it still isn’t listed in certain engines, the first place I look is the robots.txt file. Once I have corrected it and written it for his website, and once I have optimized his most important keywords, usually the rankings will go up within the next thirty days or so.
How To Correctly Write The Robots.txt File
As the name implies, the ‘Disallow’ command in a robots.txt file instructs the search engine’s robots to “disallow reading”, but that certainly does not mean “disallow indexing”. In other words, a disallowed resource may be listed in a search engine’s index, even if the search engine follows the protocol. On the other hand, an allowed resource, such as many of the public (HTML) files of a website can be prevented from being indexed if the Robots.txt file isn’t carefully written for the search engines to understand.
The most obvious demonstration of this is the Google search engine. Google can add files to its index without reading them, merely by considering links to those files. In theory, Google can build an index of an entire Web site without ever visiting that site or ever retrieving its robots.txt file.
In so doing, it is not violating the robots.txt protocol, because it’s not reading any disallowed resources, it is simply reading other web sites’ links to those resources, which Google constantly uses for its page rank algorithm, among other things.
Contrary to popular belief, a website does not necessarily need to be ‘read’ by a robot in order to be indexed. To the question of how the robots.txt file can be used to prevent a search engine from listing a particular resource in its index, in practice, most search engines have placed their own interpretation on the robots.txt file which allows it to be used to prevent them from adding resources or disallowed files to their index.
Most modern search engines today interpret a resource being disallowed by the robots.txt file as meaning they should not add it to their index. Conversely, if it’s already in their index, placed there by previous crawling activity, they would normally remove it. This last point is important, and an example will illustrate that critical subject.
The inadequacies and limitations of the robots exclusion protocol are indicative of what sometimes could be a bigger problem. It is impossible to prevent any directly accessible resource on a site from being linked to by external sites, be they partner sites, affiliates, websites linked to competitors or, search engines.
Even with the robots.txt file, there is no legal or technical reason why they should be used, least of all by humans creating links, for which the standards were not written. In itself, this may not seem a bad idea, but there are many instances when a site owner would rather exclude a particular page from the Web. If such is the case, the robots.txt file will, to a certain degree help the site owner achieve his or her goals.
What Is Recommended
Since most websites normally change often and new content is constantly created or updated, it is strongly recommended that the Robots.txt file in your website be re-evaluated at least once a month. If necessary, it only takes a minute or two to edit this small file in order to make the changes required. Never assume that ‘it must be OK, so I don’t need to bother with it’. Take a few minutes and look at the way it’s written. Ask yourself these questions:
1. Did I add some sensitive files recently?
2. Are there new sections I don’t want indexed?
3. Is there a section I want indexed but isn’t?
As a rule of thumb, even before adding a file or a group of files that contain sensitive information that you don’t want to be indexed by the search engines, you should edit your Robots.txt file before uploading those files to your server. Make sure you place them in a separate directory. You could name it: private_files or private_content and add each of those directories to your Robots exclusion file to prevent the spiders from indexing any of those private directories.
Also, if you find that you have files in a separate directory but you want them indexed, if those public files have been on your server for more than a month and are still not indexed, have a look at your Robots.txt file to make certain there are no errors in any of it’s commands.
Examples Of A Properly Written Robots.txt File
In the following example, I will show you how to properly write or edit a Robots.txt file. First, never use a word processor to write or edit these files. Today’s modern word processors use special formatting and characters that will not be understood by any of the search robots and could lead to problems, or worse, it could cause them to ignore the Robots file completely.
Use a simple ‘pure vanilla text’ editor of the ASCII type or any text editor of the Unix variety. Personally, I always use the Notepad editor that comes on any Windows operating system. Make certain you save it as ‘robots.txt’ (all in lower case). Remember that most Web servers today run Unix and Linux and are all case sensitive.
Here is a carefully written Robots.txt file:
User-agent: Titan
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: ExtractorPro
Disallow: /
The user-agent is the name of the robot you want to disallow. In this example, I have chosen to disallow Titan, EmailCollector, EmailSiphon, EmailWolf and ExtractorPro. Note that many of these robots are from spam organizations or companies that are attempting to collect email addresses from websites that will probably be used in spam. Those unwanted robots take up unnecessary Internet bandwidth and slows down your Web server in the process. (Now you know where and how they usually get your email address from). It is my experience that most of those email collectors usually obey the Robots.txt protocol.
Conclusion
Properly implementing the Robots exclusion protocol is both a simple process and takes very little time to enforce. When used as it is intended, it can ensure that the files you want indexed in your website will be indexed. It will also tell the search robots where they are not welcomed, so you can really concentrate in managing your online business in the safest way possible, away from ‘inquisitive minds’.
Author:
Serge Thibodeau of Rank For Sales
Improve Link Popularity
There are many different ways to improve link popularity for your Website. One of the most effective ways to improve your site’s link popularity is by writing articles for publication on other related Web sites and in other related newsletters or Ezines.
In this article, I will be sharing 10 techniques I use when I am writing articles to improve link popularity for one of my Websites.
Let’s use this article as an example. I have two goals in mind as I start writing this article, a primary goal and a secondary goal.
My primary goal is to inform you, my reader, about how to improve link popularity for your Website.
My secondary goal is to improve link popularity for one of my own Websites.
If, after having read this article, you can say to yourself that this article was informative and it gave you one or more ideas about how to improve link popularity for your own Website, then I will have achieved my primary goal.
If, after having written this article, I can say to myself that this article was written well enough, and was informative enough, to persuade Webmasters and newsletter or Ezine publishers to reprint this article on their Website or in their newsletter or Ezine, then I will have achieved my secondary goal.
If I do achieve my primary goal, my secondary goal will be achieved as well – you will have been informed and the other publishers will realize that and will reprint this article in order to inform their readers as well.
Now, on to achieving my primary goal, informing you…
How To Improve Link Popularity By Writing Articles
1. Before writing your article, have definite goals in mind. Ask yourself what you want to accomplish by writing this article.
Since you are now reading an article about how to improve link popularity for your Website, that should be one of your goals. I’ll leave you to decide what other goals you want to achieve by writing your article.
2. Have well thought out ideas for the beginning, middle and ending of your article.
For this article, the purpose of my beginning was to inform you about an effective way to improve link popularity for your Website.
The purpose of the middle of this article is to inform you about how you can use this effective way to improve link popularity for your own Website.
The purpose of the ending of this article will be to provide you with a way to learn more on this topic and, to achieve my secondary goal, to improve link popularity for my own Website.
3. Include your most important keyword in the title of your article. You will notice that I have used the keyword phrase “improve link popularity” in the title of this article. This will improve the odds that your article will become indexed and listed by search engines.
4. Use your important keyword throughout the content of your article. Again, you will notice that I have used the keyword phrase “improve link popularity” several times throughout the content of this article. Again, this is for the purpose of getting this article indexed and listed by search engines.
5. Write your article as if you were speaking directly to your best friend about a topic that both you and she are passionate about. Don’t try to impress her with your knowledge of the topic. Simply provide her with your thoughts on the subject, using easy-to-understand words and phrases.
6. After you have written your article, leave it alone for a day or two before coming back to it. Using a fresh pair of eyes, read your article to see if the content flows in a logical, easy-to-understand manner. Then proofread your article for typos. Make any necessary corrections.
7. After you have written and proofread your article, publish it on your own Website. Include a resource box at the bottom of your Web page that informs Webmasters and newsletter or Ezine publishers that they are free to reprint your article. This is, after all, the focus of this article on how to improve link popularity for your Website.
8. If you publish a newsletter or Ezine, also publish your article in it as well. If it is a lengthy article, publish part of the article in your newsletter or Ezine and provide a link to the complete article that is published on your Website.
9. Include a resource box at the end of your article that provides a little information about you and an invitation for your readers to visit your Website. Some readers will go to your Website, read your other articles, and choose to reprint some of those as well. This is a definite bonus when you are writing an article to improve link popularity for your Website.
10. Submit your article to Websites that archive articles that are available for reprinting on other Websites and in other newsletters or Ezines. My favorite Website for archiving my articles and for obtaining articles written about how to improve link popularity is ArticleCity.com, located at http://www.articlecity.com/. Other archives can be located by searching on “free articles for reprint” or something similar.
Summary
It took me about two hours to write this article. When you are passionate about the topic you are writing about, the words just seem to flow from your brain to your article.
If you are passionate about the topic of your article, you will be able to write an informative article which will be reprinted by Webmasters and newsletter or Ezine publishers and that will help to improve link popularity for your Website.
Author Bio:
Bob designs and builds his own Websites. His most recent site is Link Popularity IQ: http://www.lpiq.com/. Bob invites you to test your knowledge of link popularity by taking his free Link Popularity IQ Test: http://www.lpiq.com/test.
Gasp! It Costs HOW Much??
If you’ve ever gone shopping for professional yet affordable search engine optimization, the sticker shock may have caused you to reconsider that ‘$99 search engine submission special‘ you saw advertised. After all, it’s a field of expertise with no regulatory body and no simple answer to the question: Just how much should an online business pay a search engine optimization company?
I know how much we charge at eTrafficJams.com and our clients say it’s worth every cent. But there are plenty of other SEO services out there that may quote you rates much higher or much lower.
So it’s up to you to examine your priorities. Is a bargain basement package more attractive than increasing your rankings, sales, and revenues via up-to-date, effective SEO techniques that may cost a little more?
Are momentarily high rankings more important than a guarantee of ‘clean’ search engine optimization that will keep you competitive over the long haul?
Are you willing to sacrifice effective SEO strategies in exchange for a discount solution?
Search engine optimization is an incredibly time-consuming task that requires specialized skills and constant research to stay on top of the changing SEO landscape. If the fee is tantalizingly low, well, like they say, ‘if it walks like a duck…’
On the other hand, if you opt for quality SEO, here’s what your money should get you. This will explain why professional optimization is not cheap:
1. Keyword phrase identification. Finding that handful of perfect terms capable of turning a sale is the first order of business and it gobbles up vast amounts of an SEO expert’s time. And here’s a surprise: coming up with a list of 100 keywords that apply in general is far easier than developing a short list of the most relevant, hardest-working ones.
2. Web design consultation. If your site layout isn’t spider-friendly, no amount of SEO finesse will solve your traffic problems. So a search engine expert also has to be up on web design issues and know how to fix the trouble spots.
3. SEO copywriting. It isn’t enough to drive herds of traffic to your site. The masses need to be informed and sold once they get there. That’s where SEO copywriting comes in. (Bargain optimizers always scrimp on this one.)
4. HTML optimization. Keyword phrases also play a starring role in your HTML tags. Knowing which tags are hot and which are not (in the eyes of the engines) is the purview of a truly clued-in search expert.
5. Site registration. This does not mean submitting to Google and the Open Directory Project, then going for lunch. SEO pros know which engines and directories are the movers and shakers, which minor ones are becoming forces to be reckoned with, and where your site stands in the rankings of each. A full-time job in itself.
6. Link partnership building. Have you tried it yet? It’s not easy locating quality sites related to but not competing with yours who are willing to give up valuable real estate for your link. It takes solid research, smooth negotiating skills, and a large investment of time to develop a base of reputable, relevant link partners that will make the search bots take notice.
7. Monitor traffic. Notice I said ‘monitor traffic’ not just ‘monitor rankings’. Since conversions are the bottom line for most e-businesses, a good search engine optimization company will include comprehensive ROI tracking so you can see exactly what your SEO dollar is producing for you.
If you sacrifice just one of these strategies, you could be jeopardizing the effectiveness of all the others. And many SEO services sacrifice more than just one.
Show Me the Money
Okay, let’s talk numbers. My research shows that you can expect to pay between $3,000 and $7,000 for search engine optimization that includes the above vital steps, depending on how many pages are optimized.
So now you know the difference between professional, affordable search engine optimization and the ‘$99 to submit your site to 3000 search engines’ type of SEO promotion. The price tag is vastly different. So is the product.
Author Bio:
Michael Pedone is founder of eTrafficJams.com, a professional search engine optimization company that specializes in getting targeted, eager-to-buy traffic to your site! Want some tips on how to improve your web site’s optimization? Call 877-785-9977 or click to request a free search engine optimization analysis.
