Thursday, July 28, 2005

SEF URLs– The Big Misconception

“Search Engines do not like ? and such in the address. “

“People have been discussing how search engines do not like ? marks etc. and how they won't index a page with those things”

“..google turns crazy if it spiders the same document with all time different content”

Over the past 3 years of working with osCommerce, I’ve heard misconceived comments like these over, and over again. It seems that new storeowners often confuse different issues, and their questions and comments, such as the ones above, spread like wildfire to other storeowners terrified that they might not be indexed properly by search engines.

For a litlte history, the term SEF (Search Engine Friendly) URLs cam about pre 1997. Google and Yahoo were just beginning to get going, and they created automated programs called ‘spiders’ to transverse the web, looking for webpages to add to their index. Nearly all webpages at the time were static HTML, dynamic content webpages were just starting to pop up around the web. The traditional URL for a static HTML website had always looked like this.

http://www.yoursite.com/catagory/your_product.html

Search engines indexed page after page of these static html pages with no problem. However, static content webpages were laborious to maintain, and webmasters soon developed a way to have one page display different content depending on certain logical conditions. Early dynamic content webpages were mostly all in perl asd asp, and their URLs looked more like this.

http://www.yoursite.com/category/your_product.pl?product_model=101

In the early days, this caught ‘spiders’ off guard a bit, as they were not written to be able to ‘handle’ the special characters ‘?’ and ‘=’ in the URLs. They actually ended up either truncating everything after the first special character, or ignoring the URL entirely. This meant the most of the dynamic URLs would not be included in the search engines index.

Developers came to realize the issue before search engines could address it, and devised a means to ‘re-write’ the URL to make it appear as though it were a static URL, without the special characters. This was a very good solution for a couple of years, and dynamic content webmasters were then able to have all of their product pages indexed by search engines.

Sometime in 2001 however, Google resolved the issue with indexing URLs wiht special characters, and dynamic content websites with URLs still containing special characters began to pop up all over the place. All of the other major search engines followed suite. Today, in a search for specific products, nearly half of the URLs returned in a query from Google contain special characters. And there doesn’t seem to be and preference given to a URL without special characters in it over a URL that has special characters in it.

http://www.google.com/search?hl=en&lr=&q=buy+world+of+warcraft

As you can see, Google themselves use URLs with special characters in them. Some might argue that Google is not the only search engine out there, and that there may still be search engines out there that do not index URLs with special characters in them. However, Google, MSN, and Yahoo all do. And 99.99% of all of your traffic will come from these three engines. In fact, I even searched a few dozen of the smaller search engines, and I have yet to find a single search engine that still does not support URLs with special characters. If you can find one, I’d like to see it. Personally, I don’t think there are any left, or at least none that you'd actually get traffic from.

Don’t get me wrong, there are other considerations to take into account when talking about ‘optimizing’ your URLs to improve your search engine listings. I’ll discuss some of these ideas in my next article, including SIDs, key phrase embedding, key phrase proximity, and URL length.

But there is no reason to fear that all of your product pages will not be indexed because your have special characters in your URL. It simply isn’t true anymore.

Thursday, July 21, 2005

A tactical view of Analytical Reporting in osC

The long-term success of any business is dependent on the decisions it makes. And as a small business grows, so does the need to analyze the data they have collected to make these important business decisions. Forecasting, trend analysis and other aggregations of raw data are imperative to understanding your business to assist you in making these decisions. Yearly, quarterly, monthly, weekly, and daily aggregation of sales data, for example, give you great insight into how well the volume of your business is doing. And the more ways you look at your data, the better you can understand your business.

Unfortunately, these types of data aggregations can be incredibly taxing on server resources that your application relies on. So much so in fact, that eventually, your application will grind to a halt. Performance and usability of your application will become so poor that you’ll end up losing customers.

So, what does a medium to large storeowner who wishes to leverage all of that captured data without compromising performance do?

Ideally, data would be extracted, transformed (if necessary) and transferred over to an Operational Data Store (ODS) or Enterprise Data Warehouse (EDW) in a daily, incremental fashion, and then purged from the OLTP (On Line Analytical Processing) system after 30, 60 or 90 days. That would keep your application running on a ‘lean’ set of data for optimal performance. Data that would be occasionally needed by the application, such as order history, would be transferred over to the ODS, while data who’s only purpose would be to aggregate and analyze, would be moved over to the EDW.

However, this introduces a new level of complexity into the data model. Two things specifically jump out at me that would require consideration in the OLTP database in order to insure data integrity between the reporting structures, and the OLTP database. In order to support the incremental extraction, I think you’d need to have a couple of very dependable timestamps. You’d need to have one column for the timestamp of the row was inserted, and one for the timestamp that the row was last updated. In conjunction, you’d also need a solid set of triggers, that you’d use to ensure that any time a row in the OLTP database was updated, to make sure that new, updated row was included in the extractions.

In addition to the timestamps, I think you’d need to have a reliable deletion audit trail. Obviously, if data is deleted in the OLTP database, you need a way to also delete it from the ODS and/or EDW. Typically, this is handled with a delete trigger, and a ‘delete_log’ table. After the ETL (Electronic Load and Transfer) process extracts and loads data from the OLTP system into the reporting structures, it will also grab all of the deleted rows from the ‘delete_log’ table, iterate through them, and update the tables in the ODS/EDW accordingly, either deleting them, or marking them as deleted.

These are all fairly common industry standard methods of keeping the OLTP database from becoming bogged down with too much data, and to keep contention down by moving reporting away from the application.

Unfortunately, there seem to be some inherent problems with MySQL that hinder us from being able to accomplish this easily. First of all, MySQL doesn’t appear to support triggers. This requires that the application itself be responsible for ensuring that all deletions are recorded in the delete log table, and that all updates also include an update to the last updated timestamp. I’m don’t know about you, but I’m not a big proponent of trusting the application to do anything when it come to data integrity.

With all of this being said, while I’d certainly be up to the challenge of creating an ODS/EDW for an osC OLTP system (through, I’m of the opinion that once a small business grows beyond the point of being ‘small’, it’s time to start looking for a new shopping cart application.

Wednesday, July 20, 2005

Long Checkouts Suck

As I’m sure using the osCommerce cart knows, the checkout process of this application is an absolute dinosaur. The checkout process, as currently implemented in the latest csv is 8 pages long. The process for a new customer to your store goes something like this.

1.) The customer click the ‘Add ot Cart’ button from the product information page, and is taken to the shopping cart page. 1 page.
2.) They click ‘checkout’ on the shopping cart page, and is directed to the ‘login page’. 2 pages
3.) They click ‘Create an Account’ and are directed to the account detail page. 3 pages
4.) They provide their personal details, and click ‘continue’, and are directed to the ‘create account success’ page. 4 pages
5.) They click ‘continue’ are taken to the checkout shipping page. 5 pages
6.) They select their shipping click ‘continue’ and are taken to the checkout payment page. 6 pages
7.) They enter their payment details, click ‘continue’, and are taken to the checkout confirmation page. 7 pages
8.) They click ‘confirm’ and are taken to the checkout successful page. 8 pages

This checkout process becomes even LONGER is the merchant is using a 3rd party payment processor. Merchants using PayPal force their customers through an 11 page checkout!

A ‘Purchase Without Account (PWA)’ contribution was written a couple of years ago, and is still regularly updated, however this contribution doesn’t he4lp much in this capacity, as it only eliminates ONE page from this checkout process.

Those of us that have been using osC for a long time, have learned ways make small changes to the cart to minimize the checkout process without overhauling the entire process. Normally, the first things I do are…
- Install the ‘Combine Login and Create Account Page’ contribution.
- Change the checkout button on the shopping cart page to redirect the customer to the combined page mentioned above, instead of the login page. There is no reason to have a whole page of the checkout process dedicated to basically asking the customer if they have been here before.
- Change the account details page so that the header reads, “Your Shipping Details”, and remove as many of the fields as possible, and make as few fields required as possible.
- Remove the password, and newsletter field from the account details page, and instead send them a programmatically generated password in their invoice, so they can log back in to your site later for package tracking and account history.
- Change the account details page ‘continue’ button to redirect to the checkout shipping page rather than the checkout success page.

Taking these steps, you can remove 2 pages of the checkout, and ‘mask’ the fact that the customer is creating an account, without having to make wholesale changes to the checkout process. But this still leaves a 6 page checkout, and as many of you know, the more pages your checkout has, the more customers you’re likely to lose along the way. What I’d like to see is a ay to get the checkout down to a 3, or at most 4 page checkout.

Thinking long and hard about various ways to logically reduce the number of pages involved in the checkout process without unnecessarily cluttering the pages, or making one extremely long checkout page (see the one page checkout contribution), here’s what I’ve come up with.

- Combine the Shopping Cart page with the ‘account details page’, including a box to log in. The customer can either continue shopping from this page, log in, or provide their Name and address.
- The next page would be a combined ‘checkout shipping’ page, and ‘checkout payment’ page, where they would select their shipping method, and enter their payment details.
- The next page would be the checkout confirmation page.
- The last page would be the checkout success page.

This would get the checkout down to 4 clicks, which seems to be about industry standard these days. I’d like to know what your thoughts are on this checkout process, if you have any other ideas of your own on how to make the checkout process better.

Monday, July 18, 2005

The osC infobox class, what a pain.

Does anyone else wonder why we need a table with a cellpadding of 2, nested within another table just to make a border? I’ve been doing a lot of work recently, most notably over at www.package.ie/index2.php, and I’ve come to the conclusion that the various infobox and content boxes in osC are completely useless. Classes are supposed to make code more modular, more re-useable. But I have yet to find a project that it’s actually beneficial to reuse these classes. And the standard infobox of osC is nearly impossible to customize, other than the header and the footer (which can actually be done outside of the classes, as a wrapper around them) when using these classes. Simply put, formatting of the infoboxes, or virtually any formatting shouldn’t be done with nested tables. Using css entries with fixed position coordinates can accomplish this much more easily.

In the next few days, I’m going to take a crack at simplifying the infobox and contentbox content classes to use css entries rather than nested tables. I think that would make customizing the infoboxes a heck of a lot easier. If anyone is interested, feel free to submit your suggestions, or sample code here.

Friday, July 15, 2005

A Closer Look at a Payment Gateway

Continuing our previous discussion regarding PayPal, we know that starting out with a new business, and a brand new website, PayPal is a great way to get the ‘doors open’ and begin accepting payments with very little time or financial investment. We know that there are a lot of people that like to use their PayPal accounts, and will also go out of their way to make online purchases only from shops that offer PayPal as a payment method.

But we also know that there are a lot of people that would prefer not to go through an 11 page checkout, or be redirected to a third part website to make payments. Knowing this, we conclude that it’s probably best if we offer customers a ‘choice’ of how to make payment.

So, the questions become, “When is the right time to add a merchant account/payment gateway to your store?, and “What merchant account and payment gateway is the best?”.

When to add a payment gateway depends on your business. Obviously, having a monthly fee increases your overhead, and your business needs to be robust enough to incur that. But a lot of people are under the misconception that a payment gateway has some sort of huge ‘set up fee’ or ‘application fee’. We wanted to find out what the best rates around were, and we found that vendors of the popular ‘Authorize.net’ payment gateway had rates that were at, or very near the best rates in the industry. You can find a pretty good list of auth.net vendors here. going through this list, we found several vendors that had no set up, activation, or application fees. And nearly all of them have rates similar to the ones we found at Mbankcard. Here’s what their fee structure looks like.

Gateway Fee: $20.00. This is a recurring monthly access fee for the secure server, that allows the secure transaction to take place. Merchant Accounts Fees: 2.25%, 29 cents per transaction and a $10.00 monthly Statement/Customer service fee.

So, that’s $30 per month, and 2.25% plus $0.29 per transaction. Compare that to PayPal’s at 2.29% (2.25% if you process more than $3000 per month) plus $0.30 per transaction with no monthly fee and you can see that the merchant account is going to cost you about $30 (minus $0.04 per transaction) more per month than PayPal.

Keep in mind that this $30 per month is going to save you some customers that would have left otherwise once they found out that they you only provided PayPal as your payment method. For example, when we started our first website, we only accepted PayPal, because like many other new web shop owners, we balked at the thought of that extra $30 per month. After several months of marketing, and constant advertising, we were able to get our new web store up to about 5 sales per day. Unfortunately, we began receiving a lot of emails from customers complaining about the checkout process, specifically about being re-directed, or about having to use PayPal to purchase. You can find a thread I made in the osC forums discussing this matter, and including excerpts from many of those emails here.

As a result, we decided to add a payment gateway to the store, to see if that would allieviate the issue. As soon as we did this, and without changing any of our marketing or advertising, our sales DOUBLED overnight. The increase was staggering, and much, much more than we would have anticipated.

To us, this was proof positive that you should always offer the customer a choice when it comes to method of payment.

Wednesday, July 13, 2005

When Should You Use PayPal?

I see the question, over and over again. "What payment method should I use." Or, "Is PayPal the best payment method for me?". The answer sounds simple. Always. You should always use PayPal. But that answer isn’t as simple as it might sounds, because you should NOT always use PayPal by itself.

PayPal is by far, the easiest payment method currently available. You can set up a PayPal account in less than a minute. It takes a little longer to become certified, but if need be, you can accept a credit card or PayPal payment form someone with very little trouble and no paperwork. The PayPal module in osC is easy to configure and enable, and is proven to be stable (continue button neglect issues non-withstanding, but we’ll get into the IPN and Auto Return features and why they are important a little later). A PayPal account costs nothing to create, there are no monthly fees, and the per-transaction fees PayPal charges are the lowest in the industry. PayPal is the absolute best payment method available for a new ecommerce storefront owner, because you can get your store live and accepting payments, without much of an investment in time or money.

Sounds great, huh? Well, after you’ve had your store open for a week or two, consider this.

If you’ve never seen or used the traditional PayPal payment method, when the customer arrives at the Order Confirmation page of osC, summarizing their order details, and they click on the ‘confirm’ button, they are re-directed to the PayPal website. Once their, they will either log in to PayPal, create a PayPal account, or if the store owner has enables the option for customers to pay without having to create a PayPal account, they can elect that option. From there they go to the payment page of PayPal, and from there the payment confirmation page.

Ok, lets stop here for a moment to digest some of that. So far, we’ve been re-directed to a third party website to make a payment, and we’ve encountered 3 new pages to transverse at PayPal. That’s on top of the already ridiculously lengthy, 8 pages of the payment process that osC uses for a total of an 11 page checkout plus a re-direct to a third party website. Does this sound like an efficient checkout process to you? Anyone who knows anything about ecommerce will tell you that the longer your checkout is, the more customers you will lose. A checkout that is 11 pages long has got to be the longest checkout in the history of ecommerce.

Now, advocates of PayPal are going to stop you at this point, and let you know that there are over 72 million people out there that use PayPal. And that and those 72 million users will demand that they be put through the torture of an 11-page checkout for the privilege of using their PayPal account to make their purchases. But I’m here to re-assure you that not all of those 72 million users are still actively using their PayPal accounts today, as evidenced by all of the anti-PayPal websites in existence today. And even though someone uses PayPal to buy things on eBay, doesn’t always mean that they will buy a $2000 stereo from a store that offers PayPal as the only payment method.

I can tell you that there is a significant percentage of potential customers that are very loyal to PayPal. And these loyal PayPal account holders will often go out of their way (like 11 pages out of their way) to make a purchase using their PayPal account. This percentage of customers will often shop only at ecommerce stores that offer PayPal as a payment method. It is for this percentage of the online purchasing population that I recommend that you always offer PayPal.
But this percentage in no way represents even close to the majority of potential customers that will visit your store. And for the majority of customers that do not already have PayPal accounts, and that are not loyal PayPal customers, you need to offer them an alternative. 11 pages of checkout is entirely too much to ask a customer to endure. Heck, the 8 pages of osC checkout are too much to ask, but that’s a blog entry for another day.

If you are serious about your business, and you want to maximize your ecommerce storefront’s potential, the best configuration is to offer PayPal, in addition to another payment method in which the customer can enter their payment details right there, on your secure website. This offers the customers interested in using PayPal what they want, and people who prefer not standing on long DMV lines, being stuck in rush hour traffic, and 11 page checkouts what they want. The convenience to get what they want, and get the hell out of there in the shortest amount of time humanly possible.

Read all of my posts in the osC forums regarding paypal and payment gateways Here.

Tuesday, July 12, 2005

WizardsandWars.com Sold.

Well, I'm sad to see it go.

My first ever osCommerce website, WizardsandWars.com has officially been sold. I won't post the amount it was sold for, but lets just say that while it was fair compensation for a business with over 1k unique visitors per day, and several thousand dollars profit per year, it doesn't come nearly close to compensation for the time I put into it.

Why did I sell it?

Well the bottom line is that I discovered that I'm about 70% programmer, and only 30% entrepenuer. Seriously, I wanted to spend less time with the day to day operational resposibilities of owning a small business, and more time with my hands in code. As some people probably know, I'm a bit of a code junky, and I'm happiest when I'm developing.

So, while I'm still a major partner in another gaming website, www.madmacgames.com, I've re-arranged my priorities, taken on some new projects, and spend nearly all of my free time now coding.

As for my alias, wizardsandwars, I don't think I'll change it. With over 4000 posts on the osC forums, I think I'm too well known under that handle to start another.

Anyways, goodbye WizardsandWars.com. Thanks for the headaches, and the fond memories.