The Marketing Optimization Blog

This is the transcript of our recent interview with Tom Leung. Click here to listen to the audio version.

Chris Goward: Great! Looks like we’re online here. Hi, this is Chris Goward from WiderFunnel Marketing at WiderFunnel.com. I’m speaking today with our friend Tom Leung, product manager for Google Website Optimizer.

Tom, of course, it’s always good to talk with you. Thanks for sharing your time with us today. I know you’re always popular with our readers which was apparent from that sellout event we had a few weeks ago. Thanks again for your time.

Tom Leung: Well, I appreciate you having me Chris. It’s always fun to talk with you.

CG:I think the last conversation we had (or that we recorded with you, anyway) was back when Optimizer was released from private beta, so I thought this new release – some of this news coming out about Google Website Optimizer – would be a good time for an update from you. Website Optimizer has now been taken out of beta, plus it’s had some usability improvements. What’s exciting about this release?

TL: You know I think the most interesting thing about this release is not one single element that has changed but more the combination of everything that we’ve done and how they all hang together.

As you mentioned, we officially came out of beta; we also launched a standalone version of the product where you can access it directly; you no
longer have to have an AdWords account. Then we also updated the UI – cleaned it up, made it a little bit more performant – and then we also updated the multivariate set-up process to provide a little bit more assistance for the tagging step. We also expanded the discussion group, we
launched a blog, and we’ve published about 12 new testing recipes and a bunch of new video tutorials.

So, I think it’s really the combination of all of those things that’s most exciting. I would kind of describe it as evolving from an interesting beta feature inside of AdWords to a full-featured prime-time product that’s at the tip of the sphere in a really exciting new category for Google and for all the users out there.

CG: It seems like you guys have answered a lot of people’s questions about how to find some of the common questions and
separated those into their own group rather than being within AdWords, which is great.

Within Google, looking from the inside out, what’s the significance of Optimizer moving outside of the AdWords group?

TL: Well, I think any time there’s a new standalone product, there’s a certain bar that you have to cross. It’s arguably easier to put a new feature, or a beta feature, in an existing product or to put something in Google Labs where we’re still kind of playing around with it, but when you officially launch as a brand new Google product sans beta label, no disclaimers, its own help center, its own login, its own sign-up process, it’s sort of like going from the minor leagues to the major leagues.

I think it’s representative of the company’s view that the product and the adoption we had seen already had justified taking this step and really kind of doubling down on the space of content testing and website optimization. We’re very excited by it. The team is very proud of what we’ve done so far and for sure we don’t think we’re done. We’ve got a lot of other interesting things that we’re always thinking about, but this is definitely a big milestone for us.

CG: So, I know you guys are always developing the product. Is there anything you can…I know you guys often have to be really tight-lipped about future product modifications. Is there anything you can tell us that is coming down the road?

TL: I can’t be too specific and I’m not trying to be coy, but one of the product development strategies that we have is we do a lot of prototyping and we do a lot of internal experiments, if you will, on new features. Many of them don’t make it through that Darwinian process, so I wouldn’t want to set expectations. I would just say in general that we’re always listening to customer input and listening to partner input, as well as trying to be creative with how we can take this tool to the next level. Our expectation is that we will be rolling out improvements to the product on a pretty regular basis. You’ll be the first to know when we have something that we’re able to tell you more about specifically.

CG: Well, I know you guys are putting a lot of emphasis on the product and developing features quickly. You’ve always been really responsive to adding new features even from feedback that we’ve given you, which is great, so it’s exciting for us. We can’t wait for the next updates.

You’ve been at this with Optimizer since the beginning and it’s been a year now. We were there in the private beta testing back before it was launched and the product’s evolved quite a bit. What’s been your biggest surprise or some of your biggest surprises during the first year of Optimizer? Did you think we’d be in a different place now than we are?

TL: I think our hope had always been that this would sort of graduate beyond being a feature in AdWords to its own product, but we knew that we had to prove the value of this effort to make sure that the resources that would be required to make it it’s own product and to invest more in it were justified. So, we are kind of where we had hoped we would be in terms of our expectations from the outset.

I would say the most surprising thing in the last I guess year and a half that I’ve been working on this…. Hmmm…I would say the most surprising thing was when someone first describes content testing to you and they say, “Well, we split the traffic, we show a different version to
different visitors and then you compare their conversion rates.” It’s actually much more complicated than that. You have to figure out how do we
deal with people who come back multiple times? How do we deal with people who have more than one goal? How do we deal with people who want to test completely different layouts versus just swapping out sections? How do we process the data? What kinds of statistical analyses should we perform? In a lot of ways statistics is a bit like economics. There’re a lot of schools of thought out there and papers written seemingly every month about a better way to crunch experimental data, partly because there’s so much experimentation going on in the pharmaceutical industry with drug trials and things like that.

I would say the biggest surprise has been it may have appeared to be a big effort, but it’s something that was fairly like once you got it right, you’d be ok. I think it’s turning into more of a continuously evolving product that will just keep getting better, and there’s a lot of headroom in terms of how we can make the service smarter and even better than it is today. We’re always exploring those ideas.

CG: It’s surprising sometimes the kind of things that can stop people because they have questions, they don’t understand it. Some people would rather just stop rather than take a chance. With the upside potential for testing your content and increasing your conversion rate and the exponential improvement you can make by making small changes to your site, it just seems incredible that people won’t try it out. I mean, the objections we get sometimes – it’s boggling – because it seems like such a no-brainer. Test your content! It costs very little and obviously with using Google Website Optimizer you can do it for free. Why wouldn’t you be doing it? But it seems like we’re still in the early
stages.

TL: Well, that’s a good point and I would even add another perspective to that, which is that it’s not just a matter of people who may for whatever reason decide not to test. I don’t think people understand completely that every time they make an update to their site or they create a new page or they do a website redesign and they don’t do testing, they’re effectively risking harming their business. So there’s this idea that you can take an existing page, you can optimize it, and boost your conversion rate effectively for free once the optimization is
complete, and it continues to pay off benefits into infinity. But, there’s also the idea that not testing…and we’ve seen this where people will hire very qualified and respected web design firms or web designers. They’ll put together a class=SpellE>microsite or redo a current page, and
they’ll actually hurt their performance. They only know this because they did a test.

CG: Right.

TL: There’re two kinds of tests. There’s the “How do I make my page better?” test but then there’s also the “I’m going to make some changes – why don’t I make sure these changes are actually beneficial” test. That second level of testing is not intuitive and often not thought about but not doing it is almost a greater crime.

CG: It’s interesting that you raise that because I’ve been surprised in the past year that a lot of our clients have found that to be some of the biggest benefit of working with us using Google Website Optimizer is risk mitigation of a website redesign. We can actually quantitatively test which version is going to be the best for you and which is going to potentially hurt your business.

TL: Oh, totally. The interesting thing is that in previous companies that I’ve worked at, I had some involvement in direct mail, as I
think you have as well. As you know, you would never send a major mail drop without doing an initial smaller batch to see and test some different concepts, or at least make sure the one you decide to go with hits some minimum threshold in terms of its performance before you then send the millions of pieces of mail to the entire list.

It’s the same thing with your website. Why would anyone do a website redesign and show all of their potential and existing customers new content, effectively, and have no idea if it was even better than the old content other than the fact that in your opinion it seems to look better? It could have all kinds of navigation issues, it could have all kinds of legacy issues where people expected things to be in certain places and you didn’t really factor that in, or it could have just has some major flaws that weren’t initially apparent that you only learn about when it’s too late.

I think this idea of producing web content without testing is a lot like riding a motorcycle without a helmet. It really is dangerous but the only thing is it’s even worse than that because when you crash and you have damage in the web, you don’t really know it. It’s a silent accident these website redesigns or new web page designs, and we are trying to help elaborate a bit on that and provide people with some data. Then they’ll know for sure “Oh, actually the new design is great, and now we can put a 100% of traffic on it,” but it’s sad that some people just put a 100% on blind faith alone.

CG: Do you think there’s a mindset issue here? There always seems to be two or more camps in the marketing space where some marketers are just all about brand and intuition and gut feeling and others, particularly with direct marketing or database marketing experience, understand measurability and being accountable. Do you think just a lot of marketers don’t want to be measured?

TL: [Laughs] It’s sort of like do employees want their performance measured in general? I think the ones that really have the goods, as it were, are happy to be measured because I think you can imagine Marketing Manager A and Marketing Manager B. If Marketing Manager A works her heart out, is highly trained, makes the right calls, produces great results for the company, I think she would want to be much more accountable and have her performance measured whereas Marketing Manager B may be friends of the VP, they go way back, they play golf and he may be not as proficient in marketing, and he would not want to be as measured.

I think in the long run, marketers don’t really have a choice because the marketing leaders, the VPs of Marketing and the CEOs, I think are going to expect a lot more transparency and measurability out of their marketing spend and that will trickle down. So, it’s not a matter of whether more measurability and accountability are going to come, it’s just a matter of how fast and who’s going to be prepared and frankly, who’s going to
take advantage of it and propel their career because of it versus those that will get kind of weeded out because of their performance.

CG: Well, especially in a potential economic downturn or when there’s a softer economy or a softer area in particular industries, you’ve got to be more competitive. This is potentially a bit of an ace in the hole if you can pull this out and sort of leapfrog your competition a bit.

TL: You’re absolutely right about leapfrogging your competition. I would even say internally that there is an internal marketing budget market, and different product managers or private marketing managers will be asking that VP for budget to spend on particular campaigns and particular efforts. I think the ones that are able to say, “VP so-and-so, I would like to be allocated a hundred grand for this new social marketing campaign where I’m going to drive people to the site,” and for that person to say, “Just so you know, historically, our cost per
acquisition across all channels was $75, and here’s my model which suggests this new ad campaign or this new marketing effort will bring it down to $40. I’m going to be doing all kind of web analytics tracking and website experimentation and optimization.”

I think that person is going to really shine and be able to get the budget whereas the other person who is like, “Well, last quarter I got one hundred grand so I guess I want one hundred grand this quarter,” and isn’t able to answer those tough questions is really going to be at a disadvantage.

Of course, then your point: across companies when you have Competitor A and Competitor B. Frankly, if they’re about the same size and
Competitor A has, let’s say, a $100,000 budget and so does Competitor B, if B can convert at a much higher rate, they’re going to be able to outperform Competitor A sometimes 40-50%.

CG: You know what? That’s a great point you raise about the internal politics and competition internally within a company. One of our biggest challenges actually has been over the last year working with Google Website Optimizer has been that we keep losing our clients. It’s not that we keep losing our client companies, but the clients that bring us in are getting promoted, they’re getting new jobs, and they’re no longer our client contacts, so we keep starting with new people.

I don’t want to single anyone out or name names, but it’s this strange trend that our clients keep rising in the organization and getting jobs outside in other companies. They’re getting promoted by bringing in this measurability and improvement.

TL: There’s no question in my mind that the marketers of the future are the ones that get it and have experience and track records making things happen around web analytics, website optimization, experimentation, testing and continuous improvement. When someone’s interviewing another candidate for VP of Marketing or Marketing Leader job and that person’s going to be able to say, “Yeah I’ve done these experiments. I know how it works and let me tell you some examples.” There’s just no comparison next to the guy that might have started a few PPC campaigns but doesn’t really know much more about that than keywords and creative. That area of post-click optimization is really the future for marketing, at least in our view.

CG: No, I couldn’t agree more. That’s what I keep hearing on some of the leading marketing…. Obviously the bloggers and the podcasts and everyone’s talking about this because it’s getting so much easier to do it and affordable to do it, there’s really no reason to do it. It’s
just a matter of mindset, isn’t it?

TL: Absolutely.

CG: I want to talk about the impact it’s actually having. I know we don’t want to talk about specific companies, but there’s a promise of these huge numbers: 200%, 300%, 500% increase in conversion rates. Are companies that you’re seeing that are using the product – what kind of impact are they actually having on their topline revenue? Is it making a big impact? Are people actually getting traction with this or is it just taking a lot more effort than people thought it would?

TL: You know, I think it is making a big impact. I think there are absolutely cases where people have said to me that using Website Optimizer continuously for a quarter on some key landing pages has made seven-digit impact on their revenue. Then there are those who may try it and not really think with it that are not going to get as great results. It certainly depends on the scale of the company, but increasing sales
by double digits is not uncommon at all. Frankly I don’t know any other way to increase your sales by 30% or 40%.

CG: Yeah, without spending 30% or 40% more on advertising.

TL: Yeah, it really is to the benefit of those folks who are informed, who listen to interviews like this and try out new tools. They’re going to have a huge advantage over the folks that are still sending traffic to underperforming pages and may not even know it.

CG: What are the biggest barriers that you’re hearing from people? You’ve been touring the country for the last year talking to all kinds of people about the product. What are the objections you’re hearing? Why isn’t everybody doing this?

TL: Sure, I’d be happy to tell you. Chris, I think this is going to have to be our last question because I’m about to be booted out of my room. [Laughs]

CG: [Laughs] Ok. No problem!

TL: A lot of activities going on here at Google.

I would think the biggest obstacles are #1 lack of awareness, where people don’t even know that you can do testing and then secondly, lack of an understanding about how testing tools like Website Optimizer work and how easy they are to use and how we provide a network of consultants who can provide that VIP service like WiderFunnel does. Then, I think #3 is probably the importance of coordinating between IT and Marketing, and this is something you hear with almost every online ad campaign. That coordination that’s required and the dependency between Marketing and IT are not trivial. I think those are probably the biggest obstacles and with that third one we’ve done a lot of things on the engineering side to minimize the dependence on IT, such as allowing you to just add tags once and then reuse them for multiple experiments on the same page and minimizing the number of tags you have to add even the first time. We’ll continue to try and figure out ways to make that easier, but I would say overall the biggest one is people don’t realize how much they can gain by doing testing and how a testing tool like Website Optimizer is out there and free. The more that people know, the more they seem to like it.

CG: Well, I know you’ve got to jump out of your room there. We’ll get you back soon. I’ve got a bunch more questions, and I’m sure you’ve got a lot more that we could discuss about this.

TL: Alright, thanks Chris and sorry to have to run out like this. I’ll talk to you later.

CG: Alright, sounds good. Thanks Tom. Bye.

TL: Bye.