5 Lessons Learned: User Testing with Amazon’s Mechanical Turk.

My first foray into AMT was with using it to validate my startup idea for Swayable. I’ve also used it for 12 or so ways while building the site (that blog post coming soon).

I just launched our beta last Friday- October 22. Already moving in to V2 development now.  I hope to have those updates done by early to mid-December.  (Feel free of course to check out Swayable and send me feedback “Lindsey (@) Swayable.com”).

Being a bootstrapped startup, I can’t afford massive user testing through standards services that cost anywhere from $29 and up per tester. (Side note – I will use http://www.usertesting.com/ for some tests because I do want to hear verbatim feedback as users are going through the site.)  But for bulk testing to find issues to address I needed a different solution.

So I Used MTurk to do my testing!

 I have received GREAT results, and am fast adding updates to the site daily. I did the test with 120 Turkers and here are the lessons I learned as well as the survey/results I got

Lesson #1Test price points to see where the sweet spot is for your survey/feedback.

I received the absolute BEST results at the lowest price point.. (crazy odd if you ask me!)

  • $5.00 price point (hoping I’d get amazing detailed feedback) – Seemed to be picked up insanely fast, but feedback was moderate and I did have some spammer comments.
  • $3.00 price point – Great feedback, a lot more detailed in responses and no spammy comments.
  • .90 price point – I though “what the hell I’ll give it a go” at .90 and you know what, the absolute BEST feedback I got was at this price point, go figure!

Lesson #2 – Create a Survey Monkey for your Survey, vs. trying to do this in MTurk.

Survey Monkey is built for doing surveys, and has GREAT analytics. Why try to rebuild the wheel within Mturk? I posted a basic Mturk hit, with instructions to go to my survey monkey survey.

To view my exact survey on Survey Monkey go here, (um yeah – feel free to take the survey and send me feedback. J) http://www.surveymonkey.com/s/swayable

Here is what I asked in Mturk, super basic, but directed the user to my Survey monkey survey. (at this time, you can’t hardcode URL’s into Mturks self-serve system (I think there is some “amazon centric “ code, for hardcoding URL’s  in HIT’s that I am not aware of .  So I have users copy/paste the URL, if you do know what this special trick is, let me know. J)


Lesson #3 Sort and prioritize the results.

I made sure any bugs and errors were addressed first. And then took feedback that affected the current version of the product to make the site as is, as great as possible (vs. randomizing my dev team on feature updates before optimizing the current site/experience).

 Feedback beyond or suggestions beyond the current version are moved into my “V2” updates that are already in the works.

Updates addressed immediately – I found several bugs/users issues on the site doing the user testing:

  • Users were getting some errors within the Create Swayable process. I would not have found these errors on my own and was able to update them in nearly real time.
  • Design feedback (yes my design was hacked together by me.. I’m bootstrapped). I was able to update some text/design components based on great feedback from users on where they were confused or had questions while reviewing site content.
  • Dead links in my registration process.
  • Removed several registration components to make it quicker/easier
  • Clarified the “share swayable page”
  • Updated the getting started page to be more descriptive
  • Issues with number of sways not showing up accurately on the public swayable page
  • View count not showing up accurately on public swayable page
  • Removed some advertising components.
  • Updates text and some descriptions.

Updates added to V2 of the site – I also received a ton of suggestions for V2 features which I quickly added to my V2 spec document.  I would say ½ of my V2 features came from user testing feedback and the other ½ from elements that I wanted to update, but knew I need to get my MVP out, and quickly iterate to the next version.

Lesson #4 Test your survey with a small group first and keep tweaking it until you get the right level of response.

What I mean by this is that you have to be incredibly clear with your questions and make each question a “step” and very simple. Test out your survey with a small batch before going ‘big’ so you can make sure your questions are clear and your getting the kind of feedback response you were expecting.

Lesson# 5 –  Yes you CAN use MTurk outside of consumer facing products

What if your site isn’t a consumer product, can you still use MTurk for testing?

YES! Just break it down into separate surveys for specific testing. You won’t get “content” feedback from Mturk if your product/service is very targeted.  But you can get website/functionality feedback. Here are some ideas for sites that may not be consumer focused:

  • Test your registration process –  Make sure to let Mturk workers know that you are testing your registration process, and provide them an area in the HIT to leave there name or email so you can delete there account if they request. It’s against Mturk TOS to require registration without a way for them to delete it.
  • Get Design feedback – Test overall UI, site description etc.
  • Test site links – you can even have turkers test site links, and let you know if the link is taking them somewhere they don’t think it should take them.
  • Navigation – get feedback on your navigation flow, does it make sense. Etc.

Would love to know if any of you have had success using MTurk to user test your product?

And as always I have to insert – <SelfPromotion> Check out my site!http://www.swayable.com , Become a Fan on Facebook.  You can also follow me @harperlindsey on Twitter</SelfPromotion>

Cheers, Lindsey


2 responses to “5 Lessons Learned: User Testing with Amazon’s Mechanical Turk.

  1. Pingback: Using Mechanical Turk For Your Startup « carmentkelly

  2. Pingback: Using Mechanical Turk For Your Startup « dalefontenot

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s