Smartphone market update – Q4 2009

Analysts in the mobile space have been putting out copious updates with Q4 2009 data.  Here is a quick roundup.  Since I newly became an iPhone fan I poked a bit deeper for iPhone statistics.

Here are my key takeaways (source: IDC)

  • Mobile devices are selling again. Hallelujah!
  • Global smart phone share is growing at a robust pace. In Q4 325.3 million mobile devices were shipped, of which 54.5 million are smart phones (17% share worldwide).
  • Top 3 vendors for this quarter remain: Nokia (38.2%), RIM (19.6%), Apple  (16%). Nokia came back up after a poor Q3.
  • Motorola is suddenly relevant again with the launch of the Droid and the CLIQ/DEXT (4.4% share)

Add to DeliciousAdd to DiggAdd to FaceBookAdd to RedditAdd to StumbleUponTweet this

iPhone, FINALLY

After years of denial, I finally caved and switched to an iPhone after going through the following devices in the past 5 years:

  • A Palm Treo 650 (2005-2007)
  • A Samsung Blackjack II (2007-2009)
  • A BlackBerry Curve 8900 (2009-2010)

iPhoneBeing a cheapskate, I ebayed my iPhone 3G 8GB used.  So where does that place me in the technology adoption curve? Am I:

  • In the early majority for smartphone technology, or
  • In the laggards category for iPhone adoption?

I must say now that I’ve finally joined the club, I feel extremely stupid not to have chosen it over the 8900.  I miss my QWERTY keyboard tragically, but the UI! And the apps! Wow, the apps!  I’m a total convert after 3 days of use!

Add to DeliciousAdd to DiggAdd to FaceBookAdd to RedditAdd to StumbleUponAdd to Twitter

Usability research in the lab

This is the seventh post in my Customer Research series.

Usability research for consumer electronics products can be very costly.  There are companies that specialize in doing it the right way, with high end audio/video equipment and multi-stream video editing and compositing integrated into the program.  The deliverable is typically an incredibly insightful presentation with snippets of video that tell a compelling story all by themselves.

Since I work with startups and small businesses, I have never had the luxury of doing it “right”.   My theory is that some research trumps no research.  So I butcher best practices until they become unrecognizable but affordable (deepest apologies to Scott Weiss who taught me how to do it the right way!)

I usually start with a research protocol that clearly states the questions we want to answer, provides a guideline for recruitment and lists the props required for the session, which includes a mini-DVD camcorder and a tripod.   Then I design the session, which is videotaped throughout.  I try to stay inside of 1h if at all possible.

Let’s say I am comparing the usability of two smart phones for working moms aged 35-45, with at least one child under the age of 12 living in their house.  The session could look something like this:

  • Introductions and orientation – explain purpose of research to subject and let them know what to expect (5 min)
  • Execute any paperwork, such as an NDA, a photo and video release forms, and a profiling questionnaire (5 min)
  • Ask the subject to familiarize themselves with Device 1. Product manuals are provided to the subject. (5 min)
  • Repeat with Device 2. (5 min)
  • Ask the subject to execute a scripted task list for Device 1.  Tasks tend to be fairly specific – for instance, I could ask them to make a call, send and receive a text message, check traffic, take a picture, upload a picture to a computer, take a video, etc.  Ask them to verbalize what they are doing as they try things out (but do not offer hints or commentary – we are there to watch and learn, not to talk.)  (10 min)
  • Repeat with Device 2. (10 min)
  • Debrief – loosely guided interview to ask subjects to rate the usability of each task on a scale of 1 to 5, as well as answer some open ended questions about their general impressions and perceptions (20 minutes)
  • Present the incentive check (typically $50-100, depending on the nature of the study).

This format is great at providing a sanity check for the out-of-box experience for consumer electronics devices.  Can the end user figure out how to set up a new device and get it working without groaning and gnashing of teeth?  Lots of times they can’t.  I’ve learned so much about what’s wrong with the current packaging design and documentation from watching subjects struggle through product setup.   It is very hard to keep quiet and not offer suggestions along the way… but the learnings are priceless.

As with all other kinds of research, I am aggressive about inviting engineering team members to be observers in these sessions.  This is the best way to help them understand who they are designing for and why certain feature enhancements are necessary to ensure an awesome user experience.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Focus groups vs. roundtable discussions

This is the sixth post in my Customer Research series.

When I do qualitative research, invariably someone would say: “oh, so you are doing focus groups!” which usually makes me cringe.  The reality is that 99% of the time I am not doing focus groups.   I may be doing detailed interviews or observations which are 1-on-1 techniques.  Or I may be doing a photo essay or journal study.  Or I may be doing roundtable discussions.

For those, I adhere to the same best practices one uses to run focus groups:

  • Include no more than 8 participants per session.
  • Craft a screening questionnaire and recruit carefully to ensure you get the right crowd.
  • Separate the genders. You get much more candid discussions that way (especially for younger demographics).
  • Control the discussion. Ensure everyone at the table has a turn (including the reticent ones).
  • Record the discussion on video for future reference.

Such a roundtable discussion becomes a focus group only if two more criteria are met:

  • The moderator is an independent third party who is engaged by the company to do this research.
  • The discussion is held in a research facility with one-way glass.

I’m old schooled.  I insist on using an independent moderator for focus groups.  In my experience, employees tend to be too close to the products and services provided by the employer.  They have assumptions and expectations that may impact their ability to be impartial.  A third party moderator has no such baggage. He or she is free to learn about the product by asking probing questions, then lead a discussion where more probing questions are asked.  The resulting quality of the discussion tends to be much higher and more unbounded for this reason.

As for the facility, the one-way-glass room has superb benefits. It allows a lot more people from the company to observe. It also  removes employees from the participants, helping to foster a more genuine discussion. It costs more than using the company lunch room, but I have never felt like I didn’t get my money’s worth in such a facility.

What’s your take? Would you do a focus group in a hotel meeting room? I would love to hear your thoughts.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Top story of the week: iPad

iPad‘Nuff said!

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Using detailed interviews to build personas

This is the fifth post in my Customer Research series.

There are many awesome posts on personas, including this primer by Steve Johnson at Pragmatic Marketing, and this post about how personas serve the CEO and the executive team.  So I am not going to belabor the point.  I’ll simply provide a case study of exactly how I go about building personas.

Before we begin, we must decide whether we are building buyer personas or user personas.   Since I have always worked on the PD/PM side (instead of the PMM side), I almost always end up making user personas first.   Here is the process I use.

1.  Pick a market segment.  Size it.  Make sure the total addressable market (TAM) is big enough to matter.

2. Generate prototype personas.  Draw on your company’s tribal knowledge / theories about customers and prospects, as well as from your own life experiences for this.  Make sure you state assumptions about needs, wants and expectations.

3. Create a screening questionnaire.  Work with a recruiter to find subjects.  Evolve screener as necessary (tweaking the prototype personas as needed).

4. Plan the interviews.

  • Pull together the equipment: a mini DVD camcorder, a tripod, fully charged batteries, camera, laptop for notetaking. (Mini-DVD camcorders result in the least postprocessing overhead.  Finalize the disk and you can watch it instantly.)
  • Staff each interview with 1 researcher who will lead the conversation, and at least 1 support person (in the role of audio/video monkey and notetaker).
  • Every researcher should plan on going to several interviews (overlapping with each other for some of them).
  • Cycle as many staff members through the support role as possible. This exposes everyone to the process and gets people out of the office and into customers’ environments, which is always good.
  • Plan mid term and final debrief sessions with the research team.
  • Plan other work around this work.  It takes at least 1 – 1.5 person days to process each interview. You also need time to reflect on the results.  Expect each researcher to be completely consumed for a good 75% of the time.
  • Share intermediate work products with the team early and often.  Transparency is key to buy-in.

5. Do the interviews. Here are some tips I’ve collected over the past 14 years (your mileage may vary).

  • For best results, detailed interviews should be done in the target environment for product use. That said, an interview at a coffee shop or some other neutral environment is better than no interview at all.
  • Start with an introduction and a warm up section where the subject gets comfortable with the researchers, then ease into the topic of interest.
  • Keep the interview guide short and sweet. Questions should be open ended, inviting the interviewee to talk in their own style and to show their personalities.
  • Be prepared to take the conversation where it wants to go, not where you want it to go.
  • Make sure the subject talks more than you.
  • For certain types of products, you may have to match gender. (e.g. research around a feminine hygiene product will require a female interviewer).
  • Don’t even think about showing a product or product concepts. Discuss your product at the end only if there’s time left over.

6. Build the personas.  With luck, you will have recruited carefully to get the right people, and after 10-20 interviews, they will have self-organized.  You should now be able to build composite personas in each bucket of subjects who share key needs, wants, expectations, and attributes.  Check these personas against the people you met and tweak until they work well.

Once the primary, secondary and non-personas are built, you can begin to use them as a proxy to help you envision what benefits they require, what products or services can deliver these benefits, and how those products and services may look like.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Common ethnographic techniques

This is the fourth post in my Customer Research series.

The word “ethnography” has such a grand sound to it.  I’ve seen seasoned executives get intimidated by it. They say things like “Whoa – that’s big agency stuff! We don’t have budget for that!”  To that, my standard response is: “It’s not rocket science!  We can do this”

And it really isn’t rocket science. It’s just hard, detailed work.  Lots of it. You don’t need a degree in anthropology or psychology. Anyone with an open mind and great listening skills can learn to do it.  If you can’t listen more than you talk… well, perhaps you shouldn’t be a product person after all.

And the budget can vary wildly.  You can spend $250,000 if you outsource it to a big agency and you have a global market.  You can train your staff to do it themselves under some guidance for well under $25,000, using a recruiter to find subjects for you.  I’ve run projects under $2500 where I did the recruitment myself using Craig’s List.  So your mileage may vary.

Now what exactly is ethnographical research in the context of product development?  In the simplest terms, it’s a set of qualitative techniques that places the researcher in the environment and/or the mindset of the subjects.  One listens and observes subjects in the environment that the product is meant to be used in, without showing the product or product concepts. One then derives the needs, wants, expectations and workarounds for the subjects, and uses this information to drive product definition.

There are three techniques that I am a big fan of (mainly due to their high bang for the buck):

  1. Detailed Interviews. Researchers meet with a subject for one to one and a half hours.  A researcher asks open ended questions to get the subject to tell a story about the domain of interest.  The interview guide tends to be very high level and the researcher is trained to mix things up and respond to new threads that come up in conversation.  I like to have two to three researchers at each interview so one person can drive the conversation while the other person mans the audio/video equipment and takes notes.
  2. Observation or shadowing. Researchers set up their audio/video equipment in the environment where a product or set of products is being used, and simply hang out with the subject while the subject uses the product.  The researchers asks questions when necessary, but by and large they behave like flies on the wall.  They are there to watch and learn, not to talk.  The subject is disturbed as little as possible.  This could be a multi-hour proposition – I once shadowed someone for 8h with a coworker.
  3. Immersion. Researchers use the product or a related product for an extended period of time to get a first hand understanding of long term product use. Researchers can record their extended findings via a journal or a photo essay. I also like to write a debrief document at the end of the immersion to sum up my key takeaways.

There are other techniques too, based more on self-reporting by end users. Examples include journal studies or photo essays involving customers.  These are all valid approaches.  When used with detailed interviews and/or observation, they can help round out the picture of the customer’s problems, needs, wants, expectations and so on.

The biggest challenge for this technique is the amount of time it takes to do a good job. Since research is done one customer at a time, and best practices suggest we work with 10-20 customers to allow the data to converge and allow time for tweaking the technique and/or the recruitment criteria, it takes corporate level commitment to pull off a project like this.  It’s a lot of long nights too since sometimes the research must be done after hours when the customers have time to work with the researchers.  But if we keep the focus, and we recruit correctly, generally by the 5th or 6th interview or observation session, a pattern will begin to emerge.

You know you’ve nailed it when you start hearing the same thing from everybody over and over again.  It’s an incredible feeling to get there and know you’ve built the knowledge that will guide the product team moving forward.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Asking the right questions of the right people

This is the third post in my Customer Research series.

One of the trickiest things about customer research is subject recruitment.  The quality and applicability of research results is directly proportional to one’s ability to ask the right questions of the right people.

There are generally two types of customer research that directly impact product development:

  • Product discovery research.  This is what typically happens at the beginning of a new product development effort. The goal is to understand who we are developing products for, what their problems are and what solutions may serve their needs and meet their expectations.
  • Product research. This is typically done after a product launch and the goal is to investigate product utility and usability and to gauge customer satisfaction.

Product discovery research should be done with prospective customers, not existing customers, for two reasons:

  • Existing customers of a previous generation product may not fit the primary persona for the new product, so we may end up asking the right questions of the wrong people.
  • Even if existing customers are smack in the sweet spot for the new product too, their expectations for the company’s value proposition have been set by the specific feature set offered by the previous product. Their needs may not be any different but their wants will be shaped by what they know about the previous product. We would be asking the wrong questions of the right people then – these folks are much better equipped to help with product research instead of product discovery research.

Product research, on the contrary, is best done with real customers and end users who would have much more to share about long term use of the product.  One can do some product research with prospective customers too, but generally unless there is a desire to start selling the product to a completely different buyer/user persona than before, it is better to stick with existing customers.

That said, nothing is absolute in customer research.  A certain amount of insight can always be generated by a bit of cross-recruiting.   For instance, during product discovery research, one can test product features with a customer advisory board who can provide a rapid sniff test without imposing the overhead of recruiting new subjects.  They can also comment on whether key features are missing in their mainstream workflos.  During product research, one might want to test product usage with a completely new target demographic and see whether the product as it stands today can be expanded to serve new market segments.

In general I believe in being highly creative and mixing things up when doing research. Every product development project is different and when one looks at the facts at hand, it will be quite obvious which types of research efforts will yield the most bang for the buck.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

From Personas to Functional Specifications

This is the second post in my Customer Research series.

Many people have asked me how persona work leads to product definition.   Here is my personal take.

In any new product development effort, we need to ask the following questions:

  1. What is the market segment I am shooting for, and is it big enough?
  2. Who am I making this product for?
  3. What are the problems that customers are trying to solve?
  4. How may our core competencies be applied to solve these problems?
  5. What are the product features that may deliver these solutions?
  6. What is the end user’s mainstream workflow, and what minimum feature set is necessary to service that workflow?
  7. How would the end user want these features to work for Version 1?
  8. What would the roadmap look like for future releases?

Item 1 is a market sizing and analysis exercise.  It is based on quantitative data from analyst reports and from internal market analysis.  It builds the business case for a new product, and helps establish a basis for interest that justifies further effort.

Items 2 and 3 are addressed by persona development.  I like to interview prospective customers in the environment they will be using the product in about their thoughts on the problems that the future product may address.  I listen between the lines and then derive the needs, wants and expectations of the prospective customers.

The key words are “prospective” and “expectations”.   Expectations are set by experiences, which is why you will get a different result if you interview existing customers of a previous-generation product, as opposed to prospective customers that represent your target market.

Item 4 is a cross functional discussion involving the product manager, the engineering lead, and a proxy for an archetypical end user in the sweet spot, representing the primary persona we are trying to target.   For instance, if the target user is a 65 year old, there should be an older person in the discussion to represent their point of view.

If the core competencies do not support a viable solution to the customers’ problems, the team would have to go back to Item 1 and rethink the target market and the business case.   Developing new core competencies is an engineering research activity that precedes product development.

Item 5 is about envisioning features that will deliver the solutions needed by end users, and Item 6 places them in the context of user workflows.   One is meaningless without the other.

At this point is generally a lively discussion with all the possible ways to serve up benefits to customers.   Rather than going around in circles in the office, I reach out to customers to get their feedback.  I employ a few techniques for this:

  • Get customer feedback with paper prototypes.  This can be done either as a round-table discussion (2-3 groups of 8-12 participants each) or remotely by phone on a one-on-one basis.  This should be done with both prospective customers and existing customers so we can get the full range of perspectives for people in the sweet spot and for early adopters. Results should be interpreted separately, of course.
  • Beta testing with hacked prototypes where feasible. This should run for 2-4 weeks with prospects AND existing customers with separated analysis of results for the two constituencies.
  • A product concept test for the new product, executed as an on-line survey to prospective customers only.
  • A comparative product concept test, pitting the proposed product against a control product, executed as an on-line survey to existing customers and prospective customers.

Once we get through 1-6, we will have a clear idea of what features ought to be included.  The next step, Item 7, addresses how those features should work.  I like focus groups for this detailed phase, but if budget doesn’t permit, we can cover this with informal round tables or virtual focus groups.  I like qualitative techniques for this because it is difficult to get into enough detail with quantitative techniques.

When we wrap up Item 7, we have nailed all the requirements and are ready to develop a Functional Specification for Version 1 of the new product. This should represent a minimum set of features that service the mainstream workflows of end users in the primary persona.  The product team can now execute on the new product.

The learnings from 1-7 will also feed Item 8 – roadmapping.  We will have a ready list of features that deliver secondary benefits, and we can tentatively bracket those into future releases.

This process can sound like a drag. It is not. It is simply a practical way to frame the challenge of understanding customer problems.  The whole thing can take anywhere from a few days to a few months, depending on how much the product team already knows about the market and the product, and the scope for the new product development effort.   A little thinking goes a long way here – much better to invest time up front than start development without a clear understanding of what one’s doing and why.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Why do qualitative research?

This is the first post in my Customer Research series.

I will start with a discussion on why I do qualitative research.  Some companies place a disproportionate amount of trust upon quantitative, statistically significant results presented in analyst reports.  Anything involving a handful of research subjects is viewed with deep suspicion.  Insights that their own product teams bring back from the field are dismissed as anecdotal, and the knowledge is not used for decision making.  Sounds incomprehensible, right? Alas, companies like this do exist – I worked for one of them before.  That company was not a success.

In my opinion, this suspicion is dangerously misplaced.   Numbers are sterile. They are necessary; any moderately skilled product person would stay on top of market sizing reports and industry trends by following analyst reports.  However, that provides only a sliver of the knowledge needed to design the right products.

Qualitative techniques allow us to probe deep into personas, needs and wants, use cases, habits and practices and so forth with a small sample size.  We step into the customers’ shoes and get a taste of their motivations, problems they encounter, and what they may be trying to achieve with a current or future product, in their environment with a workflow that works for them.  This, coupled with quantitative research, is what drives great product development.  It is incredibly time consuming,  but if you plan and execute it properly, every minute is worth the work.

Please join the conversation if you are a product person – I would love to hear what you think.

Add to DeliciousAdd to FaceBookAdd to StumbleUponAdd to Twitter

Next Page »


About Elaine Chen

ec-small

I'm a 17-year veteran engineering and product development executive in the local Boston startup landscape. Click here to view my full profile.

Twitter Updates