 |
« | Week of Feb 29 | < | Individual Entries | > | Week of Mar 14 | » |  |
 |
 |
Nielsen: Qualitative Is Better Than Quantitative
design, psychology
March 11, 2004, 03:44 PM
Jacob Nielsen has posted an alertbox column claiming that qualitative user studies are preferable to quantitative user studies in most circumstances. His reasoning is that quantitative studies, while useful for producing metrics that can guide decision making and resolve design arguments, are too difficult to conduct properly (thus leading to biased results) and tend to produce results that are so narrow as to be useless for guiding design.
My friend "Q" (she posts anonymously, which is why I'm not disclosing her identity) takes issue with Nielsen's proclamation. She tends to favor quantitative research, and points out that it's possible to to botch qualitative results as well, and that a good research program should include both quantitative findings and qualitative observations.
I agree that both kinds of studies need to be done carefully and by people with some training in the area. I also agree that it's good to be flexible and consider many types of studies depending on your research goals. However, I can see Nielsen's point; remember that he is not writing for psychologists or social science researchers, he is writing for practitioners in the design and usability fields. Most of these people do not have significant training in statistics, experiment design, or psychology theory and are unlikely to successfully design a valid quantitative experiment. Moreover, my experience with designing quantitative studies is in line with Nielsen's observations; it took me the better part of last semester to design a study intended to provide evidence that personas helped lend focus to remote design teams in open source software (note I said "provide evidence" not "prove"). And by the end, the study was so narrow in its focus that it wasn't at all clear that it could stand up as bulletproof evidence that remote design teams should always employ personas.
Of course, as Q points out, it's possible to get bad qualitative results as well. But I've found that its fairly easy for relatively untrained individuals to uncover useful design facts about their target user population merely by watching a few members of that population do their work, or by talking to them in an interview, or by giving them some tasks to perform with a prototype interface design. Of course, sometimes you do find users that exhibit outlier behavior, but often, as Nielsen says, it's easy enough to identify such behavior by forming hypotheses beforehand and reasoning about how the behavior you're seeing fits in with your understanding of the world.
Bottom line is, even in enlightened companies a user-centered designer is likely to have only a little time (a few weeks, maybe) to run user studies. To make the most of that time, it's usually best for this practitioner to focus on broad, qualitative, fuzzy data to gain a general picture of the problem, rather than focusing in and losing all that useful information that could be guiding design just to get a few numbers.
Email Rob:
Newsable Redesign Mockups and a Request for Feedback
design, usability
March 08, 2004, 10:07 PM
I'm working on giving Newsable a facelift soon, both to improve the interaction and visual design. I've posted some mockups of the new key screens; if you're the sort of person who likes to speak words about other people's work, then here's your chance to speak some words about mine. My goals with this release are to make the visual design a little more pleasing and to improve the interaction for people who are subscribed to a large number of feeds. Also, I hope to convince Kenneth that the three-pane format isn't the be all and end all of aggregator design.
I'm also working on the design of a community system that will help facilitate a usable open source software process. The design is based on the research I did last semester in CSCW. If this sounds intriguing to you, stay tuned for more.
Posted by Andyed on March 09, 2004 at 09:36 AM
New design looks great. In the interest of ease of startup and no lockin, import/export of subscriptions lists to opml or some such would be greatly appreciated.
I have mixed feelings about the new layouts as I dramatically prefer regular repeated reading to happen in the newest to oldest view, with feed sources merged. So few other aggregators accomplish this, please don't disable it.
Posted by Rob on March 09, 2004 at 06:15 PM
Yeah, I agree that OPML import / export would be a great feature; I just haven't had the time to implement it yet. Anyone know of a decent OPML parser module for Perl?
Good point about the interleaved stories. Perhaps I can put in a preference for that behavior. I'm trying to avoid preference-overload, but for the moment it seems safe.
Posted by Geoff on March 10, 2004 at 11:31 PM
Hey, Rob. I really like the new design, it's an excellent feel.
I would second letting users choose between stories grouped by source or interleaving them. For someone like me who checks Newsable infrequently, and is subscribed to both relatively low-volume feeds (e.g., my website *sheepish look*) and high-volume (e.g., BBC news, with dozens of entries every day), it's nice to be able to see them separated by source, so the low-volume feeds don't get lost in the noise. I can see how people with different reading habits might feel differently, though.
Posted by Andyed on March 11, 2004 at 12:07 AM
...imagines a gestural mechanism for scattering feeds from headline view into chronological organization and the inline DOM code to do it...
I'd love to see a serious analysis of the Zoomracks patent and a rapid move to uncovered functionality in the "scattered cards" ui mode.
Just brainstorming, but seriously think outside the page here. It's darn easy to show/hide these days. I'd be happy with the headline view if I could easily expand a recent post. I'd even settle for load on demand via a document.createElement("script"), element.setAttribute("src", "newsable.org?feed=7&item=8").
That said, I do get a lot of value out of the interleaved view for rapid-scan using the space bar to scroll. Particularly for my topical feedster searches (see http://surfmind.com/musings/gems/forumzilla_feedster.gif) where the feed is high-frequency, sometimes irrelevant, and titles & sources are less informative as new feeds show up frequently.
){kind=link}
Keyboard navigation with n)ext, p)revious, e)xpand, c)ollapse could really help for either organization. Explicitly embedding focus into the system would allow the application to move towards the "attention.xml" notion and all sorts of fun adaptivity and collaborative filtering (based upon time in focus).
Email Rob:
Posted by Kevin Lee on March 17, 2004 at 02:35 AM
From the user-centered design perspective, we must know how to translate or convert qualitative data into quantitative data. Remember that there are usability goals based on usability attributes for each usability study. While you can get an overall "sense" of problem areas and areas for design improvements from qualitative data, at the end, you will still need to substantiate your design recommendations by quantifying the qualitative data.
Besides, just right amount of attention to the qualitative data forces you to design a well-controlled usability study which usually lacks in qualitative data driven usability study.