 |
« | Week of Jun 15 | < | Individual Entries | > | Week of Jun 29 | » |  |
 |
 |
Anatomy of a News Aggregator
design, internet, patterns, software development
June 28, 2003, 05:44 PM
For the past couple of days, I've been working on the interface and architectural design for Newsable. Since I am a strong believer in the claim that the design decisions that determine whether an application is usable are not purely screen deep, but may reach far into the application's architecture, I wanted to get a rough idea of the interface design and information architecture of the application before deciding on the core components and patterns of the internal design and how they will interact. That said, I don't yet have digital copies of the screen designs, so I can't easily post them. Once I get some prototypes together I'll put them up along with a justification for my design decisions.
So on to the architecture. Newsable is divided into two parts, the Harvester (whom I'm code-naming "Cain") and the Publisher (whom I'm code-naming "Able"). This reflects the two main functions of a news aggregator; it must collect news stories from several potentially disparate sources and pull them all together into one location. Then it must catalog and organize these stories and, most importantly, provide an interface so the user can peruse them in the ways that are natural for her. And thus we have developed Newsable's core metaphor: Newsable is a gatherer who searches the various news sources looking for new stories, a librarian who catalogues these stories and manages their metadata, and finally a publisher that arranges this information into a form that is presentable to the reader.
First, we'll examine Newsable's preliminary object model. The details of this diagram are likely to change frequently, but it's worth examining now to take a look at the key terminology and metaphors in Newsable:
Newsable works with any number of users subscribing to any number of sources. A "source" is a single news producer. Note that a website may have multiple sources. Usually, a source will be a feed, but Newsable supports providing "alerts" to users that a web page has been updated since they last visited it in case the website doesn't yet publish their content in RSS. The content Newsable extracts from sources is "stories". A story is usually a news item, an article, a weblog entry, etc., but it may include stuff like weather reports, stock quotes, etc. So "story" may not be an entirely accurate term, but since most stories published through RSS really are stories I felt this was currently the best choice of terms.
Next, we'll take a look at Cain, Newsable's Harvester:
Here's Cain's basic algorithm:
- Cain is triggered by the operating system's command scheduler; since The Labs is running Linux, this would be cron. I may add a means for the user to trigger Cain to run on the subset of feeds she is subscribed to, this means Able will have to have a way to send events to Cain, but this won't exist in version 0.1.
- The scheduler triggers the Gatherer, which reads the list of feeds Newsable is currently subscribed to from the MySQL database (the database is wrapped in a Data Abstraction component that hides the details of the underlying SQL data source and performs object-relational mapping) and checks each feed to see if it has updated since the Gatherer last visited it (using a HTTP HEAD). If it has, the Gatherer fetches the feed content with GET, otherwise it ignores the feed.
- Once it has all the new feed content, the Gatherer sends the list to the Librarian.
- The Librarian is responsible for comparing the information in the feeds with the information in the database and updating as appropriate. But the Librarian can't read RSS itself, so it sends the feeds to the RSS Parser module.
- The RSS Parser first attempts to determine the format of the feed (RSS 0.91, RSS 1.0, or RSS 2.0, or a broken variation of these. "Broken" means the feed is not well-formed XML. The "Broken" parsers will do their best to extract content anyway, probably using some horrendous kludge involving regular expressions. These probably won't exist in version 0.1). The RSS Parser then internally loads the appropriate specialized parser to extract the content from the feed, which it returns to the Librarian.
- The Librarian checks the database to see which stories are new and which already exist. It adds the new stories and updates the feeds' last updated times.
I plan to extend the Gatherer to honor the "skip" elements in RSS feeds that indicate how long a harvester should wait until re-harvesting the feed. Moreover, there is a publish-subscribe mechanism built into RSS which I believe is intended to allow clients to request that the server send them updates as soon as they are posted. I don't think this is widely implemented, however. If I discover this is possible, then I'll add another component to register and listen for these events to allow Newsable to update itself immediately when new content appears in its feeds.
And that's Cain in a nutshell.
Finally we'll examine Able, Newsable's Publisher.
Able is more event-driven than Cain, so it's a bit harder to give an algorithmic explanation for. Able follows the Model-View-Controller pattern. The Controller is played by the Input Processor Script, a CGI script that interprets the HTTP GET and/or POST parameters, determines the operation the user wishes to perform, then calls the appropriate method on the Command Processor component, which collaborates with Newsable's data source to play the part of the Model. When the Command Processor has finished its computations, it returns its results to the Input Processor Script which selects an Output Template, playing the part of the View, to display these results in.
Much more remains to be said about Able, but the details of its design are likely to change as the needs identified by the personas evolve, so I'll leave it at that for now.
And there you have it. The guts of an up-and-coming news aggregator laid bare.
mv Micah /california/eBay, make, make install
life & times, people
June 25, 2003, 06:25 PM
Congratulations to my good friend Micah, who has just accepted a job as a user interface designer at eBay. Or perhaps I should be saying "Congratulations to eBay, on their success in snagging one of the greatest interface designers to come out of the CMU MHCI program (or any other, for that matter)"! In a few months, if you buy or sell something on the world's online marketplace™, your user experience will be brought to you in part by Mr. Alpern.
Combine this with Kevin's success at landing an offer from the illustrious Google, and its looking like I'll have friends in high places in the not-too-distant future. I'm counting on you guys to remember me while you're out there changing the world... :)
Email Rob:
Personas and Open User Research
processes & methodologies, usability
June 25, 2003, 10:16 AM
As I described on Newsable's About page, the goal of all this user research and personae development is to provide a technique for open source project teams to design and test usable interfaces. However, thus far I haven't made this technique explicit. This post is a first attempt at doing that.
In The Cathedral and the Bazaar, Eric Raymond presents open source software development as a highly decentralized process of many collaborating minds spread out over a potentially large geographic distance. However, Raymond glosses over the fact that most open source software projects are almost entirely managed and worked on by one individual (the "benevolent dictator" approach, as seen on Linus's Linux kernel project) or by a small team of core project members (the "meritocracy" approach, as seen on the Apache HTTP server project). What makes open source unique is that there are an order of magnitude more people who contribute in small ways to the project by finding bugs, suggesting changes, and possibly even sending code patches to the core team. So in essence, the typical open source project is run like this:
This is not a bad thing. We want to have one or a small number of team members in charge of developing our software systems when at all possible; this is one of the core tenents of Agile software development and was advocated by Fred Brooks in his classic text The Mythical Man Month when he asserted that the design of a software system should come from one mind.
Good user interface design is, in this respect, not so different from good software design, and thus we can expect the basic open source method to apply equally well. A small number of people should be responsible for the design of the interface, but these people need to be informed by the experiences and findings of a large body of users of the software. But when we're designing an open source interface, what takes the place of the "source repository" in the picture?
The obvious answer is the user interface source code itself. But nonprogrammer "normal" users can't understand user interface code, much less make suggestions on how to change it. Using the interface code as our artifact would alienate a large and very important part of the user population, which contributes to the usability problem open source software is experiencing now.
So perhaps we can use the interface design itself as our artifact. We could produce screenshots and mockups and share these with the users to get their feedback on usability problems with the design (the equivalent of "bugs" in program code).
Unfortunately, it's well known in usability circles that users aren't very good at providing accurate feedback on interfaces they can't use. Moreover, when users make suggestions about interface designs they generally come up with localized solutions that solve a particular problem they're experiencing but may make the interface as a whole less usable. Finally, people with the same user interface problems often have different opinions of how those problems should be solved. If they are spending all their time bickering over the solution, they may not even realize what the underlying common problem is.
What I propose, then, is that open source projects produce a set of personas, or archetypal users, and scenarios, or stories of the personas using the system, that encode their understanding of who they are designing the software for. One or more members of the core team can take responsibility for maintaining these documents. Outside contributors can provide additional ethnographic study data, user test data, cognitive walkthrough results, etc. that may alter the project team's understanding of who their users are. Users can contribute by pointing out differences between their usage of the system and the usage described by the personas; if the personas don't behave like real people, the users will be able to quickly spot the error based on their own understanding of how they work. The project team can then use these personas to agree on an interface design based on a clearly articulated understanding of user needs. So now, we have a process that looks like this:
Discussions of interface design decisions should all center around the personas. Instead of arguing over personal preferences ("I like the buttons on the top!" "Well, I like them on the side!") or about the mythical "users" ("I think users will find the buttons easier if they're on top." "Well, I disagree.") the project team can talk about the concrete user archetypes they've maintained and reason about how these users would behave if the new interface design suggestions are implemented.
In essence, this technique aims to be an "open user research" technique that closely parallels open source software development techniques. By encoding the team's understanding of their users into personas and making this understanding available to all, the team can leverage the unique benefits of working in an open environment without losing the conceptual integrity that comes with having a small number of people making the actual interface design decisions.
Email Rob:
The Ways We Read
internet, psychology, usability
June 25, 2003, 12:22 AM
Earlier this month, I ran an investigation of how people browse periodically updated websites (such as news sites, weblogs, or any page whose content changes and people keep coming back to view these changes). I ran these studies to inform the design of Newsable. They are directly intended to feed into the development of Newsable's user personas, but I promised several people I'd post the results when they became available, a promise I've been remiss in fulfilling thus far. So, at long last, here's the full writeup of my findings.
Experimental Method
To collect the data, I followed the basic Contextual Inquiry technique as described in Contextual Design. In essence, Contextual Inquiry is a quick-and-dirty form of ethnography. Like more rigorous ethnographic techniques, Contextual Inquiry relies on observing study participants in their "natural environment", i.e. while they are performing the tasks you are interested in as they would normally perform them. Unlike other techniques, Contextual Inquiry calls for lots of interaction between the experimenter and the participant; the experimenter constantly interrupts the participant's work to ask questions, clarify assumptions, explore areas of particular interest in greater depth, etc. A good contextual inquiry, then, keeps the participant "in context", or requires them to show the experimenter how they do their work rather than asking them to give vague summaries that won't contain many important details, while at the same time getting at the behavior the experimenter is interested in without having to spend months as a "fly on the wall" while dispassionately observing the participant.
For me, this translated into sitting down with the participants as they read their daily websites and asking them about what they were looking at, what they were thinking, why they chose to click this link and not that one, etc. In total, I observed eight users for about an hour to an hour and a half each. People who are trying to be rigorous about their contextual inquiries often videotape the entire session for later analysis. I just took notes.
Note that the nature of the data Contextual Inquiry is designed to collect is not statistically generalizable. Ethnography emphasizes deep investigation of the full range of rich behaviors for a small number of participants, not narrowly focusing on one tiny aspect of the behavior for a large number of participants that will yield a p of less than .05. This means that the findings I'm about to present are (hopefully) sufficient to guide the design of a news aggregator but not sufficient to make timeless generalizations about human behavior.
Results and Findings
Of the eight users I observed, two already used a news aggregator (NetNewsWire and Radio Userland). There was a clear correlation between the number of sites participants read and whether they used a news aggregator. The six who did not use an aggregator read around three or four sites every day. The two who did read upwards of fifty different sites on a daily basis. I think this speaks to an interesting quality of news aggregators. At first blush they don't appear to be very useful programs, after all, all they do is save you the minimal effort of clicking on a few entries in your bookmarks sidebar to hop from site to site. But by collecting the news from various sites into one easily-scannable location, news aggregators actually greatly increase the amount of content the reader is willing and capable of skimming regularly. This has the potential to expose people to a much wider variety of news sources and their corresponding perspectives.
All the participants I observed spent a lot of their time skimming content to see if it looked interesting to them. They all liked to visually scan for content that looked interesting; they didn't approach the news-reading task with many preconceived notions of what kind of content they were looking for, but instead glanced over what they found and made a judgement call on how interesting it was on a story-by-story basis. They generally preferred having both a title and a summary or short excerpt for each story rather than just the title; more information was more better (these observations explain why Google News decided to eschew Google's characteristic minimalist design in favor of a much denser information display).
For the most part, all participants appeared to prefer to see only the items that were "new to them", i.e. that had been added or modified since the last time they visited the site. Participants expressed a preference for sites that listed news stories in reverse-chronological order so they could read until they hit a story they recognized. A couple participants, however, actually preferred to read through the listing in chronological order and would skim down to a story they recognized then read backwards up to the top of the list. Finally, one participant remarked that she actually preferred to see the old stories as well as the new ones since the old stories helped her remember the context in which the new stories took place (if they were continuing coverage of some running event, for example).
Another common theme was the importance of the reputations of the authors of the stories and the publications they were writing from. As I've already mentioned, the participants depended heavily on their knowledge of the reputation of the authors of news commentary to determine whether they were worth reading or not. Participants would frequently read their favorite authors even if the topic of the essay didn't particularly interest them. Moreover, several participants had developed an understanding of the personality and interests of the people they read, and used this knowledge to help determine whether an article by that person was worth reading. Of all the criteria people used to determine what to read, the reputation of the author was consistently the most powerful.
Almost all of the participants would occasionally send stories they found interesting to friends via email or instant messanger. One participant remarked that she viewed reading the news as a very social activity and relied on her friends to keep her informed about sites she was interested in but not sufficiently to read on a daily basis. Another mentioned that he liked it when his friends read different websites than he did so they would have something to talk about later; he liked to be able to trade interesting news tidbits with his buddies.
After running two or three of these observations I become interested in how the participants reading habits changed over time. Although I wasn't able to observe this directly, participants generally described their reading habits as changing gradually, especially those without news aggregators who only read a few sites. Frequently, they would discover new sites by following links on their current news sites, and if the new site continued to come up with interesting content, they would sometimes add it to their daily "rotation list" of websites to check. Participants reading patterns also changed with their current interests; one participant was taking a photography course and thus was starting to read more weblogs on photography than she had before. Additionally, many would learn of new sites through friends who emailed them interesting stories, and sometimes these sites would be interesting enough to check daily. Some of the users without aggregators remarked that they knew of many sites that were interesting but not interesting enough to check every day, and thus they tended to check them very infrequently since doing so wasn't part of their daily habits. Participants with news aggregators tended not to have this problem.
Finally, one common theme was that participants tended not to remove sites from their daily rotation lists very frequently; one user (who did use a news aggregator) remarked that the cost of leaving a somewhat-uninteresting site in his list was so low that it wasn't worth the couple of clicks it would take to remove it, he only removed sites if they were both uninteresting and posted a lot of content that got in his way. Of course, this participant was also confronted by upwards of 300 stories on a daily basis...
Conclusions
The observations presented here have been distilled into the user personas of Newsable, the usable news aggregator (coming soon). If you're interested in this research, you may wish to read over the personas to get an idea of how all this comes together into a coherent picture of the user that website authors can design for. The personas are the "cannonical version" of this research and will be updated as new findings become available.
Thanks to everyone who put up with my obnoxious questions about their news reading habits to participate in my studies (without any compensation, no less). My understanding of what people need in a news aggregator has changed dramatically as a result of talking to you all.
If you have any questions or comments relating to these findings, please feel free to email me or leave them below. I plan to continue these studies as an "open source user research" project; more on that shortly.
Email Rob:
Fetching Friends' Feeds
software development
June 24, 2003, 02:55 AM
I stayed up late tonight finishing up a small script I've been working on that's serving as a technology prototype for Newsable. If you direct your eyes down the sidebar column to your left and locate the "Friends" list, you'll notice that now roBlog displays the title of the first entry in each person's weblog and links to the full content of that entry. I rolled my own Perl script that harvests all my friends' RSS feeds every six hours, extracts the relevant information, and displays it for your viewing pleasure in the sidebar. And yes, I am well aware there are already scripts I could have downloaded to do this for me; this was more an exercise for me than an end in itself.
For those few that care, the script uses the XML::RSS module to help extract content from the feeds (although I'm wondering why I bothered; the module is quirky, requires an install, and doesn't seem to provide many benefits over a plain-Jane DOM XML parser) and the HTML::Template module to help generate the resulting HTML. The rest of the code is pretty standard Perl line noise. The general algorithm for the script goes:
- Read the list of feed URLs out from a file.
- Try an HTTP HEAD to see if the feed has changed since the last time we checked, if so, GET it, otherwise pull it out of a local cache (kept in a Berkeley DB file).
- Parse the feed and extract the feed title, URL, first item title, and first item URL into a bunch of hashtables.
- Load a predefined HTML::Template and set its parameters to the feed content hashtables.
- Output the resulting expanded template to a file that gets dynamically included into roBlog's pages using PHP.
And that's all there is to it. Its a rather overengineered solution for what it does, but I wanted to try out some of these libraries and techniques so I worked them in even though they were awkward. I'm not releasing the source because I'm embarassed of it.
Overall, I think I'm getting my head around this News Aggregator problem. The "standard" for RSS is in a highly politicized state of flux right now, so that's likely to be a challenge. But given that an aggregator really doesn't do anything very sophisticated with its RSS feeds, I think the risk is managable.
Ah, the late-night hacking binge. How I've missed programming.
Email Rob:
On the Properties of Links
information, internet
June 22, 2003, 11:40 PM
I've recently become peripherally aware of a discussion that's going on among the widely-read webloggers on what makes a weblog a weblog (as opposed to a normal website). This reminded me of a discussion I had with Micah after his CHI Weblogs BOF. At the session, Micah gave a quick definition of a weblog entry, which he described as necessarily having four components: a main link, some commentary on the link, a "posted at" date by which the entries are ordered, and a permalink. At the time, I took issue with the "main link" idea; I felt that this assumed a certain type of weblog, namely the "MLP-style weblog" that tends to be centered around spreading links to interesting content. Kevin, who was also participating in the discussion, remarked that he frequently posts entries that are spawned by his own thoughts and not directly from something he read on the web, a sentiment that I echoed (to be fair, Micah emphasized that he wasn't trying to present a rigorous definition so much as he was trying to give an idea of what a weblog is).
Since then, however, I've been thinking a lot about links and how they are used in weblogs. There's been a call recently for more expressive links on the web. Right now, all an <a> tag can really say is "this text refers to this other document". As a human, you may be able to infer the author's purpose in linking that particular text to that particular document, but a machine cannot do so, limiting the analysis capabilities of web robots. But what if you could, through a more powerful <a> tag, say things like "this text is rejecting the argument presented in this other document" or "this text refers to the person who wrote this other document". This is in line with the "semantic web" concept: Berners-Lee's vision of a more meaning-driven web that internalizes the separation of content and presentation. I find this idea quite intriguing; I have a strong intuition that there are lots of interesting things we could do with the web if links could express why they were made. Merely knowing whether the author agrees or disagrees with the documents he links to could open up whole new dimensions for social network analysis tools, for instance.
This lead me to reflect on my linking habits for these weblog entries I write every now and again. I realized that I have several types of links:
- Inspiration Link: I read this document, and it directly spawned this post. This is in line with Micah's "main link" concept. If you take a look at my post about Kevin's Fury redesign project, you'll note the link to his post that reads "a project to redesign Fury" is of this category.
- More Information Link: This document contains more information about the object or concept that the text refers to. The "eat your own dog food" link in my Natural Programming and Open Source post is an example of this type.
- Definitive Referent Link: This type links to a definitive depiction of the thing the text refers to, perhaps a picture of a person or their homepage (the links to people like Micah, Kevin, and Tim Berners-Lee in this post are examples) or a book's listing on Amazon. In the most direct case, the text names a specific HTML document and the link takes you there (an example is the "Natural Programming and Open Source post" link just above).
- Supporting Document Link: This document supports the point made by the link text. The "by his own admission" link in the Fury redesign post is an example of this type.
- Source Link: This document is where I got this information from. Basically the equivalent of a reference in an academic paper. In the Corporations are Plural Nouns post, the "Center for Auto Safety article" link is an example of this type (it doesn't look like I make extensive use of the Source Link type on this weblog, at least not in its pure form).
- Related Post Link: The document I'm linking to is related to the current weblog entry. Usually I use this to connect two posts on my site that are part of the same train of thought. The reference to the CHI BOF post and the Justifying My Existence post in the first paragraph are both examples. My "Threads of Thinking" idea (whoops, I just did it again :) is intended to help visualize the structure created by these types of links. Sometimes, these also seem to double-up as a Supporting Document or Source Link.
This is not meant to be an exhaustive list of the purposes of links; I can think of many more. But I think this post demonstrates that even within the limits of roBlog, the diversity of purpose that lies behind those little blue words is astounding.
Email Rob:
Email Rob: