The Problem with Data Objects
patterns, software development
September 27, 2003, 04:43 PM
I've been working on the code for Newsable a lot recently, and I'm coming across a problem I confronted when working on EE way back in my code monkey days (about two years ago). The problem relates to a software design pattern known as Data Objects.
Data objects are intended to provide a consistent representation for domain data in a multitiered architecture (in fact, often the cross-cutting concern I mentioned in that earlier post, correctness checking, is implemented in these sorts of objects). Basically, you don't want to give all your code direct access to the persistence layer, but you don't want to be passing around your domain data as primitive types or layer-specific classes (along with all the ugly conversion code) either. So you standardize on an object model for your domain data and use that consistantly throughout the system. Generally the persistence layer is responsible for mapping this object model to a relational datastore.
The problem is that you frequently need many "views" of the data in order to generate the user-interface screens, and it can be hard to predict exactly what data will be necessary for each screen unless you can guarantee that all the screen designs will be fixed in advance (often not true, since design is iterative). Even then, you may find that you need a wide variety of views to fulfill the specifications. This becomes a problem when you need to create a large number of classes to represent these views; for instance, in Newsable's design I have classes for Users, Feeds, and users' Subscriptions to feeds. Should a Subscription contain the User and Feed objects that make it up? Probably, but should a User then contain all of her Subscriptions? That would be helpful for listing the feeds for the "Edit Sources" screen, but wasteful when I just want a User to authenticate their username and password, for instance. And this is only a simple example; EE had a particular series of screens that I calculated would require 12 different view classes for the same collection of data if I developed custom objects that only returned the data necessary for its screen.
I remember seeing a pattern in EJB Design Patterns that involved using hashtables instead of objects for situations such as these to provide for more flexible data definitions (although this also eliminated type safety, of course). Maybe in my next system I'll give that idea a shot if it seems to be appropriate.
Patterns and U&SA
patterns, software development, usability
July 05, 2003, 06:00 PM
At the command of my bosses, I'm working on researching the relationships between our U&SA scenarios and the software patterns found in Design Patterns and Pattern-Oriented Software Architecture Volumes 1 and 2. So far, I've gone through Design Patterns and compared the "Applicability" sections where the Gang of Four discuss the conditions under which you may want to use that pattern to the "Responsibilities" we've identified for each scenario and thought out whether the two seem to connect in any meaningful way.
I've found surprisingly many potential relationships between the Gang of Four's patterns and our responsibilities. However, it's difficult to say much that's concrete about these relationships without dictating a particular implementation. The responsibilities alone could be fulfilled by many systems (that's their whole point); the patterns may apply to some of these systems, but others they may just succeed in complicating unnecessarily. This was certainly a problem I encountered with EE; the first architecture I came up with, taken from a book on developing Websphere applications, was far more complex than it had to be for EE's requirements. Even after several refactorings, EE's architecture is probably more complicated than it should be. For this reason, I'm wondering if its possible to give meaningful implementation advice without at least assuming a particular domain, such as Windows desktop applications, web-based applications using J2EE, etc. And even then you run the risk of suggesting solutions for problems the team doesn't have. Since we have neither the expertise nor the desire to pronounce on optimal design strategies in all these possible domains for all possible requirements, maybe its just not possible for U&SA to say much at this level of detail.
On the other hand, it may be useful for developers to have a sense of which patterns may help implement the scenarios and how. This could have several benefits, including:
- Helping the team determine how to implement the scenarios without impacting other important quality attributes, such as modifiability or performance. A fear that the scenarios will negatively impact these other attributes is one of the three main reasons a team may not implement these scenarios (the other two being that they were deemed too expensive or time-consuming to implement and that the team simply did not consider them), so this could be a major boon.
- Saving the developers time in investigating solution options. We've already collected several best-practice solutions in the patterns literature for them, they just need to make these concrete for their system.
- Getting the interface designers' input on what changes, if any, need to be made to the patterns. For instance, some of the patterns don't support returning feedback on the results of an operation (Observer comes to mind). This may be a problem if user feedback on the results of the operation is required to implement the interface design.
All in all, I think this has been a useful (if tedious) investigation so far and is helping us move forward with U&SA. We are planning to publish a paper on the U&SA framework soon, hopefully in CHI and/or ICSE, which is something I've been advocating for awhile since I think we really have come up with something with potential here. Definitely something to look out for, and it'll have Rob's name on it, so you know it's got to be good :).
Email Rob:
Anatomy of a News Aggregator
design, internet, patterns, software development
June 28, 2003, 05:44 PM
For the past couple of days, I've been working on the interface and architectural design for Newsable. Since I am a strong believer in the claim that the design decisions that determine whether an application is usable are not purely screen deep, but may reach far into the application's architecture, I wanted to get a rough idea of the interface design and information architecture of the application before deciding on the core components and patterns of the internal design and how they will interact. That said, I don't yet have digital copies of the screen designs, so I can't easily post them. Once I get some prototypes together I'll put them up along with a justification for my design decisions.
So on to the architecture. Newsable is divided into two parts, the Harvester (whom I'm code-naming "Cain") and the Publisher (whom I'm code-naming "Able"). This reflects the two main functions of a news aggregator; it must collect news stories from several potentially disparate sources and pull them all together into one location. Then it must catalog and organize these stories and, most importantly, provide an interface so the user can peruse them in the ways that are natural for her. And thus we have developed Newsable's core metaphor: Newsable is a gatherer who searches the various news sources looking for new stories, a librarian who catalogues these stories and manages their metadata, and finally a publisher that arranges this information into a form that is presentable to the reader.
First, we'll examine Newsable's preliminary object model. The details of this diagram are likely to change frequently, but it's worth examining now to take a look at the key terminology and metaphors in Newsable:
Newsable works with any number of users subscribing to any number of sources. A "source" is a single news producer. Note that a website may have multiple sources. Usually, a source will be a feed, but Newsable supports providing "alerts" to users that a web page has been updated since they last visited it in case the website doesn't yet publish their content in RSS. The content Newsable extracts from sources is "stories". A story is usually a news item, an article, a weblog entry, etc., but it may include stuff like weather reports, stock quotes, etc. So "story" may not be an entirely accurate term, but since most stories published through RSS really are stories I felt this was currently the best choice of terms.
Next, we'll take a look at Cain, Newsable's Harvester:
Here's Cain's basic algorithm:
- Cain is triggered by the operating system's command scheduler; since The Labs is running Linux, this would be cron. I may add a means for the user to trigger Cain to run on the subset of feeds she is subscribed to, this means Able will have to have a way to send events to Cain, but this won't exist in version 0.1.
- The scheduler triggers the Gatherer, which reads the list of feeds Newsable is currently subscribed to from the MySQL database (the database is wrapped in a Data Abstraction component that hides the details of the underlying SQL data source and performs object-relational mapping) and checks each feed to see if it has updated since the Gatherer last visited it (using a HTTP HEAD). If it has, the Gatherer fetches the feed content with GET, otherwise it ignores the feed.
- Once it has all the new feed content, the Gatherer sends the list to the Librarian.
- The Librarian is responsible for comparing the information in the feeds with the information in the database and updating as appropriate. But the Librarian can't read RSS itself, so it sends the feeds to the RSS Parser module.
- The RSS Parser first attempts to determine the format of the feed (RSS 0.91, RSS 1.0, or RSS 2.0, or a broken variation of these. "Broken" means the feed is not well-formed XML. The "Broken" parsers will do their best to extract content anyway, probably using some horrendous kludge involving regular expressions. These probably won't exist in version 0.1). The RSS Parser then internally loads the appropriate specialized parser to extract the content from the feed, which it returns to the Librarian.
- The Librarian checks the database to see which stories are new and which already exist. It adds the new stories and updates the feeds' last updated times.
I plan to extend the Gatherer to honor the "skip" elements in RSS feeds that indicate how long a harvester should wait until re-harvesting the feed. Moreover, there is a publish-subscribe mechanism built into RSS which I believe is intended to allow clients to request that the server send them updates as soon as they are posted. I don't think this is widely implemented, however. If I discover this is possible, then I'll add another component to register and listen for these events to allow Newsable to update itself immediately when new content appears in its feeds.
And that's Cain in a nutshell.
Finally we'll examine Able, Newsable's Publisher.
Able is more event-driven than Cain, so it's a bit harder to give an algorithmic explanation for. Able follows the Model-View-Controller pattern. The Controller is played by the Input Processor Script, a CGI script that interprets the HTTP GET and/or POST parameters, determines the operation the user wishes to perform, then calls the appropriate method on the Command Processor component, which collaborates with Newsable's data source to play the part of the Model. When the Command Processor has finished its computations, it returns its results to the Input Processor Script which selects an Output Template, playing the part of the View, to display these results in.
Much more remains to be said about Able, but the details of its design are likely to change as the needs identified by the personas evolve, so I'll leave it at that for now.
And there you have it. The guts of an up-and-coming news aggregator laid bare.
Email Rob:
A Pattern for Distributed Work
patterns, society & sociology
May 12, 2003, 05:36 PM
Earlier I wrote a post on technology and life in the future and mentioned the Community of 7000 pattern from "A Pattern Language". Alexander mentions another pattern that fits even closer with the ideas I expressed in that post; in Scattered Work he describes the many downsides that the strict separation between work-life and home-life does to a society, and recommends organizing towns so that work and home life are intertwined by decentralizing workplaces so they can be moved closer to homes and modifying the daily schedule so that workers can easily move between their homes and workplaces, work from home, take half-days, etc. This was commonplace in traditional societies, and Alexander believes that it is a viable pattern in modern societies as well.
There were barriers to implementing this pattern when "A Pattern Language" was written (early 1970s) but I believe that the advent of distributed information technologies, coupled with improvements in personal transportation systems, will tear down many if not all of these. Read Alexander's pattern and my post and see if you don't agree.
Email Rob:
Visions of a Distributed Future
information, patterns, society & sociology
May 07, 2003, 07:00 PM
While walking home today I was thinking about the direction wireless and networking technologies are moving in and what impact this might have on society in the long term.
Since at least the beginning of the Industrial Age, we humans have lived a daily routine that looks something like the following:
- Leave home in the morning and go to the workplace.
- Do work at the workplace all day.
- Leave the workplace in the early evening and go have fun, either by returning home or going to some other location like a bar, a restaurant, a club, etc.
- Go home to relax and sleep.
This has been the structure of our daily life, dictated to us by the protestant work ethic which, according to Max Weber, has defined the spirit of the Industrial Age.
But the sands of time are flowing, and the world isn't what it was. Information technology has already enabled easy, global distribution of information and rapid, decentralized communication, and we are beginning to discover how to harness this power. Moreover, information/communication technology will soon become ubiquitous with the advent of universal wireless networking, wearable computing, and other mobile and embedded technologies.
So what does this mean for the structure of our lives? This ubiquity of communication is starting to make the concept of a "workplace" obsolete. Projects such as Aura, Civium, and many others like them are looking into ways of ensuring that your information is always with you, will always be there for you when you need it. So there is no need to go to the office to fetch the information you need. Moreover, wireless communication technologies, of which cell phones are only the beginning, make it easy to communicate with the people you work with, either completely virtually or as a quick means of setting up a real-world meeting. So your "workplace" could be anywhere; your project group could teleconference and agree to meet in a coffee shop or a park, then you could go back home to do the work you took on. We're already starting to see this happen technologically with the increasing sophistication of computer-supported cooperative work systems and socially with the popularity of telecommuting.
All this also means that the traditional structuring of the day into work, play, sleep may no longer be necessary in the future. We have the option of decentralizing, working when we want, coordinating our schedules only as necessary to get together with our colleagues. In fact, the entire concept of a "work time" could be discarded, and our jobs can be seamlessly interwoven with the rest of our lives. Work and play will coexist peacefully, and frequently become one.
In "The Hacker Ethic", Pekka Himanen discusses how this has already begun in certain cultures of information technologists, who interweave their work into their lives, their passions, and their dreams. These technologists ("hackers" in the original sense of the word) prefer this method of leading their lives to the "nine-to-five rat race / daily grind" option.
Many people fear the interweaving of work and play, of public and private life. They're afraid of getting "too connected"; they worry that if work can be done anywhere then they will never have time to do what they want. I believe this is an artifact of the protestant work ethic, which has always clearly separated "work" from "fun" and emphasized duty over pleasure. But that was a fact of life in the Industrial Age, and the Industrial Age is fading.
In "A Pattern Language", Christopher Alexander describes the "Community of 7000" pattern as a means of capping the size of towns and other communes to humane limits. Alexander is big on decentralization and distribution as means of building healthy societies. His patterns describe a society that meets these criteria; it lives at peace with nature and provides a humane environment for its inhabitants. What's interesting is how much easier it could be to facilitate many of the changes required by such a society were we to have a robust information infrastructure that gave us all "floating workplaces" so we were free to spend time in the places we want to and cultivate those places to become even more appealing to us and our peers. The chains that pull us all into overcrowded cities out of the sad necessity of centralization would disappear.
In our hierarchical world with global superpowers, multinational corporations, sprawling cities, an obscene rich/poor divide, and many millions of lives choked to death by inhumane conditions of every sort, it is difficult to imagine that a modern society could really work in the way he describes. But perhaps, just perhaps, distributed information technologies can be the catalyst that will bring this about. It is far from certain. No technology can change the world; people change the world, and if this vision is to become real we must make it happen. But I, for one, believe it will soon be in our reach.
Email Rob:
A World of Patterns
patterns, philosophy
April 06, 2003, 11:30 PM
I recently borrowed A Pattern Language from Micah, which I've mentioned before. I've started reading it, and I'm already excited; Alexander describes patterns in ways I'd never really thought of. To him they are a true language, a personal and shared understanding of how we live in the world. He also makes an excellent point when discussing how the patterns fit together:
In short, no pattern is an isolated entity. Each pattern can exist in the world, only to the extent that is supported by other patterns: the larger patterns in which it is embedded, the patterns of the same size that surround it, and the smaller patterns which are embedded in it.
This is a fundamental view of the world. It says that when you build a thing you cannot merely build that thing in isolation, but must also repair the world around it, and within it, so that the larger world at that one place becomes more coherent, and more whole; and the thing which you make takes its place in the web of nature, as you make it.
Emphasis mine.
For those of us would-be creators who hope to change the world with our designs, I think these words are worth meditating on. The Law of Unintended Consequences is always in effect, and all too often we forget it.
On a different note, I'm heading off to CHI 2003 in sunny Fort Lauderdale bright and early (too early) tomorrow morning. Hopefully I'll have time to post a few weblog entries about any interesting memes I find there.
Email Rob:
Pattern Sighting in Management Habitat
design, patterns, systems
April 05, 2003, 02:12 AM
I was working with Matt a few days ago on our TCinC independent study, and he showed me part of a book on Systems Diagrams, which are a component of systems thinking as described in The Fifth Discipline by Peter Senge. What struck me as particularly interesting about these things was how similiar Senge's concept of system "archetypes" is to patterns.
For those of you haven't heard of them, patterns are generalized solutions to a set of problems that occur in a particular discipline. They were first described by Christopher Alexander, a building architect, in his 1977 book A Pattern Language. Alexander's architectural pattern language included patterns such as "Courtyard" and "Good Materials". The Gang of Four (Gamma, Helm, Johnson, Vlissides) applied this notion to the design of software systems in their 1994 book, Design Patterns, which is where I first heard of them. They also happen to be one of the ten thousand things I'm very interested in.
In software systems, one common problem is that you have a component that performs a function you want to use, but it doesn't conform to the interface you need it to. Here's an example from a fictitious drawing editor application:
In this situation, we have an inheritance hierarchy whose root is Shape that has a "getBoundingBox()" method. We want to subclass Shape with a class that implements a line of text, and we already have a component, TextView that we'd like to reuse. However, TextView doesn't conform to Shape's interface. So we create TextShape, which implements Shape's interface just by calling the corresponding method(s) on TextView. This solution is then generalized to the Adapter pattern:
These drawings were adapted from diagrams in Design Patterns.
The Fifth Discipline is not related to software design, but instead is a business / management / project planning text that attempts to analyze work environments to locate the "feedback loops" or parts of the work process that must be examined if sustainable organizational learning is to occur. Senge provides archetypes of problem systems that commonly appear in organizations so you can try to fit your system to one or more of the archetypes to better understand what sort of solutions to apply.
Note how similar this is to the concept of a software design pattern; in both cases the assumption is that an abstract problem in the domain has an abstract solution structure that can be distilled by looking at many successful solutions found in practice. Then this solution structure can be adapted to fit local conditions to solve specific problems.
I wonder how other disciplines apply patterns or pattern-related concepts to solve common problems. In a sense, patterns are nothing new; we're just using induction to build abstractions, then deduction to apply those abstractions. It's how the scientific method works, and how humans have learned and thought for millennia. Yet making this way of thinking and encoding knowledge explicit seems to have a power of its own. I wonder where it can take us.
Posted by S Cameron on April 06, 2007 at 11:17 AM
Great blogs. I applied three of them to a current dilemma at a recently-opened dog park. Problem: approx. 20-30 poops left-behind each day.
1. Some see the park as a community and sense community needs
such as keeping it poop-free.
2. Some Try to enforce individual pick-up (park police)
That doesn't work, as the attendees are able to stay
anonymous
3. Some try to educate others through modeling or casual
conversation. So far (10 months later) that hasn't worked
4. Some pick up any poop they see without judgment
5. Some (me) pick up any poop I see and also demonize the
bastards who free-ride
The challenge to understanding their position is:
1. the anonymity
2. the lack of defendable position they hold.
I guess this as, given the number of poops and the
total number using the park, individuals pick-up only
when being watched.
My current solution is to close the park for a few days. The reason, sanitary danger, would be posted. Whether this would cause a pain great enough to move freeloaders off their position us unknown. It may cause those who pick up "only after my own dog" to consider a better model of "I'll pick up two poops per visit (or even one poop on the likely chance that they do not witness their dog defecating).
Officials argue that such a move would suffer those who DO pick up. I would argue that, given the 20+ poops unattended daily, everyone needs a wake-up call as to a better model that incorporates (if not embraces) the existence of the freeloader.
Email Rob:
Email Rob: