Data Journalism and the new and exciting possibilities that the Web of Data opens up for creators and consumers of news and media online will be the topic of this first meetup in Berlin on September 1st 2010.

We have a brilliant lineup of speakers from media organisations like the BBC, The Guardian, the Deutsche Presse Agentur, the Bertelsmann Foundation, and ZEIT Online coming to Berlin and talking about data journalism and the latest development and projects in this field. Join the Berlin meetup group and sign up for the event now. Thanks to my friends at Fjord and at the Open Knowledge Foundation for their help and support!

According to Wikipedia (see page here):
Michael Ballack (born 26 September 1976) is a German footballer, playing for Bundesliga club Bayer Leverkusen and is the first choice captain of the German national football team, although he has been temporarily replaced by Philipp Lahm for the 2010 FIFA World Cup after Ballack missed out due to injury.

According to Uberblic (see his page here):
Michael Ballack (born 26 September 1976) is a German footballer, playing for Bundesliga club Bayer Leverkusen and is the first choice captain of the German national football team, although he has been temporarily replaced by Philipp Lahm for the 2010 FIFA World Cup after Ballack missed out due to injury.

According to DBpedia (see his page here):
Michael Ballack (born 26 September 1976) is a German footballer, who currently plays for Chelsea and is the captain of the German national team.

According to Freebase (see his page here):
Michael Ballack (born 26 September 1976) is a German footballer, who is currently a free agent since his release from Chelsea and is the former captain of the German national football team.

Live data matters, doesn’t it?

Quite often, the term over-engineered comes to my mind when I look at ontologies. I wonder if that’s a characteristic of the Semantic Web or of the mainly academic community behind it? For a comparison, look at Facebook’s Open Graph Protocol: People, Places, Products, and a few properties for each. Good enough, isn’t it?

Yes, I will loose every single argument over every single ontology I call over-engineered, because there will always be a specific reason to make the ontology as specific as it is. Sure.

And things must have different URIs than the documents about the things. Yes. All right.

Still, over-engineered. How about if we stop worrying about the perfect ontology and instead start curating open linked data properly? Less can be more.

The difference I’d like to use to make my point is the following: the Web is the whole thing, and in the context of the CMS discussion, it is the opposite of the enterprise intranet. The outside world as a whole, from a point of view within the enterprise firewall. On the other hand, the web emphasizes the distributed aspect, and not so much whether it’s internal or external content.

For many organisations the shift towards using external content (from the Web) in their business processes is a huge shift in mindset, and one that feels scary. Because – in theory - you can’t control the outside world as you can control the inside world.  However, the realization is that when it comes to curating data, the editorial power of the community can be so much bigger than your own, so that specialized community websites like Musicbrainz do provide higher quality and more up-to-date data than your own team could. The BBC is probably ahead of the industry, but I believe that this mindset will eventually become common.

What I’d like to argue is that this editorial power is a result of the specialization of websites, their communities, and tools. The Web as a whole is quite a messy place. But when we talk about the distributed aspect of a content web on the other hand, that is a pool of separate, specialized, connectable and interoperable services and sources. Musicbrainz is not necessarily a good data source because it is on the Web, but it is a good data source because it’s maintained and curated by a specialized community of music fans.

The distributed published nature of open linked data is vital to the health of the datasets. It means that each dataset is curated by a community that cares specifically about that data. However, this argument of distributed publishing and curation is in my opinion not only valid in the public space, but also for data and content sources within enterprises, which with their many departments and teams form webs of their own. The scientist running an experiment for example is who knows best about the resulting data, so let her curate it. The sales department has the best understanding of their sales data, the product development team of their knowledge base.

In the end this is all similar to the discussion about open data (as in data on the Web) and linked data (as in a technology for data webs). They are complementary, but each of them has its own and independent value proposition.

If we had a Web of Data, what would you build?

I’d like to start a challenge to get people to talk about some more concrete ideas for applications. The rules are easy:

1. Assume all the world’s open data is available to you, and you don’t have to worry about data formats, schemas, integration, etc.

2. Your idea is concrete enough that you can imagine yourself or someone else actually implementing it. No magic, no artificial intelligence, no algorithms that don’t exist today.

3. Write about your idea within the next 30 days (before May 8th) and ping this post.

4. If some of the data sources for your app already exist on the Web, name them.

Here’s my idea: If we had a Web of Data, I would built an application for painless travel planning. It would integrate flight plans, train timetables, bus routes, car rental offers, etc. And the user would be able to just say: I want to go from A to B: Find me the best/cheapest/fastest routes.

Because today travel planning is a pain: often your destination city has no airport, so you have to choose one of multiple airports of nearby cities. And the optimal choice for that depends on train connections from those airports to your destination city. And the choice of a train station depends on bus connections. And so on. With a Web of Data, an application could do all that combining for me, the same way flight booking sites do that today for just flights.

What is your idea? Spread the word, and post the idea either in the comments section or write it down on your own blog and ping this post within the next 30 days. I’ll set up a poll to select the best ideas from all submissions.

I stumbled upon a blog post describing the challenges of providing a single view of enterprise data sources. That post and the previous of its series describe typical corporate IT: SOAs, SAPs, IBMs, multi-million dollar integration projects, legacy systems all over the place, tons of code for special purpose application integration, etc. The usual ecosystem that has grown over decades, which from the outside might appear like a mess, but can not be replaced or consolidated. Because consolidating all those systems would be unaffordable, and might sometimes be politically difficult to pursue, because for each individual department and applications, things mostly work well enough the way they are. It’s the cross-application integration (and hence often cross-department) integration that causes headaches.

My friends at the BBC and I wrote a paper describing the challenges at the BBC and how Linked Data can help (read chapter one). Enterprise-wide IT system consolidation is not an option. The desired solution is something on top of existing systems that provides integrated views of all data repositories and workflows without the risk of breaking existing applications.

The blog post above describes a case of “small pieces of information about customers [being] littered throughout the data center”, in the billing system, marketing system, CRM system, support system, etc. The suggested solution is data virtualization (or federation): A software that allowes users to define aggregated views over the different pieces of data, and analysts and application developers query those views instead of querying the multiple different data sources directly. The integration layer acts as a middleware and retrieves the information from these disparate systems real time when requested. Whether or not to virtualize or materialize the integration layer (i.e. whether data from sources is retrieved at query time, or gets replicated into a central repository) depends on the concrete case. Complex data joins are difficult to virtualize, and query execution time is often much worse than in materialization systems. Also, in virtualized views there is the problem of availability: If one data source is unavailable, your queries will not execute, so you’d want proxies anyway.

The power of these integration layers is that they separate the business logic from the data sources. IT staff can focus on the data they need and don’t have to deal with the different systems that data is stored. They can start glueing data together instead of glueing systems together in a point-to-point manner.

Now, where does Linked Data fit in? Linked Data enables you to push the integration down to the level of the actual data. Think of it is a network (or web) of all the little pieces of information: One customer in your CRM system links to her doppelgänger in the billing systems.  So the data object about Anna Smith in the customer database links to the according Anna Smith in the billing system. And that Anna Smith links to her doppelgänger in the support system, and so on. Applications can follow these links through the different systems and that way get all the data about Anna Smith. The beauty of that? The links do not have to stop at your firewall. Your data objects can link to data sources on the Web or to your suppliers as well.You can link to whatever other Linked Data source you want, the technical barriers disappear.

Disambiguity is another important aspect. There are probably many Anna Smiths in any large enterprise customer database. With linked data, the objects become unambiguous, like a database IDs. Anna Smith lives in Cambridge? There are unambiguous IDs for the Cambridge in Massachusetts and the Cambridge in England, so Anna can link to the correct one unambiguously. And if you use a link to a linked data source on the web, like Geonames, your applications can fetch information about Cambridge from there. Once the disambiguation is done, and links between objects are established, they are available for all your applications to use.

All that makes data integration much more lightweight and agile, and at the same time much more powerful. And your integration layer software can do much more clever things in a more agile way. Is there still a need for that integration layer? Yes, there is. The integration layer becomes the place where the links between data objects get managed, where data collections get curated, where it gets defined which data sources and pieces of information to trust for which use case, where data collections are built, and where the data from all your enterprise and web data sources gets consolidated. Providing the single point of access into the web of data that exists in your enterprise and on the Web.

There are very few problems that can be mapped to a simple query-answer model. For most search tasks, reviewing intermediate results is an important part of the process. We can’t simply delegate that to the machine.

I call those exploratory search tasks: Your interests (and hence the result) changes while you are reviewing the information presented to you. Ian Dickinson and I wrote a paper when we worked at HP Labs about how faceted browsing of structured data can facilitate this kind of exploratory search through an operation we called pivoting. David Huynh, author of Exhibit and now with Metaweb, calls that set-based-browsing in his Parallax paper. Both Humboldt (the browser demonstrating our pivoting operation) as well as Parallax (David’s set based browser) were more or less just research prototypes, but I think already demonstrated what could be done in this space.

I’m convinced that the answer machines are not the future, but exploratory interfaces are. Or can you imagine yourself someday saying: “Computer, where do I want to eat today?”

One example of a exploratory search: Your inital query is to find an offer for a particular music album to buy. While searching, you discover that the band played a concert in a venue near to you that you didn’t know yet. You explore the venue and the upcoming events there, and you discover a concert of a different band. You listen to samples of their music, and then decide to buy one of their albums. So in the end, you bought a music album. But a different one than you planned to…

What started out as an idea a year ago now became reality. Today, we have released uberblic.org, a service for integrating the web of data.

The project announcement has a summary and screencast of what Uberblic is and how it works.

I’m very excited about today’s release! Many thanks to my friends at Talis for their help and support!

But there’s a huge problem: Not earning any income isn’t equivalent to non-commercial usage. A consultation done by Creative Commons has shown that a large share of content publishers consider all services and web sites run by a for-profit organisation commercial usage. Even if those offerings do not generate any income at all.

That creates a large uncertainty for companies and startups: You don’t know whether a data license actually permits the usage of content for your free service, but instead you need to sign custom agreements with each data publisher, approving that your definition of non-commercial meets their definition.

I doubt that was the intention behind data licenses such as Creative Common Noncommercial…

Let me cite from CC-BY-NC 3.0: “You may not exercise any of the rights granted to You in Section 3 above in any manner that is primarily intended for or directed toward commercial advantage or private monetary compensation.”

Both primarily and commercial advantage are quite vague, and that’s the main harm:
License terms should not be vague!

Looks ok? Maybe on the first sight. But in my opinion, it’s a big mistake.

Let’s first look at the reasons for that change in the URI schema. It aims to provide a solution for semantically ambiguous properties. For example, the word “length” can be used to describe the long dimension of an object, like the length of a bridge. But it can be also used as a synonym for the runtime of a song or movie (like 90 minutes). Now, with only the one URI http://dbpedia.org/ontology/length, it’s unclear whether the range of that property is measured in metres or minutes (let alone inch, feet, and miles ;) ) So in order to properly represent the two different semantics, we need two different URIs. Consensus so far…

Now there are to possible solutions: Either you use two different property IDs (such as length and movie_length), or you use two different namespaces. The DBpedia team chose the latter. The problem is that they did the partitioning for every single property, even those unambiguous ones. And since the DBpedia ontology wasn’t entirely carefully designed upfront, but is instead due to community refinement, that leaves us with URIs that will most probably break in the future.

See for example http://dbpedia.org/ontology/ceo. Its domain is http://dbpedia.org/ontology/SoccerClub, which seems kind of strange, but is due to the way how the ontology was created: The Infobox Soccer Club was probably the only one with a ceo property, so the domain of http://dbpedia.org/ontology/ceo became SoccerClub. Clearly, once the community starts refining the ontology, the domain will be changed to Company or something similar more reasonable. That wouldn’t be much of deal since we’d only see a changed statement about the ceo property. But with the new URI schema, the SoccerClub is part of the property’s URI (http://dbpedia.org/ontology/SoccerClub/ceo). We have a problem…

If we’d change the property URI to http://dbpedia.org/ontology/Company/ceo now, all existing queries will break. If we leave the URI as it is, the encoded SoccerClub becomes misleading. It would have been even better to use http://dbpedia.org/ontology/abc123/ceo instead, since that string doesn’t suppose to have any meaning.

The simple, straightforward solution would have been: use different IDs to disambiguate properties which actually need to be disambiguated. Or to do it like Freebase and partition very carefully by topic…