One of the things we’re most proud of with OpenCorporates is the way we quietly get on with making the world’s biggest open database of companies bigger and better, adding territories with the help of the open data community. Every day we make thousands of HTTP requests, scrapes, and downloads to OpenCorporates, constantly adding companies, filings, and other data.

There’s probably a blog post just itemising the stats behind OpenCorporates, but for today to we just want to focus on one in particular (and it ties in nicely with the presentation we’re giving this afternoon at the Investigative Reporters & Editors’ NICAR conference). This weekend, we finished adding Louisiana to OpenCorporates, the 11th US state we’ve added this year, and bringing the today number to 22, and the total number of companies we have to 36 million. For the record the other states are:

• Connecticut
• Arizona
• Idaho
• Maine
• Massachusetts
• Mississippi
• Oklahoma
• Philadelphia
• South Dakota
• Vermont

Florida doesn’t quite make that list, as it’s still being imported (we’ve just over a million Florida companies so far). While we haven’t yet got every state in the US, we’re well on our way, and we’ve got several more in the pipeline, meaning the day is in sight not only when you can do one search across all the US and get back every Google entity, for example, but get the whole thing back as openly licensed data, too.

Of course, there are some obstacles in the way, not least Delaware and Illinois, but we’re hoping that before we get to those they’ll embrace transparency and open data

This weekend, the EU published its public draft of its Business Vocabulary (along with Person and Location), to help make it easier for organisations within Europe, including governments themselves, to exchange information relating to companies.

Now if I haven’t already lost your attention, I think this is pretty important, and not just those who handle corporate data, but to also to all those interested in openness in general, for three reasons:

1. The vocabulary democratises the ability to share this important information, removing the need for restrictive central registers, which are inevitably tied up with process, governance and access issues;
2. The results are free of IP restrictions;
3. The fast, lightweight process was an example of a huge organisation (the European Commission) for once being focused on solving immediate problems rather than grandiose undeliverable: in short the EC went for a Minimum Viable Product.

I think it’s worth tackling these in a little more detail, but feel free to skip the first or second and skip to the process bit, if that’s what floats your boat.

[I should also state that  OpenCorporates was a member of the Working Group that put the vocabulary together and was fairly heavily involved in the Business Vocabulary discussions, having arguably done more to open company information than any other organisation in the recent history. It's also worth saying that this process wouldn't have achieved anything had it not be for the excellent work of the W3c's Phil Archer and the EC's Piotr Madziar and Vassilios Peristeras]

1 The Business Vocabulary

Like including mathematical equations in a book, the phrase ‘business vocabulary’ is an excellent way of putting off any ‘normal’ people who might otherwise be interested. Yet like the protocols that underlie the internet, getting this sort of thing right matters, and really all we really mean by business vocabulary is not some heavyweight XML schema, but a lightweight set of agreed terms and principles that remove the barriers for communication.

In this case it’s very lightweight, as there as there’s only really one critical part: what is a company, and how do we identify it? Despite the simplicity of the question, this is an area in which it’s easy to tie yourself into knots, should you want to. Countries have very different ways of thinking of companies (are partnerships or sole traders companies, for example?), and different ways of creating them too, sometimes handling them centrally, or giving the job to regional registers or courts.

The Business Vocabulary neatly sidesteps this, instead focusing on two core elements: a Legal Entity and a Formal Identifier. If you’re going to exchange information about a company it needs to be a legal entity, and it needs to have a (single) formal identifier. And that’s pretty much it. The only thing to add is that the formal identifier is made up of two parts: the identifier (e.g. a ‘company number’) and an issuing authority (e.g. a company register), which would ideally be identified by a URI.

See that wasn’t so bad was it

There are a handful of other properties that are listed, including the date of issue of the identifier, the registered address, company type, some of which are more clearly defined than others, but really that’s it. But already, it allows company registers around Europe to start publishing their data, and consuming other company registers’ data (for example, to understand the status of home companies for foreign branches they have registered) without the need for a highly centralised clearing house with its own closed system of data exchange.

It’s also worth stressing that this solution is not tied to any particular representation. It could be turned into a string identifier, linked data, or XML of some sort. Whichever is used, transformation from one to the other should be easy.

2. Open data and open standards

One of the best outcomes of the process, is that the resultant Business Vocabulary is genuinely open, unencumbered by IP restrictions. More than this, however, the whole process was focused on this outcome, with all agreeing this from the start. (In fact all participants were required to explicitly agree that their contributions would be free of IP restrictions, meaning the contributed use cases and discussions on them can also be openly published.)

This means, for example, that the list of identifying authorities also needs to be free of IP restrictions, and it’s this sort of detail which really matters when we’re talking about open standards – one solution, particularly if the vocabulary is to be used outside the EU (which it certainly could be) would be for the W3c to maintain and publish this list, given its interest in the semantic web and IP-free solutions.

3. Minimum Viable Product process

Although it was never called this, the concept of a Minimum Viable Product seemed embedded in the process from the start, and it took a very different route to other governmental ones I’ve been involved with. Contributions were encouraged to made on a wiki, conference calls were held weekly with a strict one hour limit, and we were given a target deadline of the end of January (we started in November). No long meetings. No backroom deals. Admittedly by technology startup standards this may be slow, but for government bureaucracy standards this was definitely agile, and the tight timings really focused people’s minds on results.

So congratulations Phil & co and let’s hope we not just get some useful feedback on the vocabulary, but that other governmental  organisations can learn from the process too.

We’re going to split this post into two parts. The first is for those who have no knowledge of javascript, and just want to add the widget; the second for those who want to understand how it works.

The widget is simple both to install and use, and allows visitors to your site or blog to search for companies by name without leaving the site. There’s a demo page at http://api.opencorporates.com/tools/widget_example.html so you can try it out:

When you search, it simply adds the results below the search box:

1. Installing the widget

If ever added a widget to your site or blog, this should be very familiar. All you need to do is to paste these two lines inside the HTML wherever you want it to appear:

<script src="http://api.opencorporates.com/tools/widget.js" type="text/javascript"></script>
<div class="ocwidget-container"></div>


…and that’s it! If you want to change the styling, you can – just have a look at the CSS that’s pulled in by the javascript, and override it in your own CSS file.

2. Understanding how the widget works

The widget has two parts. The second part is an empty <div> element that will hold the widget content; the first imports a javascript file from http://api.opencorporates.com/tools/widget.js and runs the javascript within it. This file has been ‘minified’ to remove all the redundant spaces and comments, so it’s easier instead to look at the commented, nicely spaced version at https://github.com/openc/OpenCorporates-tools/blob/master/widget.js .

The first part is semi-boilerplate, which checks whether a useful javascript code library called jQuery is available and if not loads it (if you were writing your own widget just for your website and already include this you could omit most of this). When it’s finished loading it, and the document itself has finished loading, it runs the setupDocument() function:

function setupDocument () {
var form_stuff = '';
  form_stuff += '<div class="ocwidget-header">
<h3><a href="http://opencorporates.com">OpenCorporates company search</a></h3>
<div class="scope">30 million companies, 35 jurisdictions</div>
</div>\n<form id="ocwidget-search" class="search" action="http://opencorporates.com/companies" method="get" accept-charset="UTF-8">
<div style="margin: 0; padding: 0; display: inline;"><input type="hidden" name="utf8" value="✓" /></div>\n<input id="openc_api_search_q" type="text" name="q" />\n<input class="button" type="submit" name="commit" value="Search" />\n</form>
<div class="loading" style="display: none;">Searching...</div>
<div id="ocwidget-result" style="display: none;"></div>
<div class="ocwidget-footer">Powered by <a href="http://opencorporates.com">OpenCorporates :: The Open Database Of The Corporate World</a></div>';
 jQuery('.ocwidget-container').html(form_stuff)
 jQuery("#ocwidget-search").submit(function(event) {
 event.preventDefault();
 if (jQuery("#ocwidget-result").is(":visible")) {jQuery("#ocwidget-result").slideUp("slow");}
 jQuery(".loading").show();
 var $form = jQuery( this ),
 term = $form.find( 'input[name="q"]' ).val(),
 url = $form.attr( 'action' ).replace('http://','http://api.') + '/search?callback=?';
 jQuery.getJSON( url, { q: term },
 function( data ) { insertCompanyData(data, term); }
 );
 });
 }

This function builds the basic HTML for the widget (first the CSS style and then the HTML itself), and puts it in the div you’ve already got on the page.

Next it uses jQuery to add a listener to the form you’ve just put on the page, looking out for when the submit action happens, and running the function attached when this happens. And that’s it for the setup. The rest is dealing with what happens when a form is submitted.

When this does happen, the first thing is that the normal behaviour of the submit button – to make a fresh HTTP request and treat the response as a new page – is disabled. Then if the results area is visible (i.e. because a search has already been made), it’s slid out of the way. Next a ‘Searching…’  message is shown to let the user know something’s happening.

After that the information on the form is obtained and submitted  via the jQuery.getJSON call. This makes an Ajax request to the url, and does something with the result, in this case runs the function insertCompanyData(data, term). A couple of things worth noticing here:

• The URL in the form isn’t actually for the OpenCorporates API, but for the main OpenCorporates websites. There’s no reason why you couldn’t use the API address in the form, but as this is, in part, a demonstration of how to use the API, and some people might want to use the javascript not as a widget but as a javascript-enabled HTML form on their website, it was worth showing a some nice graceful degradation. By using the OpenCorporates URL on the form, if javascript isn’t enabled  (e.g. visitors using screenreaders) the search will still work.
• We’ve added ‘?callback=’ to the URL. This tells jQuery we want to make a JSONP request, and that it should add a callback parameter, and handle the JSONP response accordingly. This gets over the ‘same-origin policy‘, which prevents scripts handling data from a different domain than you’re already on. It’s the fix that allows all these javascript widgets to work, but does require the API to understand and handle callback requests. Fortunately the OpenCorporates API does

Having made the request, the browser gets on with whatever it was doing, until the response comes back, and the the data (together with the term we searched for) is passed to  insertCompanyData():

function insertCompanyData (data, term) {
var companies = data.companies;
content = "<div class='summary'>Found <a href='http://opencorporates.com/companies?q=" + term + "'>" + data.total_count + " results</a></div>";
content += listAll(companies);
jQuery(".ocwidget-container .loading").hide();
jQuery( "#ocwidget-result" ).html(content).slideDown("slow");
}

This does what it says on the can, building up the results content with some useful information (the total number of responses, with a link to all the results on OpenCorporates). It iterates through each of the companies returned (via the listAll function), and then uses listItemFor to build up a <li> list element for each, consisting of the company name and link to the company page on OpenCorporates, the incorporation date if known, the curent status, and the jurisdiction code, which is made a little more readable by converting ‘gb’ to ‘GB’, and us_dc to US (DC). A CSS class is also added if the company is inactive, to allow us to show those in a different colour.

function listItemFor(company) {
  var jurisdiction = company.jurisdiction_code.replace(/_(\w+)/," ($1)").toUpperCase();
  var li = "<li class='" + (company.inactive ? 'inactive' : '') + "'>";
  li += " <a class='company' href='" + company.opencorporates_url + "'>" + company.name + "</a>, " + jurisdiction;
  li += company.incorporation_date ? ', ' + company.incorporation_date : "";
  li += company.current_status ? ', ' + company.current_status : "";
  li += "</li>";
  return li;
}

function listAll(coll) {
  if(coll.length && coll.length > 0){
    var listResult = "<ul>";
    for (var i = 0; i < coll.length && i < 8; i++) {
      var li = coll[i].company;
      listResult += listItemFor(li);
    }
    listResult += "</ul>";
    return listResult;
  }
  else { return '';}
}

And that’s it. It’s on GitHub, so feel free to fork it, or use the code for your own purposes, but please remember that OpenCorporates is under the Open Database Licence and so if you’re using the API, you must link back to the OpenCorporates as per the links requirements.

Who would have thought that 12 months later we would have become 10 times bigger, with over 30 million companies and over 35 jurisdictions, and lots of other data too. So we could use this as an example to talk about some of the many milestones in that period, about all the extra data we’ve added, about our commitment to open data, and the principles behind it.

We’re not going to do that however, instead we’d rather talk about the new API we’ve just launched, allowing full access to all the info, and importantly allowing searches via the API too. In fact, we’ve now got a full website devoted to the api, http://api.opencorporates.com, and on it you’ll find all the documentation, example API calls, versioning information, error messages, etc.

Here are the brief highlights:

• A versioned API, which means that you don’t have to immediately change your code when the API changes, as you can specify a version number to work against.
• Lightweight JSON responses (XML will be coming soon), perfect for widgets, smartphones and modern web applications.
• No registration is required, but an optional free API token allows higher usage limits.
• No additional licence restrictions – same open data terms as the rest of OpenCorporates, even for heavy users.
• All search parameters supported, allowing complex filtering by company type, status, data held, etc. Search data is paginated, with ability to increase results size. Search data can be sorted alphabetically or by match score.
• Full provenance with data, including source url and when it was retrieved.
• Finally, it’s worth mentioning that the API service will be shortly using different servers to the main website, ensuring that heavy usage of one won’t affect the service of the other

The API is not just for heavyweight users of corporate data. The JSON format is supported by a huge number of non-techie applications too, including visualisation tools, widget makers, and increasingly Google applications too, and ver the next month or so, we’ll give some examples of how to use it… unless you beat us too it.

Merry Christmas

The OpenCorporates Team

Company registers were some of the earliest forms of public registers, and the need for them to be open was recognized from the time they were set up. Today, companies play a much larger role in society, and in a data-driven world, it’s important that all of society has access to information about them, in a form and licence that allows for free reuse.

But it’s one thing to talk about open data; another thing to have it as your core principles, and that’s why we’ve decided to set out the first draft of the OpenCorporates principles – a codification, if you like, of what we do, and how we do it.

In part, this was down to a recognition that OpenCorporates has become an important part of the open data ecosystem, and is increasingly relied upon by journalists, the wider community, even governments. In that context, it’s important that it’s clear how we work, and that there are also clear benchmarks by which we can be judged.

It’s also a recognition that as with all things in the area of online information, there are conflicts between different rights and obligations, and it’s not always clear which is the right path to choose, and we hope the principles will help guide us when balancing those conflicting demands.

The OpenCorporates principles

1. Purpose: OpenCorporates exists to make information about companies and the corporate world more accessible, more discoverable, and more usable, and thus give citizens, community groups, journalists, other companies, and society as a whole the ability to understand, monitor and regulate them.
   To the extent that we have rights to the information in OpenCorporates (including database rights), we will licence it under a share-alike attribution licence, allowing free and open reuse even commercially. Organisations or companies that wish to use the information on a non-share-alike basis will need to pay for a non-share-alike version (for the privilege of not releasing the resultant information to the community), thus ensuring OpenCorporates has a sustainable business model and giving an incentive to release information back to the community.
2. Scope: The definition of a company varies from jurisdiction to jurisdiction, includes a variety of different legal forms used for a variety of purposes.
   OpenCorporates follows the basic principle that a company is a legal entity registered or incorporated by a company register or jurisdiction. We do not intend to create records on individuals, except insofar as they are registered as a business or connected with a business or other corporate entity on a public document or register. We do include dissolved companies as this is an important part of the public record, allowing customers, suppliers, and society at large to understand the past actions of companies and of those involved in them.
3. Accuracy and Transparency: OpenCorporates aims to be an accurate reflection of the official public record, especially company registers, and we will always be clear about from where and when we obtained the information. Should there be errors in the importing of data from a company register or other public record, in the automatic matching of information to companies, or in user-contributed information, we will aim to correct those errors in a timely manner once we have been notified of them.
4. Privacy. The owners of companies and the directors of them are often (though not always) individuals. Most countries make this information part of the official public record, to avoid fraud, corruption, and to ensure that companies aren’t able to avoid their responsibilities to the wider society.
   There is also in virtually every country a requirement to provide an official address, not least to ensure that legal redress can be obtained. Where directors or companies choose to submit a residential address there is arguably a conflict between the right to know, the right to free speech and the right to privacy.
   Where the balance is drawn between these potential conflicts is not always clear and varies from country to country. OpenCorporates shouldn’t seek to be an arbiter in this regard and we will defer to the official company register on what information is published about the individuals connection with companies, especially the owners and directors.
   Should a company register or other public data source remove or redact information about a person connected with the company, because, for example, their personal safety is at risk, we will update the OpenCorporates record in a timely manner once we’ve been informed, and will institute a procedure for temporarily removing the information while this is being carried out.

We’d welcome feedback on these principles. They are a work in progress, and we’re reviewing our internal processes to ensure we follow them.

Well, now at last you can do that, and not just company owners, but anyone who’s logged in. The requirement to log in is pretty important, as we’re almost obsessive about provenance on OpenCorporates. Put simply, anybody using OpenCorporates should be able to know where (and when) that information originated, whether from a company register, a data source such as the UK’s public spending data, or from an OpenCorporates user.

How do you do it? Just log in using the subtle ‘login’ link at the top right of the page. As well as allowing logging in using your Twitter account, you can also now log in using LinkedIn. The procedure’s the same – just click on the link, log in to your LinkedIn account, confirm that you’re happy to use LinkedIn to authenticate your OpenCorporates account, and you’re done. The whole thing should take no more than 30 seconds.

Then on the company page, just click on ‘Add Data About This Company’:

Add the data – in this case, we’re adding data about Trafigura BV, a Dutch company that’s part of the Trafigura group:

And that’s it. And of course, the information you’ve added is available to the community as data in a variety of ways under the same open licence as the rest of OpenCorporates.

Last week, the UK Companies House took the wraps off its URIs for companies. Now this is actually rather more significant than you might think (though frankly we are going to struggle to make it sexy).

First, it’s significant from the UK open data perspective, as in part it was the lack of a persistent URL which led to Companies Open House, an early precursor to OpenCorporates. Like quite a few company registers around the world, Companies House has URLs based on user sessions, which means that they change each time you visit them. The lack of permanent URLs for companies is a real problem when trying to reference companies, as anyone will know who’s ever been sent an email containing a Companies House URL only to find it takes you to the search page, not the page for the company concerned.

Second, it is significant because – with a little bit of arm-twisting by the Cabinet Office – Companies House has created these with the input of the open data community in the shape of OpenCorporates. Now URIs for companies is something we’ve got a bit of an opinion on, publishing over 27 million of them in over 30 jurisdictions, including 7 million for UK companies, and while much about the process was cumbersome and frustrating, what was gratifying was that Companies House listened to our input and changed their URI structure (for help with understanding the difference between URL and URIs see this Wikipedia Article and this StackOverflow answer).

Now the information you can get from those URIs is pretty bare to be honest, but that’s not really the point here. The point is that the government has now produced URIs which not only permanently identify the companies, but follow the same principles to those used by the Open Data community, being based only on the company number.

So, the Companies House URI for Vodafone PLC is http://business.data.gov.uk/id/company/01833679 and the OpenCorporates one is http://opencorporates.com/id/companies/gb/01833679. Both of these redirect to a HTML web page with a similar URL by default, or to pages that return JSON, XML or RDF data if that is requested. You can see how easy it is to switch between the two, and in fact every UK company page on OpenCorporates now has a link to the Companies House page.

Perhaps some of the more astute business people out there are surely asking, “Why would they do that? Why would OpenCorporates help Companies House come up with a URI system that mirrors their own and so, arguably, helps make it less essential, and with it reduce a chance of being the monopoly supplier.”

Well, it’s not naïvity; nor is it just a determination to do the Right Thing, although that’s very important to us, and in fact the choice of URI structure in OpenCorporates is designed to prevent anyone, including OpenCorporates, being able to have a monopoly over company identifiers. It’s also a realisation that unless we pull governments kicking and screaming into the open data world, we’ll end up with a world with little power for citizens and little opportunity for those who want to build a business based on open data and doing the right thing have a chance.

This is especially true for corporate data, and in a world where company registers publishing all their data in a free and open way is the exception rather than the rule (shamefully, none in the EU do so), this is an important step, and one we were happy to support.

Now the question is: will this just be another isolated island of URIs, or will the government actually become part of the open data community, by linking out to OpenCorporates and other open data projects? And will they be using OpenCorporates reconciliation service to match their own data, suppliers, etc to real-world legal entity or continue with using proprietary and closed identifiers such as Dun & Bradstreet’s poisonous DUNS numbering system? Answers in comments

One of the key aspects of OpenCorporates has always been that word ‘open’. Open here means something very specific. It means freedom (as in free speech), not just free beer.

The open source movement has travelled this road before, not without some struggle, and so one of the key moments in the early days of the open data community was to quickly agree on what it meant by ‘open’ – the result was the Open Definition:

A piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the requirement to attribute and share-alike.

However, when open data becomes a sexy meme, seized upon by governments that don’t even have a Freedom of Information law, still less actively publishing open data, and companies whose idea of open data is ‘all rights reserved’, it’s important that we emphasise why this is so critical, and why it sets OpenCorporates apart.

In a world of Big Data, when power comes from the ability to combine data together, unless you have power to use, reuse and redistribute, you are on the powerless side – whether you a citizen, a small company, an NGO, or a government department.

And openness is about another thing too – provenance, or where you’ve got the data from and when (it’s why Wikipedia keeps a record of who edited what). With a complex and important dataset such as companies, which requires a degree of crowd-sourcing, it’s even more important that you show the provenance, and make that available as open data too.

These then have been the guiding principles behind OpenCorporates from the beginning, and why openness as in free speech (although we like free beer too) is so important to us, and why we’ve concentrated on this and on working with the community to add more countries, rather than sexy visualisations.

But despite this, it matters how easy it is to reuse the data. Our Google Refine reconciliation service was (and remains) groundbreaking, but to be honest, there was a fair bit of scope for improvement in other areas. And so now over the past few weeks, we’ve been working on making it easier to get the information out.

The first step, is exposing all of what you see on the web page as data, and while we have always done this for companies, we’re now rolling it out sitewide. What does this mean in real terms? If means that if you want the latest filings for a company as data you can have them; if you want to know (as data) the source of the official journal notice about a company you’ve got it. You can see information on the bottom right of each page, together with the ways we make it available (also as RSS feeds and RDF in places).

How can you reuse this? Well, OpenlyLocal, for example, has some 80,000 companies who are suppliers to local authorities, and now they use OpenCorporates to pull in and show not just the basic information, but also the previous names and the latest filings too. And because the data is available as JSON (and JSONP) as well as XML, it took just a few lines of javascript to add that to all of OpenlyLocal company pages.

We’re now working on ways of making the faceted search function available as data, and would welcome feedback on that or any other aspects of the API. And of course, with every API call, and every contribution you make to OpenCorporates, every tweet about a company on OpenCorporates, you can be sure you’re contributing to a database that’s not only free as in free beer but free as in free speech too.

Oh, and that milestone? Less than two months after hitting 20 million companies, we’ve broken the 25 million mark, and are fast heading for 30 million.

Well, that didn’t take too long. It was barely 3 months ago that OpenCorporates hit 10 million companies, and in what seems like no time at all we’ve doubled that.

In that time, we’ve:

• Added important countries such as Spain and, today, Ireland (we’re still importing the Irish data, and reckon there’s probably something under 100,000 still to go)
• Added a way of grouping companies together, linked to the Wikipedia (and DBpedia) entries, and thus made the job of mapping out multinationals feasible for the first time
• Started collecting information on statutory filings (we now have over 40 million)
• Improved our Google Refine reconciliation service for matching company names to the likely legal entities.
• Shown concrete examples of how OpenCorporates can be used cleaning up broken government data
• Won an award at the European Open Data Challenge

Not made for a self-funded micro-startup (and, of course, the open data community, without which we couldn’t have done it).

What should we do next? You tell us. Work on the API, to make it easier to get information out of it? Concentrate on adding more jurisdictions? Add more data? Allow users to add data and connections between companies?

In the meantime, please help us by tweeting about OpenCorporates, +1′ing us, liking us on Facebook, and above all linking to us. With your help, we can make sure the biggest and best database of company information in the world is an openly licensed one.

The green paper on EU Corporate Governance is helpful as far as it goes, but it skips over or ignores two key issues.

The first is the position of stakeholders other than shareholders in listed (and other) companies). This is mentioned in the preface, but there are no proposals for how their position may be improved.

The second is the problem that much information about listed (and other) companies is available only either for a fee, or with restrictions on reuse, meaning it is impossible to combine the data with other datasets, and so get a rounded view of companies, restricting the ability of to understand and monitor companies.

On the first point, the preface says, “Corporate governance is traditionally defined as the system by which companies are directed and controlled and as a set of relationships between a company’s management, its board, its shareholders and its other stakeholders” . Later on the paper states that “some companies may face risks that significantly affect society as a whole: risks related to climate change, to the environment (e.g. the numerous dramatic oil spills witnessed in recent decades), health, safety,  human rights, etc.

However, despite this, there are no suggestions for how to improve the position of the other stakeholders, particularly with respect to the core problem of being able to monitor and understand such companies. Whether you live in the shadow of an oil refinery, work for a company that may be in financial difficulties, or a small business trading with a company that is ultimately owned in another jurisdiction, your ability to understand and monitor the corporate body that affects your life is severely restricted, requiring you to subscribe to multiple expensive services, visit websites on a daily basis, and be able to pull together strands of information from multiple sources.

Compare this with institutional investors, and short-term professional share traders, who have the time and money to access information, and it is clear that there is a serious and damaging information asymmetry, and also one that favours those with a short-term perspective. Finally, some of the proposals in the green paper, for example disclosure of remuneration policy or risk appetite (Q 10 & 11), while in theory admirable, may actually increase these asymmetries, as they would in practice be only easily accessible and comparable for those with easy access to the data, with the result that there may be a greater short-term focus and reduced social responsibility.

On the second point, as Commissioner Neelie Kroes recently stated. “Data and information fuel innovation. Getting out the data should be seen as an investment.” She was specifically talking about ‘open data’ – public data that is licensed for reuse to allow it to be combined with other data without restrictions.

This is particularly relevant to information about companies, which have increasingly complex structures and multiple interactions with the world, and in multiple jurisdictions too. In addition, most large companies, especially listed companies, are often just one part of a group of companies, or have multiple subsidiaries, and occasionally multiple parents too.

This makes the job of good corporate governance almost impossible, and many of the financial crises and failures of corporate governance of recent times were down to the opacity this complexity brings (sometimes the opacity is part of the appeal of a complex corporate structure). An additional problem is that the data behind this complexity is ‘owned’ by several different bodies, in multiple jurisdictions, and with the exception of the US Federal government and increasingly the UK government, none of them has a policy that encourages it to be brought together.

If there is to be good corporate governance it is essential that this complexity can be made sense of, particularly by combining multiple datasets together, to understand, for example, the corporate structure of Europe’s biggest companies, to see whether the work they are doing in one jurisdiction contradicts the claims made in another, and to bring together multiple sets of accounts to understand how solvent a company is.

Ironically, most of the data is often collected by public bodies (company registries, regulators), or as part of a statutory framework (e.g. filings of listed companies), yet almost without exception it is not made available in a form or with a licence that allows for reuse, still less encourages it. Sometimes even the basic information about a company — its legal name, registration number, and status — is hidden behind paywalls, depute it being collected and published by public bodies for the public benefit.

In addition, company accounts, and information about the chief officers and controlling shareholders are almost never published as data, either by the companies or the public bodies that oversee them. Although the UK is moving towards XBRL to represent company accounts, this is the exception rather than the rule, meaning that access to this information is highly problematic, either not being collected as data (for even quite large companies), or being done in a labour-intensive way, increasing the cost of collection to a level that it is priced out of the reach of all stakeholders except professional and institutional investors.

Finally, one of the core tenets of an open and free market is equal access to information. As far as corporate information goes, this exists neither for shareholders, other stakeholders, nor for competitors, leading to multiple failures in corporate governance at all levels.