How should we assign weights to various aspects in the OECD's 'better life' index?

The second edition of the OECD’s ‘Better Life’ index is now available. As with the first edition, the index is presented in an interactive form.  Users are able to compare the quality of life in different OECD countries by assigning their own weights to a range of factors that are relevant to assessment of well-being.

When the first edition of the index came out I played around with it to see how different weighting systems and inclusion of additional information on factors, such as perceptions of freedom and corruption, might affect the comparisons. In order to make much difference I had to make some fairly extreme assumptions. My conclusion was that all well-being indicators tend to tell similar stories at a national level.

In this post I want to give some further consideration to the question of what weighting systems might be most appropriate. I want to allow for the possibility that because some of the factors included in the index measure similar aspects of well-being, there may be potential for users to inadvertently give excessive weight to the factors that seem important to them. For example, some people who assign a high weight to life satisfaction might also give high weight to factors that could be expected to have a strong influence on life satisfaction (e.g. jobs and work-life balance). That might mean that they end up giving excessive weight to factors that are strongly correlated with life satisfaction, at the expense of other factors (e.g. education and the environment).

The approach I have adopted in deciding on the weightings was to begin by giving highest weight (5/5) to safety (low crime), life satisfaction and the environment and then to assign weights to other factors depending on whether I consider their influence to be adequately reflected in the factors already covered. I have assigned a weight of 4/5 to income because I think high income adds to quality of life in ways that are not fully taken into account when people are asked whether they are satisfied with life (the value of the future security that a high income can provide). I have assigned a 4/5 weight to education because the contribution of education to quality of life may not be adequately reflected in either life satisfaction or income. Similar reasoning applies to community (covering support networks and volunteering). A weight of 3/5 is assigned to housing, jobs and health because these factors are already taken into account to a considerable extent in measures of life satisfaction and income. I have assigned a relatively low weighting (2/5) to civic engagement because important aspects of civic engagement are reflected in other factors (e.g. safety) and voter turnout seems to me to be a poor indicator of civic engagement in countries with compulsory voting. Finally, I have assigned a rating of 1/5 to work-life balance (rather than 0/5) because it might possibly cover some aspects of the quality of life that are not adequately reflected in other factors.

The ranking of countries according to the weights I have assigned can be found by clicking here. (The OECD offers a facility to embed it, but I am not clever enough to use it.)

The ranking according to my weights differs somewhat from the ranking if equal weights are assigned to all factors. The top five countries under my weights are: Australia, Switzerland, United States, Canada and Norway. With equal weighting, the top five countries are: Australia, Norway, United States, Sweden and Denmark.

However, this difference in rankings doesn’t mean a great deal because the difference in ratings of the top ranked countries is small. That doesn’t surprise me - all well-being indicators tend to tell similar stories at a national level! 

Are citizens' juries susceptible to groupthink?

In my last post I considered the question of whether citizens’ juries might help to lift public debate on issues which professional politicians find difficult to deal with. After looking at the process involved in a citizens’ jury on container deposit legislation, I concluded that it was likely to be particularly important for citizens’ juries to be asked the right questions.

The post provoked two different responses. Evan suggested that citizens’ juries could make a valuable input to policy, but there was a problem of legitimization i.e. the process required government backing. Anon (kvd) asked why it was necessary to place between ‘the people’ and their elected representatives a layer of ‘non-responsible, unelected, single-issue, unaccountable’ citizens’ juries. My response was that I agreed with both Evan and kvd, but what I meant was that both had raised good points. Citizens’ juries would certainly need to be seen to have legitimacy in order to have the desired effect on public debate. And the onus should certainly be on those who propose such innovations to show why they are necessary and are likely to succeed.

I think we should be looking for innovations to make the democratic systems work better because there is a tendency for democratic governments to over-reach – to raise public expectations of what they can achieve and to take on more responsibilities than they are able to cope with. This is reflected to some extent in the tendency for many governments to take on excessive debts, but I see the problem as much broader than that. I will write more about that at a later stage.

 I have had an interesting discussion with Shona (the person who suggested last year that it might be a good idea to think up a new political system) about the possibility that citizens’ juries might be susceptible to groupthink. The discussion arose because of the example in my last post of a citizens’ jury coming to a unanimous decision, while successive polls looking at the same issue found that support for the proposal diminished after provision of information to participants. Shona suggested that I should take a look at the relevant chapters of Stuart Sutherland’s book, ‘Irrationality’.

Sutherland makes the point that not only do people tend to conform to majority decisions but group attitudes tend also to be more extreme than those of individuals. Rather than the attitudes of each member of the group tending toward a mid-position, as members of the group interact they tend toward a more extreme position. One possible reason is that if the group attitude is tending in one direction, individual members may express more extreme views in order to gain approval from other members. Another possible reason is that the group is willing to take greater risks than its individual members because membership of the group spreads responsibility.

Research relating to legal juries suggests that conformity to majority views is common. For example, a US study by Nicole Waters and Valerie Hans indicates that while about 6 percent of criminal juries fail to reach unanimity over one-third of  jurors say that if the decision had been entirely up to them they would have voted against the jury’s decision.

The potential for members of citizens’ juries to conform to majority decisions and for group decisions to become polarised has been recognized by some researchers in this field. For example, R K Blamey, P McCarthy and R Smith of the ANU discussed these issues in their paper ‘Citizens’Juries and Small-group Decision-making’. The authors conclude that the fact that citizens’ juries are susceptible to biases does not necessarily mean that individual-based decision-making (e.g. via opinion surveys) should be preferred. They make various recommendations designed to minimize the weaknesses of group processes and to maximize their strengths.

There are some situations where the tendency of small groups to conform to majority decisions is desirable. For example, a fairly high degree of conformity to group decisions is required for a system of cabinet government to work effectively. It seems to me, however, that the tendencies for bias discussed above would pose substantial problems in using citizens’ juries in the manner I had contemplated. There is a good chance that the outcome of a citizens’ jury process would not be representatives of the views that a group of randomly chosen group of citizens would come to if they had the opportunity to consider an issue independently. 

What questions should citizen's juries be asked?

Since writing on the question of whether citizen’s juries might be better than focus groups I have been thinking more about the kinds of questions a randomly selected group of citizens could usefully deliberate upon. The answer probably depends on the objective of the process.

When Gordon Brown became prime minister of Britain in 2007 he announced citizen’s juries as his ‘new big idea’. It seems from his speech announcing the policy initiative that his aim was to improve communication between his government and the general public. His emphasis was on new ways of listening to people, consulting on new ideas and engaging in a dialogue and deliberation. He attended some of these citizen’s juries himself. An internet search indicates that the response at the time ranged from strong support to cynical suggestions that his motive might have to obtain more photo opportunities. Remarkably, within about a year the big new idea seems to have vanished without leaving a trace on the internet. I don’t know what happened. It is odd that there doesn’t seem to have been a public post mortem about what went wrong (or right) with the process. My guess is that the novelty of listening and consulting might have worn off.

It seems to me that the most useful objective of citizen’s juries would be to influence public opinion by providing a considered point of view that people might consider more trustworthy than the views of political leaders, interest groups or experts making judgements outside their areas of competence.  In other words, this process might help to address the combination of rational ignorance and hubris (or narcissism or whatever) that tends to lower the level of public debate on policy issues.

So, what types of questions might usefully be put to citizen’s juries? In my view the relevant questions would arise mainly from the issues that professional politicians find difficult to deal with because public opinion is ill-informed or unstable. Establishing citizen’s juries to look at such issues could be viewed as experiments to help a government to decide whether to take the political risks involved in introducing reforms.

I am not sure to what extent citizen’s juries have previously been viewed in that light. From the little I have read on the topic, the inclusion of unorganized citizens in policy processes is often seen as a desirable end in itself.

Greg Cutbush, a visiting economist at ANU Enterprise, responded to my last post by suggesting that before I get too enthusiastic about citizen’s juries I should take a look at a citizen’s forum on a proposal for a container deposit scheme in New South Wales.

In brief, the citizen’s forum was part of a review of container deposit legislation conducted by Stuart White of the Institute for Sustainable Futures at UTS in 2001. Other parts of the review included a cost benefit study of container deposit legislation (CDL), stakeholder and community consultations and a ‘televote’.  The ‘televote’ involved two polls, the second of which followed provision to respondents of ‘balanced written information on the various arguments in favour and against CDL’. The eleven members of the citizen’s forum were provided with essentially the same written information, presentations from government officials and independent CDL experts, as well as exposure to three days of deliberation.

As a result of the provision of information in the televote process, support for CDL fell from 71 percent to 59 percent. By contrast, the citizen’s forum unanimously agreed to the implementation of CDL. I wonder whether the emergence of consensus reflects the impact of three days of deliberation.

In the aftermath of the exercise one member of the project team, Carolyn Hendriks, commented:

‘The introduction of new players to the policy debate, particularly in terms of their representativeness and legitimacy, was one of the key concerns raised by the interest groups in a recent Australian citizens jury held on Container Deposit Legislation (CDL). There were two key issues here. Firstly, interest groups questioned the capacity of ordinary citizens to comprehend their arguments. Secondly, most of the CDL interest groups saw themselves as the only legitimate stakeholders. According to this viewpoint, citizens with an opinion or interest in the issue can only enter the policy debate via a valid group. Extending public participation to virtual stakeholders such as lay citizens appeared to insult interest groups because it down-played their expertise and their long-term investment in the issue.’

The problem she alludes to was the withdrawal of some stakeholders from making presentations to the citizen’s forum, which necessitated reliance on written material. It is not clear whether this had any impact on the outcome of the deliberation process.

However, I have doubts about the value of the whole exercise because it isn’t clear that participants in the forum were asked the most appropriate question. As pointed out by Access Economics, in its review of the White report, the question of how best can governments increase recycling rates requires a ranking of CDL against other feasible options rather than an a consideration of whether a CDL scheme might be better than the status quo.

If citizen’s juries are asked the wrong questions they might reach unanimity about matters that are not particularly relevant to the policy choices that governments have to make. 

Would a citizen's jury produce a better policy outcome than a focus group?

In an article published on his blog last Sunday Jim Belshaw argued that risk management by governments has come to focus too heavily on political risk avoidance because of concerns that a more balanced approach would be too difficult to sell to the public. As a generalization, I think the point is correct. It is possible to cite a few examples of recent Australian governments taking excessive risks (e.g. the infamous home insulation and school hall construction programs). In my view, however, the Hawke, Keating and Howard governments showed stronger leadership than those that have followed and were less prone to allow focus groups, polling and brainstorming on talk back radio and twitter to set the political agenda.

That is just a personal impression. It is difficult to cite hard evidence. Mike Steketee provided some examples in support of the view that poll-driven policies have become more common in an article in ‘The Australian’last year. In my view the strongest point he made, however, was to quote Rod Cameron, a veteran pollster, who has observed that politicians are now more inclined to accept the prejudice and narrow-minded bigotry coming out of focus groups as a basis for policy, rather than to seek to neutralize such views.

Irrespective of whether Australian governments have become more poll-driven in recent years, the temptation for governments to opt for politically safe options rather than those requiring courageous political leadership exists in all countries in which public opinion counts for anything. In many instances, of course, the politically safer option is much more risky in the longer term. In my view that makes it preferable to have a system of government in which it is clear which political parties are accountable for the decisions that are made or not made by governments, rather than proportional representation, in which responsibility is shared by the centre right and centre left - and the only parties that have clean hands are the extremists at either end of the political spectrum. I have in mind the situation in Greece, of course, but I am straying from the topic I want to write about.

Coming back to the importance of political leadership, I remember a conversation that I had with a wise person about 20 years ago. I made the point that Australia needed more courageous political leadership to pursue economic reforms. I expected the wise person to agree, but his response was that those who want governments to pursue economic reforms need to accept that governments don’t lead, they follow. The point he was making was that it is important to keep in mind that political leaders can never get far ahead of public opinion. The leader who prepares the ground for reform by attempting to raise the level of public discussion of an issue will often be more successful in promoting reform than the one who shows great courage in attempting to forge ahead ignoring ill-informed public opposition.

So, how should those who aspire to leadership seek to raise the level of public discussion of issues? One method that has had some success in Australia is the system of independent policy advice provided by the Productivity Commission and some of its predecessor organizations. The strength of that system - as Gary Banks, the chairman of the Commission, pointed out in a speech last year - has been the independence of the Commission, the process of public scrutiny of underlying research and analysis before the advice is submitted to government, and the educative role of the organization in helping to promote a broader understanding of issues and advocating initiatives to the public and parliament.

However, it has not been possible for the Productivity Commission to be as effective as it should be. In an interview with Alan Mitchell in the Australian Financial Review a couple of months ago (10 March) Bill Carmichael, member of the Tasman Transparency Group and former chairman of the Industries Assistance Commission, suggested that the Rudd and Gillard governments had sidelined the Productivity Commission as an independent advisor on microeconomic reform:

‘They have created a plethora of carefully selected inquiries and institutional arrangements designed to minimize bothersome critical analysis and produce outcomes more to their liking’.

The report of the interview ended with Bill Carmichael suggesting that political leadership is ‘a quality that has been missing from the present debate about economic reform’. An element of leadership is clearly required to move forward on difficult issues, even with the help of sensible public inquiry processes.

So, could citizen’s juries help to compensate for weak political leadership? According to Nick Gruen, in the transcript of an ABCbroadcast in which he spoke about ‘deepening democracy in the internet age’, a citizen’s jury or consensus conference is ‘a small jury-sized randomly selected group’ which ‘deliberates at length’ on policy issues. The body hears evidence from professional experts and advocates, and its conclusions are published.

It seems to me that the potential benefit of such a system would lie mainly in helping to lift the level of public debate on contentious issues, by providing members of the public with a point of view that they might consider more trustworthy than the partisan views of political leaders, or judgements of experts who might seem to be out of touch with the values of ordinary people. Citizen’s juries would certainly not be a substitute for sensible public inquiry processes, but they might help avoid policy development being placed at the mercy of focus groups and political point scoring. Citizen’s juries could perhaps be particularly helpful in development of policies that involve important value judgements e.g. deciding appropriate levels of immigration.

I will write more about citizen’s juries later.