Assessing Global Risk: Comparing Apples and Orangutans

From: http://www.weforum.org/

From: http://www.weforum.org/

Last week, the World Economic Forum (WEF) released its 10th Annual Global Risk Report for 2015 ahead of the annual meeting in Davos, Switzerland. This very ambitious report seeks to identify the biggest risks to the world, their causes, and potential solutions. The report is a flagship effort of the WEF and is used as a tool to help decision makers prioritize resources and implement efforts to prevent or mitigate economic impact of global events. Stakeholders of the report have also identified its importance in building scenarios and exercises, informing vulnerability assessments, and in galvanizing more detailed risk modeling efforts.

The major value of this report is its role as an awareness raising tool. The report highlights the interconnectedness of major global events that–while they may occur in one part of the world or in one sector–inevitably affect economies and governments throughout both the developing and developed world.

Methodology

The data collection and resulting report are organized into 5 categories: economic, environmental, societal, and geopolitical risk. Within these 5 categories, each year the WEF chooses 28 risks for experts to rank in terms of likelihood and impact (consequence). The Global Risk Report defines global risk as: “an uncertain event or condition that, if it occurs, can cause significant negative impact of several countries or industries within the next 10 years.”

The 5 categories and most of the 28 risks identified remain the same each year, although risks may be substituted in and out by the authors. Examples of risks in the report include interstate conflict, extreme weather events, water crises, cyber-attacks, energy price shock, etc. These identified risks are each accompanied by a brief description. Additionally, this year’s report for the first time identifies 13 trends that may be driving global risks. A trend is defined by WEF as “a long term pattern that is currently taking place and that could amplify global risks and/or alter the relationship between them.” Examples of trends are: ageing population, climate change, urbanization, etc.

To collect data for the report, the WEF Staff constructs and distributes a Global Risks Perception Survey. The survey is sent to an undefined denominator of WEF “global multistakeholder community members” including  leaders from business, government, academia, NGO’s, and international organizations. It’s worth noting that economic subject matter expertise appears to be predominant in their sample.  For this year’s report, 896 members of the WEF community completed the survey.

The survey asks community participants to do the following:

  • For each of the 28 risks, rank the likelihood of occurrence in the next 10 years from 1-7 (1 being low and 7 being high).
  • For each of the 28 risks, rank the impact from 1-7 (1 being low and 7 being high).
  • Identify 3-6 pairs of risks that are “strongly interconnected”
  • Identify 3-6 pairs of risks and trends that are “strongly interconnected”
  • Select the top 5 global risks that are of highest concern in the next 18 months and 10 years respectively
  • Identify the top 3 risks for which they feel the most progress has been made over the last 10 years
  • Identify the top 3 risks for which they feel the least progress has been made over the last 10 years

Findings and Comments

The findings of the report include composite rankings of each risk by impact and by likelihood, as well as graphics illustrating the interconnectedness of risks and of trends and risks.

Notably, a number of risks relating to health security are highlighted in this report, including natural and environmental catastrophes, large-scale terrorism and WMD use, and rapid and massive spread of infectious diseases. In this year’s report, 2 of the top 5 risks in terms of impact are related to health security: spread of infectious diseases and WMD risks. Inclusion of health security risks is important, as it shines a spotlight on these issues in the WEF community–an influential forum for addressing global economic challenges.

However, it is important to take results of this report in context and not place too much reliance on the report as a definitive list or ranking of global risk. The report is a snapshot of expert opinion of risk in a given year. The report is not a sophisticated measure of the relative importance of different risks.

It is also critical to note that the report is qualitative, not quantitative. While experts were asked to assign numerical values to their judgments about likelihood and impact, they did so based on their own experience and opinion. The results reported here are not based on historical data.

In addition to the points mentioned above, WEF methods give me some pause and make interpretation of the report difficult. Some issues with the methodology include:

  • Sampling bias: the WEF uses a convenience sample which includes inherent bias. Responses this year were heavily weighted toward men (74%), toward economics expertise (41%), toward businesses (41%), and toward European views (34%). Other perspectives are comparatively underrepresented in their sample.
  • Risks are predetermined: The WEF pre-selects the 28 risks for the survey based on limited expert opinion from their steering group. The survey does not allow for recommendations of new risks by survey participants. This may result in important risks being excluded from the report.
  • Recency bias: although the report is intended to examine risks over a 10 year time horizon, recent events tend to rise to the top of people’s minds, which may result in a higher ranking in the survey. For example, likely due to the Ebola outbreak this year, “rapid and massive spread of infectious disease” was ranked as the number 2 risk in terms of impact. Whereas this risk had never ranked in the top 5 in any other year since the survey began.
  • Relative risk is not adequately addressed: The survey asks participants to rank 1 risk at a time based on their opinion of likelihood and impact. The survey does not ask participants to compare the likelihood and impact between risks. Yet, the report places a good deal of emphasis on the ranking of risks, which is misleading.

Despite my concerns about the report, I laud the authors for their work; this is a very difficult undertaking and one that no other organization has dared to take on each year.

Risk assessment in its traditional form–focused on one hazard at a time–is hard enough. One never has the perfect data or knowledge to reduce uncertainty about a single risk to zero. The US EPA spends years and many millions of dollars on human health risk assessment for a single chemical or pesticide in order to be able to set regulations or rules about acceptable exposure levels, for example.

A higher level of difficulty in risk assessment is a comparative assessment among hazards in the same class of risk (e.g., comparing human health risk among a number of different chemicals)–this requires standardization of data categories and inputs and data quality that are very hard to achieve.

I would argue that the highest level of difficulty in risk analysis is a comparison of risk between very different types of hazards (apples and orangutans) with different data, different measures of probability, and different types of expected consequence. This is what the Global Risk Report attempts to do. There are problems with this type of risk analysis, but there is a policy need for it as well. I look forward to the next WEF Global Risk Report. Hopefully the authors will continue to strive for improved methodology in years to come.