Measuring Research Impact: Limitations and Opportunities

Dani Guzman, Ex Libris

Continuing our analysis of the dynamics of research in the academic world, we are going to dive into the detailed survey of American, British and Australian researchers and research office leaders commissioned by Ex Libris and carried out by the Alterline research agency.

We have already reviewed key priorities for the research office, the effects of COVID-19 on researchers, and the role of Open Access in research. Here, we’ll take a look at the challenge of defining and measuring the impact of academic research.

As one UK-based university faculty member put it, “Impact is quite an indirect and nebulous thing. I have to report on impact for funding purposes, but I hate doing it and feel that it’s a box-ticking exercise.”

Of course, that academician is not alone. 71% percent of researchers say they have to report the impact of their research to funding partners “always” or “some of the time.” Similarly, 51% of research office members stated they had to report research publication data to funders or government agencies in the last three years. For many funding organizations, public or private, impact is a measure of their “return on investment” and can affect future financial support.

Researcher Frustration with Traditional Research Metrics

The question of how the impact of research is measured has become an increasingly complex one.

Traditionally, the number of academic citations of published research was the primary – of not the only – yardstick. Even today, according to the survey data, 90% of researchers are measuring impact through citations.

However, researchers also named several other, more modern forms of engagement with their work as impact metrics. These include direct views (34%), legacy media mentions (30%), social media mentions (29%), and references in public policy papers and government regulations (16%).

Researchers report frustration with citations and other metrics dependent on academic publications. This is especially so in fields where academic citations are less frequent, such as in the humanities and the arts. Moreover, many argue that current metrics provide only a narrow view, missing the real-world impact research may have on the general society. Fueling social change can include influence on governmental policies, regulations, patents, collaboration with industry ventures, influence on local community initiatives, and more.

Research Offices Struggle to Find a Consistent Way to Measure Impact

Among other aspects of their work, research offices often contribute to helping researchers monitor the impact of their work. 38% of senior research office personnel believe this is one of the main priorities at their institution. This may be part of improving the university’s prestige, which they named as the second most important activity of their office.

Like researchers, research office members across the US, UK and Australia gave heavy weight to citations as an impact metric (63%). However, 50% of respondents named journal impact factor as the second most important KPI. In addition, they significantly diverged with researchers on the value of public policy papers and regulations as indicators of research impact (38%), as well as on social media (19%) and direct views (13%).

The conflicting responses are indicative of a lack of consistent measurement, as researchers and research offices adapt to changing circumstances, requirements, and technologies.

As one American university faculty member put it, there are “so many different figures – h-index, Impact Factors, media mentions, and the like. Everyone counts different things and includes different journals in different counts. It’s very nonstandard.”

Even if a research office decides to include journals or media mentions in its impact assessment, there is no clear standard as to which publications are considered valuable in this regard. Should it be based on circulation, prestige, specialization, popularity, or something else?

The Future of Measuring Research Impact

Research offices and researchers identified challenges in accurately assessing the research impact metrics in the short term and linking them back to individual publications. In many cases, especially public policy and regulations, the impact only becomes truly evident after a lengthy period of time.

With more outlets and broader audiences able to engage with research than ever before, demonstrating its true impact is becoming increasingly difficult. Nonetheless, the pressure to demonstrate impact has not relaxed. While citations remain the most popular choice, they are increasingly seen as a necessary evil, a sort of “best guess”.

One approach to improving the quality and accuracy of impact metrics is promoting cross-institution, cross-disciplinary collaboration to identify the true influence of research globally. This includes the direct use of research in innovative technologies, products and procedures, as well as its use as building blocks for novel theories, societal behavior changes, and methodologies. Within institutions, better impact metrics can be obtained with greater collaboration between researchers and research office staff in the sharing of data and resources.

In the UK and Australia, government agencies (the UK’s Research Excellence Framework and the Excellence in Research for Australia) have developed a system for evaluating research quality over the long term. Recognizing that research impact is often realized over the course of years, these frameworks aim to reflect the wider effects of research on the economy, culture, public policy, health, the environment and quality of life. Although far from perfect, such impact frameworks can point the way forward.

While most researchers would prefer to spend all their time on their research, the need to track its impact and other administrative duties take up a lot of their time. In our next installment, we’ll see how this may be a key opportunity for the research office and library to step in and reduce this burden.

For more on how researchers and research offices see the impact of their institutional obligations, read the full Ex Libris-Alterline study here.