Perspectives and challenges for an impact agenda in the area of Education: a process of construction

Iandra Maria Weirich da Silva Coelho^[1]

Abstract

The aim of this study is to deepen the discussion on the impact on the area of Education. Through a literature review and analysis of Capes documents, it aims to highlight the diversity of perspectives and the challenges associated with assessing impact, both in the academic and social spheres. The study emphasizes the fundamental role of the scientific article in disseminating knowledge and generating impact, underlining the need to expand understanding of the benefits that research provides to society. It also emphasizes the importance of developing a theoretical and methodological framework capable of addressing multidimensional impact assessment, taking into account the complexity and diversity of educational contexts. Finally, it emphasizes the necessity of fostering institutional strategies and policies that promote the production of research with significant impact and make it possible to continuously monitor and evaluate its results.

Keywords: education; research impact; evaluation; scientific production; scientific article.

Perspectivas y desafíos para una agenda de impacto en el área de la Enseñanza: un proceso de construcción

Resumen

Este estudio tiene como objetivo profundizar la discusión sobre el impacto en el área de la Enseñanza. Mediante una revisión bibliográfica y un análisis de documentos de Capes, busca evidenciar la diversidad de perspectivas y los desafíos asociados a la evaluación del impacto, tanto en el ámbito académico como en el social. El estudio destaca el papel fundamental del artículo científico en la difusión del conocimiento y en la generación de impacto, subrayando la necesidad de ampliar la comprensión sobre los beneficios que las investigaciones aportan a la sociedad. Además, enfatiza la importancia de desarrollar un marco teórico y metodológico capaz de abordar la evaluación multidimensional del impacto, considerando la complejidad y diversidad de los contextos educativos. Finalmente, se resalta la necesidad de fomentar estrategias y políticas institucionales que promuevan la producción de investigaciones con un impacto significativo y permitan el monitoreo y la evaluación continua de sus resultados.

Palabras clave: enseñanza; impacto de la investigación; evaluación; producción científica; artículo científico.

Introduction

The search for impact and internationalization has been consolidated as one of the main transformations in science in recent decades (Wood Junior; Wilner, 2023). According to Lima (2023), the definition, measurement, and incorporation of this impact as an institutional practice and culture has gained significant strength in this period. This ongoing cultural change, which is intensifying in research and higher education institutions, requires not only understanding this concept, but also promoting it strategically. This transformation, still under debate, has the potential to shape the future of scientific production and dissemination, redefining its guidelines and priorities.

In this sense, it is essential to discuss how the intellectual production generated by students and teachers in the area of Education can be disseminated widely and generate significant impact. This production and dissemination of knowledge are central aspects for assessing impact in terms of scientific and technological progress.

In order to disseminate the contributions of this research, it is essential to invest in processes for creating knowledge and mobilizing paths to impact (Lima, 2023). Thus, scientific communication stands out through the publication of articles, recognizing that “[...] the main element that any scientific article must present and develop throughout the text is its contribution to the field in which it is inserted” (Bispo, 2023, p. 2).

In addition, it is essential to go beyond the mere dissemination of scientific results, seeking to highlight the potential social impact of the research carried out. This search for a broader impact has been the subject of debate in the national and international scientific community (Bispo, 2023; Boer et al., 2015; Nakano; Muniz Júnior, 2018; Sandes-Guimarães; Hourneaux Junior, 2020). Identifying the potential benefits that research can bring to society highlights the social relevance of science and encourages the application of scientific knowledge to solve problems.

From this perspective, evaluating the impact of research carried out in the field of Education is justified by the need for greater transparency and the potential for social transformation of scientific results. According to Lima (2023), understanding and engaging in the research impact agenda implies recognizing the transformative role of science for society.

As Coelho (2024a) points out, there is a clear need for specific debates on the creation of this agenda, impact assessment indicators and innovation in education. However, the inherent complexity of this evaluation, marked by its dynamic and multifaceted nature, requires the adoption of rigorous methodological approaches and the consideration of multiple factors.

Given the above, this study aims to discuss academic impact, with a focus on the publication of articles, and educational impact, often “[...] associated with the production of teaching material or content that supports and innovates teaching and learning processes” (Wood Junior; Wilner, 2023, p. 2). The aim is to contribute to understanding and promoting the educational and social impact that can be generated, especially through the development and implementation of educational processes and products, integrated with rigorous, high-quality scientific research.

To this end, the concept of impact, its relevance and the main challenges in communicating the social benefits of research are addressed, in light of the criteria defined by Capes' guiding documents (Brasil, 2019a, b, c). These guidelines are complemented by definitions from different studies, analyzed and related through a critical review (Alla et al., 2017; Bispo, 2023; Coelho, 2024a,b; Boer et al., 2015; Lima, 2023; Mendes-da-Silva, 2020; Nakano; Muniz Júnior, 2018; Sandes-Guimarães; Hourneaux Junior, 2020).

Impact and construction of science

The literature presents different perspectives and definitions on the impact applied to research results, highlighting the diversity of understandings on the topic. It is observed that there is no consensus on a single definition for the term (Brasil, 2019a; Lima, 2023; Sandes-Guimarães; Hourneaux Junior, 2020), which poses a challenge for the planning and comprehensive assessment of this factor.

The term impact is considered a multidimensional (Coelho, 2024a; Wood Junior; Wilner, 2023) and polysemic construct, allowing for several interpretations (Coelho, 2024a). “In addition, it can take place over time and is the result of multiple interactions with different agents. Measurement can also take up considerable time and resources” (Wood Junior; Wilner, 2023, p.16).

The diversity of impact concepts reflects the complexity of their evaluation, which depends on the nature of the research, the type of contribution generated and the perspectives of the different actors involved (Sandes-Guimarães; Hourneaux Junior, 2020). This heterogeneity requires understanding and applying methodologies adapted to each context, taking into account the specificities of each area of knowledge and the objectives of the research.

Sandes-Guimarães and Hourneaux Junior (2020, p. 283) state that the impact of science on society has gained more strength in the last two decades, considering some main reasons:

i) a demand for transparency of investments in research financed with publicresources, mainly in terms of the generated outcomes; ii) a more significant concern with the use of scientific evidence to support decision-making (in both public and business sectors; and iii) the need to demonstrate how research is contributing to the society’s grandchallenges – in other words, its relevance and impact on society – and not just forscience itself (Sandes-Guimarães; Hourneaux Junior, 2020, p. 283).

Alla et al. (2017), in a systematic review, identified four main categories of research impact definition: i) in terms of positive changes or effects resulting from the application of evidence in policies and practices; ii) as measurable results; iii) defined by bibliometric metrics; and iv) based on the effective use of research results.

In general, impact is treated “as some kind of contribution, influence, value, change, effect or benefit that scientific research brings to society in its most diverse spheres (economic, social, health, public policies, etc.), directly or indirectly” (Sandes-Guimarães; Hourneaux Junior, 2020, p. 284). Lima (2023) corroborates this, highlighting definitions such as consequences or effects of research. For the author (2023, p. 68), “these effects are multifaceted and complex, and vary significantly over time and among the communities involved”.

In short, the impact can come from both the research process and its results, and be positive or negative, potential or real, direct or indirect. In addition, it can be classified by different types or natures, manifest itself at different times and scales (immediate, short, medium and long term) and cover multiple levels of scope (local, national or international) (Brasil, 2019a, b, c; Coelho, 2024a; Lima, 2023; Wood Junior; Wilner, 2023).

According to the document from the Coordination for the Improvement of Higher Education Personnel (Capes) (Brasil, 2019a, p. 49), the impact “represents the result, consequences or benefits perceived by society, caused by the introduction of products and technologies developed in postgraduate studies”. In this context, the impact “needs to be demonstrated based on its results and effects, in the medium and long term, in terms of the advancement of scientific and technological knowledge in the area, as well as the repercussions of the investigations on the quality of Brazilian education” (Brasil, 2019a, p. 10).

Therefore, for a postgraduate course to have an impact on society, it needs to result in a set of benefits perceived by users, that is, it must satisfy two conditions simultaneously: 1) it must be perceived as something relevant, filling a gap and solving problems; 2) it must present results (in the form of ideas, products and services) with satisfactory performance (Brasil, 2019b, p. 15).

In this scenario, it is important to take into account definitions related to the terms benefit and relevance. Benefits are understood as the set of real gains perceived by the public when using a given product. According to Lima (2023, p. 56), “the creation of benefits and social value from scientific research is the result of a process that can take place throughout the research cycle, not just linked to its results or publications [...]”. On the other hand, relevance/importance is an exclusive attribution of the individual user and not the author of the work (Capes, 2019a).

The documentation analyzed also highlights the importance of distinguishing between the potential and actual impact of research. The potential impact refers to the expectations of researchers (teachers and students) in relation to the results of their research and the generation of significant impacts on society, while the actual impact materializes when the products resulting from the research are actually used by the target audience (Brasil, 2019a).

In this sense, it is important to emphasize that the impact assessment must take into account the analysis of a comprehensive time perspective, as some effects may arise in the long term, allowing for greater chances of capturing the impact trajectory more effectively. This need to consider different time scales is evidenced both in Capes documents and by other scholars (Brasil, 2019a, b; Coelho, 2024a; Lima, 2023; Wood Junior; Wilner, 2023).

It is believed that understanding the different terms mentioned here can help the researcher structure what they want to achieve with their educational product, evidenced by results, effects, benefits, contributions linked to the training process, learning content, developing skills, in accordance with the research objectives (Coelho, 2024a).

Challenges to foster practical impacts

As pointed out by Wood Junior and Wilner (2023), academic production traditionally focuses on topics originating in the academic sphere and developed within specific disciplines, ending with publication in scientific journals aimed at researchers. This approach, although important, can limit the interaction between research and society. Therefore, the authors reinforce the need to “[...] foster practical impacts - organizational, social and in public policies - in parallel with academic impacts - scientific and educational” (Wood Junior; Wilner, 2023, p.15).

This is a relevant demand, but there are different challenges related to the complexity of evaluating the impact of research, both in its conception and operationalization. The evaluation of social impact is also a complex matter, especially due to the fact that, “[...] it is difficult to isolate what is in fact the result of research and what are external factors that can affect the results” (Wood Junior; Wilner, 2023, p.4).

Considering the context of the Teaching area, some of these challenges are evident. This reflection begins by taking into account that “[...] in the humanities (understood here as the set of areas of knowledge that are classified as humanities, applied social sciences, letters and arts) the impacts are more intangible than in the so-called exact sciences”[2] (Brasil, 2019a, p. 10). According to Coelho (2024a), the impacts of research in the Arts and Humanities are difficult to identify and measure, due to the lack of specific qualitative indicators and criteria that can help analyze the data.

An additional challenge relates to evaluating the impact of educational products, a multifaceted and complex process, especially when considering the need to monitor results on different time scales (immediate, short, medium and long-term impact). As pointed out by Lima (2023), some effects may take longer to manifest themselves. Given this scenario, and considering the area of Education, how can we guarantee a continuous evaluation, after the research has been completed, to measure the medium- and long-term impacts? And who would be responsible for monitoring these impacts on learning in the medium and long term?

The perception of relevance, which varies significantly between students and teachers, also makes impact assessment complex. This subjectivity makes it difficult to generalize the results (Brasil, 2019a, p. 14). “It follows that it is difficult to measure the benefits provided by a postgraduate course, because for each target audience receiving the solution, the perception of benefit will be different”.

In order to boost the social impact of these products and related research, it is necessary to develop strategic actions involving the institution, graduates and other social actors. These actions should include concrete activities, such as creating mechanisms for generating, communicating and disseminating knowledge, carried out at defined time intervals, identifying the potential types of social impact generated and indicators for evaluating them.

It is also worth noting that assessing the impact of educational products is complex, as the perception of their relevance varies from individual to individual (students, teachers). This subjectivity makes it difficult to generalize the results, as CAPES (Brasil, 2019a, p. 14) points out: “Hence, it is difficult to measure the benefits provided by a postgraduate program, since the perception of benefit will be different for each target audience receiving the solution”.

Impact of research from postgraduate studies

This section discusses the evaluation of the impact of two outputs of a postgraduate program, especially in the professional field: scientific articles and educational products. In this sense, impact “[...] designates a measure of how much a postgraduate output is capable of generating positive effects for a community when the solution is available for use” (Brasil, 2019a, p.15). To this end, it is essential to reflect on how to broaden the dissemination of results and the impact of these results, in order to highlight practical applicability and the resolution of real problems, seeking the continuous improvement of teaching and learning processes, both in formal and informal environments. The impact, whether immediate or long-term, can be seen at different levels, from learning content to developing skills.

It is worth noting that impact is one of the axes pointed out by the guiding documents for evaluating educational products. In order to evaluate this axis, it is necessary to understand the demand (the reason and relevance of the creation), the objective of the research (the focus of the product's application) and the area impacted by the research, making it possible to evaluate in which areas changes can be perceived (Brasil, 2019c).

Therefore, evaluating the contributions (impacts) of these products can contribute to the continuous improvement of teaching and learning processes and meet the demands of contemporary education. This evaluation considers multiple dimensions, such as repercussions on teaching practice, scope of use, ease of access, applicability in different contexts, adherence to user needs, degree of innovation, possibility of replication in other environments, complexity of implementation, alignment with postgraduate program guidelines, among other aspects. The analysis of these dimensions makes it possible to identify the effective contributions of the products and direct future development actions (Coelho, 2024b).

Given the relevance of educational products for improving the quality of teaching and learning, disseminating the results of research in this area is essential. Publication in scientific journals, after defense, emerges as an effective mechanism for sharing the knowledge generated with the academic community and society in general, generating a significant initial impact. In the current context, marked by a new system for classifying intellectual production, this means of dissemination is even more important, consolidating itself as a privileged channel for disseminating new practices and knowledge in the field of Education[3].

It is essential that scientific articles clearly demonstrate the development, application and evaluation of educational products, based on rigorous data and a solid theoretical framework (Coelho, 2024a, b). In this context, the formulation of research questions capable of identifying the contributions and limitations of these products becomes essential to ensure the relevance and applicability of the results.

According to Coelho (2024a), this approach seeks to encourage impact assessment based on previously established indicators, making a significant contribution to building knowledge in the area and disseminating contextualized practices. Considering that the development of educational products is a complex process, evaluating their impact is equally indispensable.

This involves collecting evidence on the sustainability of the effects of interventions and providing information on what works and what does not, considering the objectives, circumstances and dimensions involved. Furthermore, it aims to inform possibilities for expanding, modifying and replicating processes and products (Coelho, 2024a, p. 9).

This approach seeks to transcend the limits of academia and can encompass a potential assessment of social, cultural, political and/or environmental impacts, allowing for a more assertive alignment of research with the demands of society. Despite recognizing the importance of science being more responsive to social needs and fostering multidimensional impacts, as advocated by Wood Junior and Wilner (2023), there is still much to be done. Debates need to be deepened to guide researchers in their search for these impacts and to create the institutional conditions to support them.

In this context, it is important to note that evaluating the social impact of research and the educational products generated is not a trivial activity (Brasil, 2019a). It is a complex task, which requires identifying, measuring and communicating the transformations and contributions generated by research in society, which requires a critical and reflective eye, as well as the adoption of specific methodologies and the consideration of various factors, such as the social, cultural and historical context.

As Lima (2023) points out, resistance to social impact assessment in the academic environment is still a challenge to be overcome. In order to make progress in this direction, it is necessary to deepen the discussion on impact concepts and indicators, as well as to develop robust methodologies for measuring them. As Coelho (2024a) states, the adoption of multidimensional approaches, which involve careful planning and rigorous methodology, is fundamental to promoting significant improvements in this process.

Given this complexity, Education research stands out as a fertile field for reflecting on the impact of scientific production. In order for this to happen, it is essential to establish an impact agenda that guides studies, allows the effectiveness of actions to be assessed, and identifies possible paths to impact. According to Lima (2023, p. 56), the paths to impact “are the processes of knowledge creation and mobilization that help bring about these changes” and “[...] include initiatives related to scientific communication, co-creation, dissemination and exploitation of research, including commercialization and technology transfer”. Identifying these processes is part of this agenda.

In this sense, the definition and evaluation of the social impact of research and educational products emerge as crucial issues to be debated in postgraduate programs, considering how this practice will be adopted and which actors will be responsible for conducting and maintaining it.

This implies the development of new evaluation approaches to help assess the social effects of educational research and products. This task requires the consolidation of key concepts, the mobilization of resources and the adoption of diversified research strategies.

Initially, it could be proposed that evaluation models be adapted to the specific characteristics of the area, taking into account the criteria, methods, expectations and priorities specific to the field, incorporating multiple dimensions of impact that go beyond simple measurement by indicators. This adaptation requires valuing qualitative approaches and accepting the subjectivity inherent in evaluating complex and multidimensional phenomena.

Given this scenario, the importance of institutional positioning with the community as a catalyst for the adoption of social impact stands out. It is essential to recognize that the effort to generate social impact is not restricted to researchers, nor does it have to be an individual requirement.

To do so, it is necessary to build an ecosystem that encourages and sustains it. This involves creating an institutional culture that values impact generation and investing in policies, structures and resources that enable all projects to reach their maximum potential for social transformation. In addition, the area of knowledge must establish clear guidelines on which initiatives can contribute to social impact in this field (Lima, 2023).

Conclusions, implications and future research

This study sought to contribute to the debate on the impact of research in Education, stimulating reflection on how to define and measure this impact both in the academic sphere and in society. Based on the discussions presented, the study concludes that the scientific article, as the main product of research, plays a fundamental role in disseminating knowledge and generating impact.

Based on the discussion, we highlight the need for a careful, reflective and critical look to assess the impact of research, processes and products developed, seeking to understand the indicators that allow us to identify the contributions, effects and changes generated by the research. By intentionally defining an impact agenda, we can stimulate the search for evidence that demonstrates the social value of research and educational products developed.

The diversity of definitions for impact highlights the need for a clear conceptual framework and specific strategies to foster, measure, evaluate and communicate the impact of research in Education. The complexity inherent in this assessment requires a paradigm shift, both epistemological and attitudinal. In this way, institutions can adopt strategies and initiatives that promote the generation of impact, encouraging more practice-oriented models of knowledge production and the creation of innovative and contextual evaluation tools that can encompass the specificities of the area.

The discussions presented here open the way for future research that explores innovative approaches to evaluating the impact of scientific production in the field of education, taking into account the complexity and diversity of the contexts in which they are applied. Studies can further develop integrated impact assessment models that combine quantitative and qualitative indicators and incorporate the perspectives of the various actors involved in the process (researchers, managers, educators and benefiting communities).

References

ALLA, K. et al. How do we define the policy impact of public health research? a systematic review. Health Research Policy and Systems, [S. l.], v. 15, n. 1, 2017. Disponível em: https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-017-0247-z. Acesso em: 8 jan. 2025.

BISPO, M. S. Contribuições teóricas, práticas, metodológicas e didáticas em artigos científicos. Revista de Administração Contemporânea, [S. l.], v. 27, n. 1, e220256, 2023. Disponível em: https://www.scielo.br/j/rac/a/gvLGLDq49hfqkcgg3z4jrMD/?format=pdf&lang=pt. Acesso em: 8 jan. 2025.

BOER, H. et al. Making a meaningful contribution to theory. International Journal of Operations & Production Management, Denmark, v. 35, n. 9, p. 1231-1252, 2015. Disponível em: https://vbn.aau.dk/ws/portalfiles/portal/219125921/ijopm_2015_boer_et_al_om_theory.pdf. Acesso em: 8 jan. 2025.

BRASIL. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. GT impacto e relevância econômica e social: relatório final de atividades. Brasília, DF: CAPES, 2019a. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/2020-01-03-relatorio-gt-impacto-e-relevancia-economica-e-social-pdf/view. Acesso em: 5 dez. 2024.

BRASIL. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. Documento de Área – Ensino. Brasília, DF: CAPES, 2019b. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/ENSINO.pdf. Acesso em: 5 dez. 2024.

BRASIL. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES. Grupo de trabalho Produção Técnica. Brasília, DF: CAPES, 2019c. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-producao-tecnica-pdfAcesso em: 5 ago. 2024.

COELHO, I. M. W. S. Estratégias para a elaboração de perguntas de pesquisa científica e recuperação de informações no contexto educacional. Cadernos de Pesquisa, Maranhão, p. 1–24, 2024a. Disponível em: https://periodicoseletronicos.ufma.br/index.php/cadernosdepesquisa/article/view/23654. Acesso em: 19 dez. 2024.

COELHO, I. M. W. S. Desenvolvimento e avaliação de produtos educacionais na pós-graduação profissional em Ensino: metodologias e desafios emergentes. Revista Ensino em Debate, Fortaleza, v. 2, p. e2024039, 2024b. Disponível em: https://revistarede.ifce.edu.br/ojs/index.php/rede/article/view/82. Acesso em: 4 dez. 2024.

LIMA, G. M. R. Práticas Institucionais de Adoção do Impacto Social da Pesquisa. In: MARCOVITCH, J. (org.). Repensar a Universidade III: Saberes e Práticas. São Paulo: Com-Arte; Fapesp, 2023. p. 55-73.

MENDES-DA-SILVA, W. Lessons that can be learned from the rejection of an article. Revista de Administração Contemporânea, Paraná, v. 24, n. 4, p. 369-375, 2020. Disponível em: https://www.scielo.br/j/rac/a/y6h83M57x9DxYVHQtKLqrtq/?format=pdf&lang=pt. Acesso em: 8 jan. 2025.

NAKANO, D.; MUNIZ JUNIOR, J. Writing the literature review for empirical papers.

SANDES-GUIMARÃES, L. V.; HOURNEAUX JUNIOR, F. Research impact – what is it, after all? Editorial impact series part 1. RAUSP Management Journal, [S. l.], v. 55, n. 3, p. 283-287, 2020. Disponível em: https://www.scielo.br/j/rmj/a/Q6yvK3FXQdDVtfJb456GKYm/?format=pdf&lang=en. Acesso em: 7 jan. 2025.

WOOD JUNIOR, T.; WILNER, A. Impacto multidimensional da pesquisa: desenvolvimento e teste de um modelo para avaliação. Cad. EBAPE.BR, Rio de Janeiro, v. 21, n. 5, e2022-0258, 2023. Disponível em: https://scielo.br/j/cebape/a/wMwB9tCJr7gKtfPztXmcRpK/?format=pdf&lang=pt. Acesso em: 7 jan. 2025.