An analytical tool for the appraisal of Educational Products related to construction research

Abstract

The process of conceiving an Educational Product (EP) in Professional Postgraduate Programs in the area of Education requires, at one of its stages, the appraisal of related products. Although the analysis of related research is well explored in books on scientific methodology, it is not common to find guidance in the literature on how to promote the analysis of EPs. This article aims to present an analytical tool to assist graduate students in analyzing research-related EPs, in understanding existing gaps, and in identifying essential characteristics and criteria that can contribute distinctively to the conception of their own EPs. Bibliographic and documentary research was adopted as the methodological path. The result is an analytical tool that presents five categories of analysis for educational products: target audience, conceptual layer, communicational layer, didactic-pedagogical layer, and aesthetic and functional layer. In addition, a set of guidelines for preliminary analyses is presented to assist in the interpretation of the results. Given the result, the analytical tool presents the possibility of identifying improvements and modifications in the researcher's own EP, directing the generation of prototypes based on the items described, recording the analysis, and using the data to justify choices and decisions in the educational product.

Keywords: product evaluation; evaluation instrument; applied research.

Una herramienta analítica para la valoración de Productos Educativos relacionados con investigaciones en construcción

Resumen

El proceso de concepción de un Producto Educativo (PE), en Programas de Posgrado Profesionales del Área de la Educación, requiere en una de sus etapas la valoración de productos relacionados. Aunque el análisis de investigaciones relacionadas está bien explorado en libros de metodología científica, no es común identificar en la literatura orientaciones sobre cómo promover el análisis de PE. Este artículo tiene como objetivo presentar una herramienta analítica para ayudar a los posgraduados en el análisis de PE relacionados con la investigación, en la comprensión de las lagunas existentes, y en la identificación de características y criterios esenciales que pueden contribuir de forma distintiva a la concepción de sus propios PE. Se adoptó la investigación bibliográfica y documental como camino metodológico. El resultado es una herramienta analítica que presenta cinco categorías de análisis para productos educativos: público objetivo, capa conceptual, capa comunicacional, capa didáctico-pedagógica y capa estético y funcional. Además, se presenta un conjunto de orientaciones para análisis previos para ayudar en la interpretación de los resultados. Frente al resultado, la herramienta analítica presenta la posibilidad de identificar mejoras y modificaciones en el propio PE del investigador, dirigir la generación de prototipos basados en los ítems descritos, registrar el análisis y utilizar los datos para justificar elecciones y decisiones en el producto educativo.

Palabras clave: evaluación de productos; instrumento de evaluación; investigación aplicada.

Introduction

Scientific research conducted within a Professional Graduate Program (PPG) in the field of Education yields two distinct yet interconnected outcomes: the dissertation/thesis and an educational product. The dissertation/thesis "must be a reflection on the development and application of the educational product, supported by the chosen theoretical-methodological framework" (Brasil, 2019, p. 15).

Mendonça et al. (2022, p. 3) argue that the dissertation/thesis and its corresponding educational product relate to the problem space and the solution space, respectively.

In this process, it is essential to consider both the problem space and the solution space. The former pertains to the understanding and formulation of the research question/problem, which emerges from the professional practice field and informs the research objectives and the definition of its theoretical-methodological foundations. The solution space, in turn, encompasses the aspects that lead to a response to this problem (Mendonça et al., 2022, p. 3).

During the initial stages of research development, it is common for graduate students to encounter difficulties in conceptualizing the educational product, often prioritizing the writing of their dissertation or thesis. This, in turn, delays the planning of the educational product. According to Mendonça et al. (2022), this challenge may arise because graduate students dedicate significant time to coursework and adapt to modifications in their research projects, such as the (re)definition of the research problem and the comprehension of theoretical and methodological foundations.

In the same vein, Araújo, Farias, and Mendonça (2023) highlight that graduate students tend to dissociate the conception of the educational product from the research process, failing to recognize it as a continuous endeavor. They also struggle to identify the needs of the target audience and to ensure the product's aesthetic quality.

Moreover, it is still common for dissertations and theses to present research-related works (such as articles, theses, and dissertations) while neglecting the discussion of related educational products. One way to address this gap is to encourage the analysis of related works with a specific focus on educational products.

Although the literature provides guidelines on the structure of educational products or materials—addressing aspects such as key dimensions and layers (Kaplún, 2003; Filatro; Cairo, 2016; Moreira, 2010; Mendonça et al., 2022) as well as validation and evaluation checklists for products, learning objects, and similar materials (Sousa; Turrini; Poveda, 2015; Rizzatti et al., 2020; Rocha et al., 2024)—graduate students entering Professional Graduate Programs (PPGs) in the field of Education may face difficulties in applying these guidelines when analyzing other educational products during the construction of their theoretical and methodological framework.

To contribute to the literature on educational products, this study presents an analytical instrument for evaluating such products. The goal is to enable graduate students to qualitatively differentiate educational products and refine their analytical perspective regarding the essential elements that constitute these products. Given that educational products serve as valuable sources of information, they should contribute both to the theoretical framework and the conception of the researcher's educational product.

Theoretical FoundationsTheoretical Foundations

The analysis of educational products (EP) as primary sources assists graduate students in understanding the functionality, structure, and composition of such products within their defined research topic. It also enables them to map existing educational solutions, identify research gaps, and recognize opportunities that may not yet have been explored by other scholars.

To support the development of the analytical instrument, this study adopts the framework proposed by Mendonça et al. (2022), which conceptualizes the design of educational products through a layered structure, organized into the following dimensions: i) Conceptual, ii) Didactic-Pedagogical, iii) Communicational, and iv) Aesthetic and Functional.

The conceptual layer encompasses the theoretical and technical-technological aspects that facilitate the target audience’s understanding of the educational product—its composition, purpose, and underlying knowledge base. This layer should integrate the essential knowledge required to achieve the intended objectives of the product.

The didactic-pedagogical layer guides the educational pathway or learning trajectory, structuring the available information and resources to support the achievement of learning or professional training objectives. This layer focuses on instructional processes and pedagogical strategies embedded within the product.

The communicational layer pertains to how the educational product interacts with its target audience, which may include teachers, teacher educators, students, and educational administrators. As stated by Mendonça et al. (2022, p. 10), "this layer represents the 'voice' of the product, which engages with the reader and facilitates the learning process."

The aesthetic and functional layer aims to enhance the visual experience by presenting information in a clear and engaging manner. Additionally, it addresses usability by ensuring accessibility and ease of use.

While each layer is informed by research data, they interact dynamically throughout the product’s development in an iterative, interactive, and incremental process (Figure 1).

It is interactive because it requires observing the context surrounding the problem and engaging with the individuals involved. It is iterative because the solution demands a process of constant refinement, involving repeated cycles of analysis and revision to deepen the understanding of the problem and develop its solution. It is incremental since new information can be incorporated over time (Farias, 2019, p. 15).

Figure 1 – The layers that constitute an educational product.

Figure 1
– The layers that constitute an educational product

Source: Mendonça et al. (2022).

Although Mendonça et al. (2022) do not provide specific guidelines on how to analyze educational products (EP) about their layers, several authors—including Kaplún (2003), Moreira (2010), Filatro and Cairo (2016), Filatro (2018), Leite (2019), Souza, Moreira, and Borges (2020), and Brito Junior, Aguiar, and Moura (2021)—discuss elements that align with these layers and contribute to the development of the analytical instrument presented in this study. Additionally, Sousa, Turrini, and Poveda (2015) offer a significant contribution by translating the Suitability Assessment of Materials (SAM) from English to Portuguese, a 30-item instrument designed to evaluate the comprehensibility of educational materials.

Kaplún (2003) identifies three core axes that underpin educational materials, which can also be applied to educational products: the communicational, pedagogical, and conceptual axes. In these, the author emphasizes the importance of understanding the target audience, their prior knowledge, and experiences. He argues that the pedagogical axis is the central organizing element of educational material, guiding the selection of key ideas to be presented and ensuring that communication efforts avoid an overaccumulation of "technically accurate but pedagogically ineffective and overly complex information" (Kaplún, 2003, p. 58).

Moreira (2010) categorizes educational materials into three dimensions: semantic, syntactic, and pragmatic. The semantic dimension pertains to the content, information, and messages conveyed—essentially, "what the medium communicates." The syntactic dimension concerns "how the message is presented," including how information is structured, organized, and symbolized. The pragmatic dimension relates to the use of the medium, which the author refers to as a teaching tool.

Filatro and Cairo (2016) identify five key dimensions for the development of educational content: pedagogical, techno-scientific, communicational, technological, and organizational. The organizational dimension pertains to models, processes, and resources applied over time for educational content production in alignment with institutional policies. The techno-scientific dimension is linked to curriculum organization, representing what is deemed essential for achieving learning objectives. The pedagogical dimension addresses epistemological foundations and approaches to teaching and learning in the context of content production. The communicational dimension relates to the didactic dialogue established between the educator and the learner, aiming to ensure communication quality that enhances the learning experience. Lastly, the technological dimension concerns the selection of technologies that facilitate the communicational process and enable learning activities.

Filatro (2018) takes a pragmatic approach, providing guidelines for developing content, particularly for distance education, some of which can also be applied to educational products. The author offers recommendations such as keeping introductions to didactic texts or units brief and engaging, reserving technical illustrations for sections that specifically address the topic, ensuring that conclusions mirror introductions, and incorporating summaries that emphasize key points. Additional guidance is provided on visual identity, supporting materials, the use of images, multimedia scripting, and audience considerations.

Leite (2019) structures her analysis of educational materials around several axes, including aesthetics and organization, chapter structure, writing style, content, didactic proposals, and critical engagement. For each axis, she poses guiding questions that help researchers critically reflect on their educational products.

Souza, Moreira, and Borges (2020) developed an instrument for evaluating the visual design of health education technology, which includes ten items assessing illustrations, colors, and figures used in educational technologies.

Brito Junior, Aguiar, and Moura (2021) propose a taxonomy for evaluating digital learning resources, organized into four dimensions: pedagogical quality, usability, software quality, and hybrid quality. Each dimension comprises categories (17 in total) and criteria (49 in total) for analyzing digital learning objects. For example, within the pedagogical quality dimension, the content complexity criterion evaluates the level of complexity in learning activities, while the foundations and pedagogical objectives category assesses the identification of learning objectives, and the prior knowledge category examines whether learners’ previous knowledge is considered.

The Suitability Assessment of Materials (SAM), translated into Portuguese by Sousa, Turrini, and Poveda (2015) as Avaliação de Adequação de Materiais, is an instrument used to assess the appropriateness of educational materials for patients. It evaluates six categories: content, language, organization, layout and typography, illustrations, and learning and motivation. Each category includes a set of items rated on a scale from zero to two, ultimately classifying materials as excellent, adequate, or inadequate based on their scores.

This study highlights the need to include an additional analytical category: the target audience. While not considered a distinct layer of an educational product, understanding the characteristics, knowledge, and context of the intended audience is crucial for effective product analysis, as emphasized by Kaplún (2003), Moreira (2010), Filatro and Cairo (2016), Filatro (2018), and Aguiar and Moura (2021).

To address this consideration, an educational product may be designed for direct use by the target audience—allowing for autonomous adoption—or it may require mediation by an instructor or facilitator. Thus, the term target audience refers to those who directly benefit from the product.

For example, an educational product designed as a didactic sequence for first-year high school students learning physics would require mediation by a teacher who receives guidance on how to implement the sequence in the classroom. In this case, the target audience consists of first-year high school students, as they are the direct beneficiaries, whereas the teacher, despite utilizing the product, serves as a mediator rather than a direct recipient.

If the product is intended for teacher training, it is essential to determine whether it will be used autonomously (i.e., a teacher using the product for self-study) or whether a teacher educator will facilitate its implementation. In both cases, the target audience consists of teachers, but in the latter scenario, mediation is required.

The definition of the target audience is informed by research data, taking into account the identified problem and the context in which the educational product will be applied. This consideration significantly influences writing style, instructional design, content selection, and visual language, all of which are integral components of an educational product's layers.

Table 1 summarizes the relationships between the cited authors and the layers identified by Mendonça et al. (2022), highlighting their contributions to the development of the analytical instrument presented in this study.

Quadro 1- Theoretical foundations of the analytical instrument.

Table 1
summarizes the relationships between the cited authors and the layers identified by Mendonça et al 2022 highlighting their contributions to the development of the analytical instrument presented in this study

LAYERS (Mendonça et al. (2022)				TARGET AUDIENCE
CONCEPTUAL	DIDACTIC-PEDAGOGICAL	COMMUNICATIONAL	AESTHETIC AND FUNCTIONAL
Kaplún (2003)	Kaplún (2003)	Kaplún (2003)	Kaplún (2003)	Kaplún (2003)
Moreira (2010)	-	Moreira (2010)	Moreira (2010)	Moreira (2010)
Filatro & Cairo (2016)	Filatro & Cairo (2016)	Filatro & Cairo (2016)	-	Filatro & Cairo (2016)
-	Filatro (2018)	Filatro (2018)	Filatro (2018)	Filatro (2018)
Leite (2019)	Leite (2019)	Leite (2019)	Leite (2019)	-
-	-	-	Souza, Moreira & Borges (2020)	-
Brito Junior, Aguiar & Moura (2021)	Brito Junior, Aguiar & Moura (2021)	Brito Junior, Aguiar &e Moura (2021)	Brito Junior, Aguiar & Moura (2021)	Brito Junior, Aguiar & Moura (2021)
-	Sousa, Turrini & Poveda (2015)	Sousa, Turrini &Poveda (2015)	Sousa, Turrini & Poveda (2015)	Sousa, Turrini & Poveda (2015)

Source: Prepared by the authors (2024).

An important point is that the analytical instrument considers only the typology of didactic/instructional materials. Despite this limitation, the instrument remains relevant given the significant number of educational products in this category compared to others. Table 1 presents data from the Sucupira platform (2024) over the past five years, considering only the typologies identified by Rizzatti et al. (2020).

Table 1 – Number of educational products over the past five years.

Table 1
– Number of educational products over the past five years

CATEGORY	2019	2020	2021	2022	2023
Development of didactic/instructional material	1524	1618	1534	1456	1639
Short-term course	1092	566	610	713	667
Event organization	1068	722	813	709	782
Radio or TV program	418	812	580	393	301
Research report	105	95	86	66	66
Application development	72	57	66	76	77
Letters, maps, or similar items	16	8	1	5	5

Source: Sucupira (2024).

The didactic/instructional material category includes teaching proposals, didactic sequences, intervention plans, and workshop guides; textual materials such as manuals, guides, support texts, articles in technical or outreach journals, textbooks, supplementary educational books, comic books, and similar resources; educational media, including videos, simulations, animations, video lessons, virtual experiments, and audio materials; learning objects; learning environments; websites and blogs; tabletop or digital educational games, among others.

The next section presents the analytical instrument along with guidelines to assist graduate students in conducting their analysis.

Analytical Instrument for Evaluating Educational Products

The analytical instrument for educational products is theoretically grounded in the contributions of the authors referenced in Table 1, where elements corresponding to the layers described by Mendonça et al. (2022) were identified.

The instrument is organized into five categories: target audience, conceptual layer, didactic-pedagogical layer, communicational layer, and aesthetic-functional layer, comprising a total of 22 questions distributed across these categories. Given the nature of some products within the "didactic/instructional materials" category—such as videos, simulations, animations, video lessons, virtual experiments, and audio materials; learning objects; learning environments; websites and blogs; and tabletop or digital educational games—the instrument may not always be directly applicable.

For each question in the instrument, preliminary guidelines are suggested to assist graduate students in interpreting the criteria. These guidelines also help students justify their evaluations when writing their dissertation or thesis, particularly when a product does not meet a given criterion. The analysis of responses is not restrictive or prescriptive, as various factors may influence the assessment, including the type of product, its application context, the researcher’s available resources, technical knowledge, and more.

For scoring each question, the following scale is recommended 2 to Excellent; 1 to Adequate; 0 to Inadequate; N/A to Not applicable to the educational product. The maximum possible score is 44 points (100%), indicating an "excellent" rating across all questions. To ensure accurate calculation, the formula outlined in Table 2 should be followed. This formula is adapted from the study by Souza, Turrini, and Poveda (2015) on the Suitability Assessment of Materials (SAM).

Table 2 – Formula used in the analytical instrument.

Table 2
– Formula used in the analytical instrument

FORMULA FOR QUALIFYING THE ANALYSIS OF EDUCATIONAL PRODUCTS

N= N/As x 2=___ Pt= (Pmax-N) Score Percentage = (Tt / Pt) x 100

Source: Souza, Turrini, Poveda (2015).

Caption:

Tt = Total score

Pmax = Maximum total score = 44 (fixed value)

N = Number of "Not Applicable" (N/A) responses × 2 = ___

Pt = Adjusted maximum total score = (Pmax - N)

Interpretation of score adequacy (Excellent, Adequate, Not Acceptable):

0 – 59%: Not Acceptable – The product does not meet the minimum requirements for proper application / 60% – 79%: Adequate – The product meets the minimum requirements for proper application but has room for improvement / 80% – 100%: Excellent – The product is fully suitable for application.

Note: If the educational product scores zero in any category, it is automatically considered unsuitable for application.

Example:

A product received a total score of 17 points in the analytical instrument, with 3 questions marked as "Not Applicable." Therefore, Tt = 17, N = 3 (Number of "Not Applicable" responses) × 2 = 6. Therefore, Pt = 44 (fixed value) - 6 = 38. To calculate the score percentage, apply the formula Score Percentage = (17 / 38) × 100 = 44%. Score interpretation: Not Acceptable

To facilitate understanding, the instrument will be presented by category, along with analysis guidelines. To download the complete instrument and its application, please access the provided link.

Target Audience Category. This category includes questions that assess whether the target audience is clearly identified in the educational product, whether the need for a mediator is specified, whether clear instructions for implementation are provided, whether prior knowledge is required for use, and whether the content aligns with the expected knowledge level of the target audience. Failure to meet these criteria may indicate issues in defining the target audience, potentially leading to inadequate content and guidance, which could compromise learning outcomes. The detailed breakdown is presented in Table 3.

Table 3 – Analytical instrument for educational products – Target Audience Category.

Table 3
– Analytical instrument for educational products – Target Audience Category

ANALYTICAL INSTRUMENT FOR EDUCATIONAL PRODUCTS
CATEGORY	QUESTIONS	SCORE	CONSIDERATIONS FOR ANALYSIS
TARGET AUDIENCE	Is the target audience clearly identified in the product? (e.g., specified in the technical sheet, introduction, or another section of the product)		If the target audience is not clearly identified, the content may be inappropriate for its intended users, leading to disinterest or confusion and ultimately hindering the learning process. Additionally, users may be unable to determine whether the material is relevant or suitable for their needs, which could result in its incorrect application.
	Is it clear whether the product requires mediation by someone (e.g., a teacher, teacher educator, or pedagogical coordinator) or if it is intended for autonomous use?		If it is not clear that the product requires mediation, the target audience may attempt to use it autonomously and inappropriately, which could compromise the learning process. Lack of clarity regarding the need for mediation may also create confusion about the roles of teachers, facilitators, or pedagogical coordinators, leading to gaps in the teaching process.
	If mediation is required, are there clear instructions on how to implement the product with others? (e.g., Does the text provide guidelines specifically directed at the mediator?)		Without clear instructions for the mediator, there is a risk of misinterpreting the content and methodology. Mediators may feel unprepared or lack confidence in using the material correctly, which could compromise the educational outcomes.
	If necessary, does the product provide clear instructions on the prior knowledge the target audience should have to use or apply it? (e.g., proficiency in specific software, experience with a particular subject, specific skills, etc.)		If the required prior knowledge is not specified, the target audience may struggle to follow the content or apply it correctly. The lack of information on prerequisites can lead to a frustrating experience, making users feel unprepared or inadequate to engage effectively with the material.
	Does the content align with the expected knowledge level of the target audience? (e.g., Does the product present technical or advanced physics concepts that are not suitable for the audience’s educational background?)		When the content does not match the expected knowledge level, it may be either too advanced or too basic, leading to demotivation or confusion. Overly technical or complex information that is not adapted to the target audience’s background can make learning inaccessible, while overly simplified content may be perceived as irrelevant, reducing engagement.

Source: Prepared by the authors (2024).

Conceptual Layer Category. This category includes questions related to the theoretical foundations and their application throughout the product. The absence of these foundations or their misalignment with other components of the product compromises its overall objective, its implementation, and the teaching-learning process. The detailed breakdown is presented in Table 4.

Table 4 – Analytical instrument for educational products – Conceptual Layer Category.

Table 4
– Analytical instrument for educational products – Conceptual Layer Category

ANALYTICAL RECORD FOR EDUCATIONAL PRODUCTS
CATEGORY	QUESTIONS	SCORE	CONSIDERATIONS FOR ANALYSIS
CONCEPTUAL LAYER	Does the educational product (EP) present the theoretical foundations applied throughout the material or in a specific chapter/module/unit? (e.g., in the design of activities, in the concepts presented, etc. Note: In some formats—such as games and applications—the structure may not allow for this presentation.)		The absence of clear theoretical foundations may render the material superficial, thereby undermining its credibility and the attainment of its educational objectives. Furthermore, the target audience might fail to comprehend the rationale behind the activities and content, resulting in fragmented and disjointed learning. This impedes the practical application and the construction of knowledge based on consistent educational principles. Lastly, there may be challenges in contextualizing or elucidating the content, as there is no clear theoretical reference to guide them.
	Are the theoretical foundations interconnected throughout the entirety of the product? (For example, are the foundations coherent with other chapters/modules/units through the themes, activities, and/or other resources of the PE, rather than being isolated within their respective chapter/module/unit)		When theoretical foundations are not integrated throughout the material, a disconnection between theory and practice may arise. This impedes learning by creating gaps between what is taught and how it is applied. By failing to relate them to the rest of the material, the target audience may struggle to understand how theoretical concepts apply to the activities and topics addressed, potentially feeling lost or overwhelmed. This negatively affects both engagement and comprehension of the material.

Source: Prepared by the authors (2024).

Didactic-Pedagogical Category. This category addresses issues related to the pathway that leads the student to learning, the instructions/guidelines presented, the internal articulation of the content, and its relationship with conceptual foundations. The inadequacy of the items may indicate a disorganization of the educational program, rendering it incomprehensible to the target audience and impacting their learning. Table 5 presents a detailed description of this category in the instrument.

Table 5 – Analytical Instrument for Educational Products – Category: Didactic-Pedagogical Layer.

Table 5
– Analytical Instrument for Educational Products – Category DidacticPedagogical Layer

	ANALYTICAL SHEET FOR EDUCATIONAL PRODUCTS
CATEGORY	SOURCE	SCORE	CONSIDERATIONS FOR ANALYSIS
DIDACTIC-PEDAGOGICAL LAYER	Does the product present a pathway that leads to learning in its proposal? (For example: Is the material organized with instructions that demonstrate a strategy for training teachers; or does it present an organization of information that allows the target audience to engage in autonomous study, with complex content presented gradually, etc.)		Without a clear pathway, the material may appear unstructured, hindering progress in learning. If the information is not organized gradually, the target audience may have difficulties studying independently, especially when facing more complex content. Mediators may not identify a clear teaching strategy, which compromises the efficient application of the material.
	Does the product provide examples or guidelines that facilitate its application in the proposed context and/or in various contexts?		The lack of adaptable examples or guidelines for different contexts can be perceived as a restrictive or inadequate educational product for various educational situations. The absence of practical guidelines may make it difficult for the target audience to adjust the material according to the specific needs of their class or teaching environment.
	Does the educational product exhibit internal pedagogical articulation among its topics? (For example: Do the chapters/modules/units follow an interconnected and coherent sequence of information and/or activities?)		If the topics are not coherently articulated, learning may become fragmented. The absence of a logical and interconnected sequence of chapters, modules, or units generates a confusing educational product, affecting the target audience's perception of quality and confidence in the teaching-learning process.
	Does the teaching-learning pathway or strategy align with the theoretical foundations presented in the educational product?		If the pedagogical strategy is not aligned with the theoretical foundations, the target audience may not understand how to apply the learned concepts. When there are inconsistencies between the learning pathway and the theoretical foundations, the material may appear contradictory.

Source: Produced by the authors

Communication layer. The questions are related to the manner in which information is presented in the educational product (PE), including its organization, spelling, clarity, conciseness, coherence, and logic among the presented topics. The absence of these elements compromises the understanding of the product, and its credibility, and may generate frustration and low adherence from the target audience. Table 6 presents a detailed description of the instrument pertaining to the communicational layer.

Table 6 – Analytical Instrument for Educational Products – Category: Communicational Layer.

Table 6
– Analytical Instrument for Educational Products – Category Communicational Layer

	ANALYTICAL SHEET FOR EDUCATIONAL PRODUCTS
CATEGORY	QUESTIONS	SCORE	CONSIDERATIONS FOR ANALYSIS
COMMUNICATIONAL LAYER	Does the organization of the material enable the target audience to understand what they should do or learn?		If the organization is not clear, the target audience may not understand the steps they need to follow or what they should learn or do, making it difficult to follow the flow of activities or comprehend the concepts. The target audience tends to spend more time understanding the instructions than applying the content.
	Are the artifacts produced by the author (if any) or indicated understandable and do they assist the target audience in what they should do or learn? (For example: language appropriate to the target audience's profile, coherent, concise, without the use of jargon or terms that hinder user comprehension)		If the artifacts (tools and resources) use inappropriate or overly technical language without clear explanations for the target audience who do not master the specific vocabulary, it may lead to a lack of understanding, and adherence to the product, and hinder its application.
	Does the product's text use correct grammar, is it understandable for the target audience (clarity), is it expressive using the minimum number of words (conciseness), and does it possess connection (coherence) and logic between the presented topics (cohesion)?		Incorrect use of grammar can undermine the product's credibility, making the text confusing and difficult to follow. If the text is not clear, the target audience may not understand the instructions or concepts. A verbose and non-concise text can overwhelm the user with unnecessary information, diverting attention from the main content. Additionally, if the presented topics do not have a logical connection between them, learning becomes unstructured, making it difficult to build knowledge in a fluid and sequential manner.

Source: Produced by the authors (2024)

Aesthetic and Functional Layer Category. The questions relate to the adequate use of images, colors, typography, the organization of the composition between text and images of the product and its artifacts, as well as access by the target audience to the PE. The inadequacy of these elements can cause visual discomfort, and difficulty in reading and understanding the information, which compromises learning and adherence from the target audience. Table 7 presents the detailing of the communicational layer in the analytical instrument.

Table 7 – Analytical Instrument for Educational Products – Category: Aesthetic and Functional Layer.

Table 7
– Analytical Instrument for Educational Products – Category Aesthetic and Functional Layer

	Analytical Instrument for Educational Products
CATEGORY	QUESTIONS	SCORE	CONSIDERATIONS FOR ANALYSIS
AESTHETIC AND FUNCTIONAL LAYER	Is there the use of images, graphs, infographics, or other resources that aid in the understanding of the message?		The absence of visual resources can make the content abstract or complex, hindering the assimilation of knowledge, rendering the material monotonous, and reducing interest.
	Is the adopted color palette uniform throughout the product and pleasing to the target audience?		A lack of uniformity in the color palette can create a sense of visual disorder. The incorrect selection of colors, whether poorly chosen or excessively contrasting, can cause visual discomfort, impairing readability and comfort while studying the material.
	Do the organization of text and graphical elements maintain uniformity across chapters/modules/units?		A lack of uniformity in graphical elements and text across different parts of the material creates an inconsistent learning experience and can make it difficult to locate information, as changes in formats and layouts confuse the target audience when searching for specific topics.
	Does the organization of text and elements allow for adequate reading and comprehension? (For example: The choice of fonts, size, organization of information hierarchy – titles, subtitles, topics, etc.)		Inadequate choices of fonts (type, size) and information hierarchy (titles, subtitles) can hinder the reading and comprehension of the content, making learning slower and more confusing. An unorganized layout, with poorly formatted text blocks or inadequate spacing, can cause visual fatigue, reducing the target audience's attention and focus.
	Do the images, graphs, infographics, or other presented resources possess visual quality? (For example: Images have low clarity)		Poorly defined images and unclear important details can appear unprofessional, diminishing the credibility of the educational material and making the content difficult to understand. In contexts where images are essential for conveying information, such as illustrations of processes, anatomies, or experiments, low clarity can prevent the target audience from correctly understanding the concept or information. Individuals with visual impairments or learning difficulties may be even more affected, making the material less inclusive.
	Does the product present an excess of information or graphical elements that hinder its comprehension?		If the educational product presents an excess of information or graphics that overload the target audience's cognition, it can impede the assimilation of content by attempting to process multiple pieces of information simultaneously. Additionally, the presence of too much data or imagery can make the material less clear and can create difficulties in identifying what is truly important for learning.
	Is the product made available on a platform, website, or other medium that facilitates its access?		If the product is not on an accessible platform or medium, users may have difficulty finding, downloading, or using the material, which can significantly reduce its usability. Poorly organized, unstable platforms that require many technological resources or are difficult to navigate discourage the target audience, who may abandon using the product due to frustration caused by an unfriendly interface.
	Do the artifacts produced by the author (if any) or indicated possess visual quality and are easy to use?		Artifacts with poor graphic quality (low resolution, confusing design) and that are difficult to use can impair the target audience's experience, making it harder to understand and apply the content.

Source: Produced by the authors (2024)

The application of the analytical instrument contributes not only to the analysis of other products by postgraduate students but also to the practice of postgraduate instructors. Additionally, it reveals aspects that are not easily perceived by novice postgraduate students.

How to use the analytical instrument and what it can reveal

The use of the instrument is recommended for the product analysis phase during the research´s data collection. However, other possibilities benefit the learning of postgraduate students who are beginning to understand educational products. Below are some highlighted applications of the proposal.

As an exercise to develop the postgraduate student's perspective. The analysis of products primarily benefits novice students in master's or professional doctoral programs, or doctoral candidates who have completed academic master's degrees, by assisting them in training to identify the minimal elements of the layers that compose the educational product (PE). This provides a reflective process on what constitutes an educational product, its purpose, and its composition through its layers.

To qualify and differentiate educational products. The instrument provides a means to qualitatively evaluate products based on established criteria. This allows for the ranking of products with higher scores and establishing relationships with one's own research.

Guidelines for the conception of the educational product. Considering the developed categories and questions, the researcher can adopt the instrument as a parameter to guide the conception of their own product, ensuring at least the application of the layers.

Assistance for postgraduate instructors. It helps instructors use it in their courses as an analysis exercise or in discussions with research groups during the guidance of dissertations or theses, thereby guiding postgraduate students to consider elements that should be mapped during the development of their theoretical framework.

The analytical instrument considers five categories based on the research of Mendonça et al. (2022). For the operational structuring of the categories, foundations were sought in Kaplún (2003), Moreira (2010), Sousa, Turrini, and Poveda (2015), Filatro and Cairo (2016), Filatro (2018), Leite (2019), Souza, Moreira, and Borges (2020), Brito Junior, Aguiar, and Moura (2021). These authors provide indications that can be applied to the context of educational products (see Table 1) such as the development of the analytical instrument or to support other research.

This proposal aims to assist postgraduate students in their research under construction and enhances the researcher's understanding of the connection between research and the educational product. However, adaptations may be made according to the type of product and/or other issues pertinent to the researcher in the analysis of products.

Final Considerations

A postgraduate student beginning their studies in a master's or doctoral program has a range of activities to complete. The initial focus on coursework and the structuring of their research project is a natural process; however, in a Professional Program in the field of Education, the initiation of the educational product tends to be postponed. This article presents an analytical instrument for educational products with the objective of contributing to postgraduate students in their ongoing research.

The analytical instrument for educational products (PE) assists in the first phase of research by enabling the researcher to: (1) structurally analyze the characteristics and criteria of educational products, facilitating the comparison of the analyzed PEs; (2) present a view of the minimal elements related to the layers that compose the PEs and align them with the research objectives; (3) identify gaps, improvements, and modifications in their own PE by analyzing other products; (4) direct the generation of prototypes based on the described items and their monitoring, avoiding deviations and reducing the need for late revisions and corrections; (5) record the analysis and utilize the data to justify choices and decisions, which aids in communication with advisors and examination boards.

Among the limitations, the instrument is restricted to the typology of didactic/instructional materials; even though it provides guidelines for analyzing the questions, it is necessary for the postgraduate student to have a minimum knowledge of the layers present in the article by Mendonça et al. (2022). Additionally, although it is an instrument capable of qualitatively evaluating PEs, the researcher needs to consider the results and the impact on the product of items with low scores.

Acknowledgments

This research was developed with the support of the Government of the State of Amazonas through the Fundação de Amparo à Pesquisa do Estado do Amazonas, with the granting of a scholarship.

References

BRASIL. Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Documento de Área – Ensino. Brasília, 2019. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/ENSINO.pdf. Acesso em: 10 out. 2024.

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Plataforma Sucupira. Disponível em: https://sucupira-v2.capes.gov.br/painel. Acesso em: 16 out. 2024.

FARIAS, M. S. F.; MENDONÇA, A. P. Design Thinking como percurso metodológico para construção de produto educacional: uma experiência no mestrado profissional na área de ensino. Educitec-Revista de Estudos e Pesquisas sobre Ensino Tecnológico, Manaus, v. 7, e103621-e103621, 2021.

FILATRO, A. Como preparar conteúdos para EAD. São Paulo: Saraiva Educação SA, 2018.

FILATRO, A.; CAIRO, S. Produção de Conteúdos Educacionais. São Paulo: Saraiva, 2016.

BRITO JUNIOR, O. de O.; AGUIAR, Y. P. C.; DE MOURA, H. P. Taxonomia para Avaliação de Recursos Digitais de Aprendizagem–TARDA–Versão 2.0. Revista Ibérica de Sistemas e Tecnologias de Informação, [S. l.], n. 42, p. 120-135, 2021. Disponível em: https://scielo.pt/pdf/rist/n42/1646-9895-rist-42-120.pdf Acesso em: 02 out. 2024

KAPLÚN, G. Material Educativo: a experiência do aprendizado. Comunicação e Educação, São Paulo, v. 27, p. 46-60, 2003.

LEITE, P. de S. C. Proposta de avaliação coletiva de materiais educativos em mestrados profissionais na área de ensino. Campo Abierto,[S. l.], v. 38, n. 2, p. 185-198, 2019. Disponível em: https://dehesa.unex.es/bitstream/10662/10066/1/0213-9529_38_2_185.pdf. Acesso em: 02 out. 2024

MENDONÇA, A. P. et al. O que contém e o que está contido em um Processo/Produto Educacional?: Reflexões sobre um conjunto de ações demandadas para Programas de Pós-Graduação na Área de Ensino. Educitec-Revista de Estudos e Pesquisas sobre Ensino Tecnológico, Manaus, v. 8, p. e211422-e211422, 2022.

MOREIRA, M. A. Los medios de enseñanza: conceptualización y tipología. Web de Tecnología Educativa. Universidad La Laguna, 2010. Disponível em: https://cmapspublic3.ihmc.us/rid=1K3790S11-6Y1FXR-TVJ/medios%20de%20ense%C3%B1anza_area.pdf. Acesso em: 20 set. 2024

RIZZATTI, I. M. et al. Os produtos e processos educacionais dos programas de pós-graduação profissionais: proposições de um grupo de colaboradores. ACTIO: docência em ciências, [S. l.], v. 5, n. 2, p. 1-17, 2020. Disponível em: https://periodicos.utfpr.edu.br/actio/article/view/12657. Acesso em: 15 set. 2024

ROCHA, S. L. et.al. Validação de produtos educacionais em ensino em saúde. 2021. Disponível em: https://educapes.capes.gov.br/bitstream/capes/739793/5/VALIDA%C3%87%C3%83O%20DE%20PRODUTOS%20EDUCACIONAIS.pdf. Acesso em: 25 out. 2024

SOUZA, A. C. C. de; MOREIRA, T. M. M.; BORGES, J. W. P. Desenvolvimento de instrumento para validar aparência de tecnologia educacional em saúde. Revista Brasileira de Enfermagem, [S. l.], v. 73, e20190559, 2020. Disponível em: https://www.scielo.br/j/reben/a/j4nNFSCVRjLFkTfXYBkLWgk/?format=pdf&lang=pt. Acesso em: 16 out. 2024.

SOUZA, C. S.; TURRINI, R. N. T.; POVEDA, V. de B. Tradução e adaptação do instrumento “suitability assessment of materials” (SAM) para o português. Revista de Enfermagem UFPE on Line, Recife, v. 9, n. 5, p. 7854-7861, 2015. Disponível em: https://repositorio.usp.br/item/002721301. Acesso em: 16 out. 2024.