Introduction

College courses are a stage that is often a tough test for students, as highlighted by Dewi (2020). This challenge is particularly felt in core courses such as soil mechanics, where students are faced with understanding complex and intricate concepts. A deep understanding of such material is not only key to academic success, but also reflects students' ability to deal with serious scientific challenges (Zaenudin et al., 2023).

In order to improve the evaluation process and create a more in-depth learning experience, question makers continue to develop more rigorous assessment methods (Arif, 2016). One approach used is through the preparation of multiple choice questions. However, not limited to assessing knowledge, the multiple choice questions are designed to test students' analytical intelligence, as revealed by Maryani and Martaningsih (2020). Thus, improving the quality of this evaluation is expected to provide a more holistic picture of students' ability to master the material (Fauzan, 2019). As well as encouraging the development of analytical skills that are essential for dealing with the complexity of science in the real world (Maharani, 2020).

Carefully designed multiple choice questions have the potential to fool students and expose weaknesses in understanding soil mechanics concepts (Wahyuni, 2019). There is a tendency that students not only choose similar but incorrect formulas, but also often make mistakes in calculations, even though the chosen formula is correct. This creates an interesting dynamic where students can produce wrong answers even though they have chosen the correct approach.

In this context, the main focus is not only to detect the correct answer, but also to identify students' problem-solving strategies and present challenges that can stimulate critical thinking. Therefore, educators need to set strategies in the preparation of multiple choice for soil mechanics courses, highlighting that students are not only often wrong in choosing formulas (Rohim, 2019). But students can also make calculation errors despite using the right formula (Anugrahana, 2020).

The discussion in this study covers the concept of developing questions that create situations where students may be motivated to choose the wrong answer. In addition, strategies for presenting challenging multiple choice questions without compromising fairness in assessment will be discussed. Thus, educators can play a role in motivating students to hone their analytical skills, deepen their understanding of soil mechanics, and improve accuracy in performing calculations (Hapsari et al., 2021). The purpose of this research is to create a unique multiple-choice question and determine the level of difficulty of the question, the differential power of the question, descriptive statistics, validity, reliability, and item response. In addition, it is also to determine the suitability between the tendency of choosing answers on multiple choice questions and predictions in making questions.

Research methodology

This research was conducted on the questions of the final exam of the odd semester of the 2023/2024 course. The topic studied was soil mechanics.

Research design

This study is a research and development (R&D) study. The objective of this research was to develop multiple-choice questions that challenge.

Investigation procedure

Developmental research begins with identification of needs, followed by planning, development of initial questions, expert validation, pilot testing, analysis of pilot test data, interviews, and question review. This research was limited to a single trial, the pilot trial. Question grids, questions, validation forms, and pilot forms were used in this study. The questionnaires were designed according to the specified strategic objectives.

Data

Data obtained in the form of question instrument validation data, pilot results data and interview results.

Data Analysis

Several analyses, both qualitative and quantitative, are carried out in this study. The first analysis is expert analysis of the instrument validator's data, question difficulty analysis, question power difference analysis, descriptive statistics, validity analysis, reliability analysis, and item response analysis.Análisis de expertos. Contiene una revisión de los resultados de la validación por expertos para obtener opiniones sobre la adecuación de las preguntas al plan de estudios y el nivel de dificultad deseado.

• Analysis of the difficulty of the questions. Contains the calculation of the level of difficulty of the question by means of the difficulty index and the percentage of students who have answered correctly.

• Question power difference analysis. Evaluate the differentiating power of each question to ensure variation in student performance.

• Descriptive statistics. Contains summary descriptive statistics, such as mean, median and standard deviation, to provide an overview of the overall test results.

• Validity analysis. Contains the results of the validity test calculation for each item.

• Reliability analysis. The test-retest reliability calculation uses Cronbach's alpha to assess the extent to which the questions are reliable in measuring the desired concept.

• Item Response Analysis. Contains the results of the evaluation of individual student responses to each question to identify patterns of confusion or misunderstanding..

Research Activities

The test consisted of 12 multiple-choice questions and was validated by a teacher who acted as an expert validator in this study. The questions were labeled QUESTION 1 to QUESTION 12. Each question has four answer options with symbols A, B, C, D. Some explanations of each answer option in each question number can be seen in Table 1.

Table 1. Explanation of response options

Research Data

Expert Validation

The validators completed the form with a score from 1 to 5. The validators compared the question grids with the curriculum and the RPS. The form consists of four aspects of evaluation: material validity, construct validity, linguistic validity, and overall validity. The validity results can be seen in Table 1. According to the results of the validity test conducted by experts, questions 2, 3, 4, 9, 11, and 12 received very good scores from the validator, specifically a score of 5. While other questions did not achieve an average score of 5, all questions received an average score above 4. The validity results for each question can be seen in Table 2.

Table 2. Results of the expert validity test

As can be seen in Table 2 and Table 3, the validator gave the lowest score, 3. Therefore, it can be concluded that there are no questions that have poor to very poor scores in each aspect of the tests. The question that received the lowest mean was question number 8 with a score of 4.44. This question received a low score because it only scored sufficient in the aspect of "clarity of instructions and questions" and in the aspect of "Appropriateness of difficulty level of questions". Based on these results, it can be said that all questions are valid.

Table 3. Results of the expert validity analysis

Statistical analysis of results

Problem difficulty

The difficulty of the questions can be determined by calculating the percentage of students who answer each question correctly. The results of the percentage calculation can be seen in Table 4. The level of difficulty of the questions can be seen in Table 5.

Table 4. Percentage of students who answered correctly to each problem

Table 5. Criteria for problem difficulty level

• Questions with a medium level of difficulty (Questions 1-5)

The percentage of correct answers between 51% and 63% indicates a medium level of difficulty. Most students had a moderate ability to answer these questions..

• Questions with a low difficulty level (Questions 6, 7, 11, 12)

Problems 6, 7, 11 and 12 have a percentage of correct answers above 70%, indicating a low level of difficulty. Students tend to answer these questions correctly.

• Questions with a high level of difficulty (Questions 8, 9, 10)

Questions 8, 9 and 10 have a percentage of correct answers below 20%, indicating a high level of difficulty. Most of the students had difficulties in answering these questions.

Differential power of the question

The results of the differential power analysis can be found in Table 6, while the criteria for the level of differentiation can be found in Table 7.

Table 6. Results of the differential analysis

Table 7. Criteria for the level of distinctive power

• Questions with low distinguishing power

Questions 1, 2, 3, 4, 6, 6, 7, 11, and 12 have a high percentage of correct answers for both high and low achievers. These questions may be less effective in measuring the difference in ability between high and low achievers. They need to be revised or improved to challenge higher-achieving students.

• Questions with high differentiation potential

Questions 5, 8, 9, and 10 had significant differences between the highest and lowest performing students. The high power of differentiation suggests that these questions may be more effective in identifying differences in ability among students. Further review is needed to ensure that the differences are adequate for the purpose of the assessment.

• Underperformance questions

Problem 8 had a very low percentage of correct answers, even among the best students. This question should be further evaluated for formulation problems, inadequate difficulty, or ambiguous distractors.

• Problems with a downward performance trend

Questions 9 and 10 showed a low percentage correct in both groups of students. A thorough evaluation of these two questions is needed to understand the reasons for the low percentage correct and to make improvements if necessary.

Descriptive statistics

Table 8. Descriptive statistics

Analysis of the statistical description of the data in Table 8 reveals some key information about the distribution of scores for a data set with a total of 57 entries. The mean score of this data set is approximately 52.92, while the median, which is the average value of the ordered data set, is 58.33. The mode, which represents the most frequent value or category, is 66.33. The mode, which represents the most frequent value or category, is 66.67. The standard deviation, which measures the dispersion of the values of the data set with respect to the mean, is approximately 20.86. The range of values between the minimum (8.33) and maximum (83.33) scores is 75, which gives an idea of how much the scores vary in the data set. In addition, the standard error is approximately 2.76 and the 95% confidence level is approximately 5.54, which gives an idea of the extent to which the mean score can be trusted to represent the population. These data give an overall picture of the distribution of scores, taking into account centering, variation and statistical confidence in the results.

validity and reliability

Table 9. results of the analysis of the validity of the questions

The results of the question validity analysis in Table 9 show the assessment of the validity of each question of the assessment instrument. In the assessment using the R-count, where the R-count is the correlation between the actual response and the expected response, all questions obtained significant R-count values. Comparing the R-count with the R-table value, which is the critical correlation value at a given significance level, it can be concluded that most of the questions (Question 1-7, Question 10, Question 11 and Question 12) are valid. This indicates that the questions are able to measure the desired concept according to the evaluation standard. However, there were two questions (Question 8 and Question 9) that were declared invalid because the calculated R was lower than the expected value from Table R. Therefore, special attention needs to be given to reordering or revising these questions to make them more in line with the assessment objectives. The results of this item validity analysis provide critical insight into the quality and accuracy of the assessment instrument, allowing improvements to be made to increase the validity of the tool in measuring students' comprehension.

Table 10. Results of the reliability analysis of questions (Cronbach's alpha)

The results of the reliability analysis of the questions, as illustrated in Table 10 with a Cronbach's Alpha value of 0.632, indicate a high level of reliability. High reliability indicates that the questions of the assessment instrument provide good consistency in measuring the concept or capability being measured. Cronbach's Alpha values close to 1.0 indicate that the questions in the instrument are closely related to each other and that each individual's responses contribute consistently to the overall measurement.

Answer to questions

Table 11. Answers to questions

The item response analysis in Table 11 shows the pattern of student responses to each question number in the form of number of answers given to each option (A, B, C, D). In question number 1, it can be observed that most students chose option D as the correct answer, with the number of responses reaching 30. On the contrary, in questions numbers 2 and 3, most students chose option B as the correct answer, with the number of responses reaching 36. In question number 4, there was a variety of responses, but option D obtained the highest number, reaching 31 responses. Meanwhile, in questions numbers 5, 6, 7, 9, 10 and 11, the correct answer options were C, C, A, B and D, respectively, with a significant number of responses. In questions 8 and 12, the correct answers were A and B, respectively, and the number of correct answers was higher. This analysis provides insight into the comprehension patterns and difficulties that students may encounter in each question, which can serve as a basis for evaluating and improving the learning process in the future.

From the data from the interviews with several students, it was found that their choice of answers conformed to the predictions that can be seen in Table 1. In question number 1, 27 students used the incorrect formula, and 20 students calculated by subtracting the upper and lower segments of the cup weight. This error is a recurring error that has been found in several tasks. In question number 2, in question number 4, 18 students used the inverse formula, i.e., water content minus liquid limit, this is very likely to occur because basically these two quantities are identical, only the water content in question is the natural water content.

Discussion

The Soil Mechanics 1 course test in the form of 12 multiple-choice questions is made by predicting the answers that students could choose. The creation of answer options from A to D is performed with certain details, namely by changing the formula or numerical operation. The creation of the options is based on the results of the homework answers, in which many students make mistakes in the use of the formulas. Even some students continue to make mistakes in performing arithmetic operations..

Prior to testing, the test was validated by experts with an average score of 4.44 out of 5. Of the 12 questions declared valid. From the test results, the results of data analysis in the form of difficulty level of questions with low results in questions number 6, 7, 11, 12, medium difficulty level in questions number 1, 2, 3, 4, 5, high difficulty level in questions number 8, 9, 10. From the results of the differentiation data analysis, it was found that the results were poor in questions number 8 and 9, the level of differentiation was sufficient in questions number 1, 3, 6, 7, 10, 11 and 12, the level of differentiation was good in questions number 2 and 5, and the level of differentiation was very good in question number 4. In addition, from the results of descriptive statistics, it was found that the mean was 52.92; median 58.33; minimum value 8.33; maximum value 83.33 with a total of 57 data. While the reliability value is 0.632 with a high reliability category.

From the pilot test conducted and corroborated by the results of the interviews with the students who continued to answer incorrectly, it was found that students continued to use the incorrect formula as predicted by the researcher. As with question number 1, many students continue to choose option A, and some even choose option B. Students who choose option A are students who have the wrong formula, while students who choose option B are students who keep getting the calculation wrong. Even in question number 10, more students chose the wrong answer option because they chose the wrong formula.

Conclusion

From the data and discussion, conclusions are drawn to respond to the research objectives, namely.:

· The multiple-choice questions on the Soil Mechanics 1 exam were created by predicting the answers students might choose

· Of the 12 questions, there are 4 questions with a low difficulty level, 5 questions with a medium difficulty level, and 3 questions with a high difficulty level. Of the 12 questions, there are 2 questions with a low differentiation level, 7 questions with an adequate differentiation level, 2 questions with a good differentiation level, and 1 question with a very good differentiation level. Additionally, based on the results of descriptive statistics, the mean is 52.92; the median is 58.33; the minimum value is 8.33; the maximum value is 83.33 with a total of 57 data points. From the validity test, there are 2 questions that are not valid, but the reliability value is high at 0.63.

· There is a correspondence between the tendency to choose answers in multiple-choice questions and the predictions in question formulation.

References

Anugrahana, A. (2020). Analisis kesalahan matematika konsep operasi hitung bilangan bulat mahasiswa calon guru sekolah dasar. Sigma, 5(2), 91-99.

Arif, M. (2016). Pengembangan Instrumen Penilaian Mapel Sains melalui Pendekatan Keterampilan Proses Sains SD/MI. Ta'allum: Jurnal Pendidikan Islam, 4(1), 123-148

Dewi, S. N. (2020). Dampak Covid-19 terhadap pembelajaran daring di perguruan tinggi. Jurnal Pendidikan Ilmu Pengetahuan Sosial (JPIPS), 12(2), 87-93.

Fauzan, R. (2019). Pemanfaatan gamification Kahoot. it sebagai enrichment kemampuan berfikir historis mahasiswa pada mata kuliah sejarah kolonialisme Indonesia. En Prosiding Seminar Nasional Pendidikan FKIP (Vol. 2, No. 1, pp. 254-262).

Hapsari, F., Desnaranti, L. y Wahyuni, S. (2021). Peran guru dalam memotivasi belajar siswa selama kegiatan pembelajaran jarak jauh. Research and Development Journal of Education, 7(1), 193-204.

Maharani, A. (2020). Computational thinking dalam pembelajaran matematika menghadapi Era Society 5.0. Euclid, 7(2), 86-96.

Maryani, I. y Martaningsih, S. T. (2020). Motivasi guru sekolah dasar pada pelatihan pengembangan alat evaluasi berorientasi HOTS dalam pembelajaran jarak jauh di masa pandemi covid 19. In Prosiding Seminar Nasional Hasil Pengabdian kepada Masyarakat Universitas Ahmad Dahlan (Vol. 2, No. 1, pp. 789-798).

Rohim, D. C. (2019). Strategi penyusunan soal berbasis hots pada pembelajaran matematika sd. Jurnal Riset Dan Konseptual, 4(4), 436-446.

Wahyuni, Y. (2019). Analisis Tingkat Kualitas Soal Pilihan Ganda Penilaian Akhir Semester Gasal Mata Pelajaran Pendidikan Kewarganegaraan Kelas III MI NU Sholahiyah Pedawang Kecamatan Bae Kabupaten Kudus Tahun Pelajaran 2018/2019. [Tesis doctoral, IAIN KUDUS).

Zaenudin, M., Nugraha, D. y Faizah, S. (2023). Workshop penyusunan skripsi, publikasi ilmiah dan pembuatan poster ilmiah bagi mahasiswa tingkat akhir jakarta global university. Jurnal Abdi Insani, 10(3), 1543-1554.