Web Log Analysis of a Digital Game-Based Learning System in Vocabulary Enrichment for College Students

This study examines the gameplay patterns of college students in a web-based digital game-based learning system using various data mining techniques such as clustering and classification and usability evaluation. The objectives of the study are to design and develop a web-based digital game-based learning system using an input-process-output model game design which includes: fantasy, rules/goals, sensory stimuli, challenge, mystery and control; a knowledge-paced strategy with easy, average and difficult levels; and, the game would also collect attributes on: score, hint used, hint time, answering time, tries or repetitions, and pre-test and post-test to get the learning gain of students. In evaluating the effectiveness of the game, the game logs are collected from the database of the digital game-based learning system and were analyzed and described using K-Means algorithm, the game play patterns and performance model of students was produced using Naïve Bayes and J48 algorithms, and a usability evaluation was also performed to measure the user’s experience when interacting with the game. The study was conducted in a private college wherein students who have taken at least one language course participated as respondents. Four clusters of students are identified: no gain, low gain, average gain, and high gain. The findings show that in the easy level, the attribute hint and hint time were maximized resulting to an improvement in learning gain; in the average level, the number of tries was maximized thus, resulting to a higher learning gain; and lastly, for the difficult level, the most essential attribute was the time spent on a hint which gave the students an insight of what the base word means. A model was produced and used to classify the gameplay patterns of the students using Naïve Bayes and J48 algorithm with J48 Algorithm having a higher accuracy and kappa rating than in all game levels of difficulty.


I. INTRODUCTION
In this generation, the internet has become one of the main sources for gathering information, especially in terms of education. It has helped mostly students, in their search for needed information with only just a click of a button which provides easier and faster access compared to the traditional Manuscript  way of finding information through books. And with the rapid evolution of technology [1], this generation is less motivated to study with the use of the traditional way of teaching and learning. For this reason, various studies have proven that Digital Game-based Learning is the key to enable the students' engagement in learning while having fun.
Digital Game-Based Learning System (DGBL) is an effective alternative method for teaching and one of its advantages over traditional instruction is the motivation it gives the user, especially when the subjects are too "monotonous" for the students. Educators support DGBL as a learning medium, because the students enjoy and at the same time, have fun while learning when they are playing games [2]. Designing a DGBL that would provide learning and fun at the same time is challenging. A DGBL should be able to incorporate a game design model which comprises of the following game characteristics: (1) fantasy involves game activities that are out of the context or separate from real life; (2) rules/goals describe the goal structure of the game; (3) sensory stimuli refer to a sensation of the user; (4) challenge involves specific goals; (5) mystery is an external feature of the game which evokes curiosity in the individual; and (6) control refers to the ability to regulate, direct, or command something [3].
Recent trends show that the use of Educational Data Mining (EDM) in the design of the digital game-based learning can be further improved. Educational Data Mining is the area of scientific inquiry centered on the development of methods for making discoveries within the unique kinds of data that come from educational settings, and using those methods to better understand students and the settings which they learn in [4]. The game logs or data gathered using EDM was used to develop new tools for discovering learning patterns and better improve the learning experience of the students. The data obtained from the game logs can be further analyzed with the help of Machine Learning. Machine learning is one of the fields in computer science that makes use of algorithms in making predictions from a particular set of data. In this process, what it does to feed data into the algorithm, and the results are the predictions that might possibly happen [5].
The objectives of this study were to design and develop a Web-Based digital game-based learning system based on an Input-Process-Outcome Game Model [3]. Several means of analyzing the effectiveness were performed: (1) describe the game play patterns of the web logs generated from the web-based digital game-based learning system with the use of K-Means algorithm; (2) and produce a model that classifies the game play patterns of the students in a web-based digital game-based learning system using the Naïve Bayes and J48 Algorithms; and, (3) user acceptance Web Log Analysis of a Digital Game-Based Learning System in Vocabulary Enrichment for College Students Ellenita R. Red, Aira Jessica B. Corpuz, Genrev C. Arambulo, and Gabriel G. Delgado testing to get the satisfaction of students in the game. The study was limited to students from Malayan Colleges of Laguna who are considered to be graduating or have at least taken an English for the Workplace course. The nature of the system restricted the subject to English. There was no preference to the gender of the students.

II. METHODS
The study was conducted in a private academic institution which offers a course in English for the Workplace in their programs since the game's domain is vocabulary enrichment to help the students in the preparation for on-the-job training. A sample size of 72 students out of 284 or 25% students who are about to graduate participated in the experiment.
The students play the developed digital game-based learning system entitled, "Web-based Word Infection" which is a web-based version of the DGBL system. The Word Infection DGBL for vocabulary enrichment has gone through several versions in the past years, it first started as mobile version [6], PC based version, and its recent version which was made LAN based with Dashboard for Teachers for monitoring of students' activities [7].
The study used a simple input-process-output framework as shown in Fig. 1. The input phase consists of inputs such as content, strategies, game model, game attributes, and usability evaluation. The input phase would need the following: (1) content which is the list of vocabulary words; (2) linear sequencing and knowledge-paced strategies to provide students an option when they are assessed by the game to be in the advanced level. Linear Sequencing Strategy was used in the previous versions of Word Infection, in this strategy; the student will only be able to progress through the game sequentially, from one level to another. Consequently, the learner can only go forward one way. This was found to be effective to students who acquired a low score for pre-test. However, there are students with advanced knowledge, hence; Knowledge-Paced Strategy lets the student skip levels depending on the student's score in the pre-test of each difficulty. If the student gets a perfect score in the pre-test, the student has the opportunity to skip to the next level of difficulty. This is to make the game more challenging and more suitable for the student; (3) The Input-Process-Outcome Game Model was utilized in the game design to ensure the completeness of game elements. Input refers to the instructional content and game characteristics which are divided into six categories or dimensions such as Fantasy, Rules/Goals, Sensory Stimuli, Challenge, Mystery and Control. Fig. 2-Fig. 5 are screenshots of the digital game-based learning system exhibiting the game elements that have been incorporated.
In the current study, the Web-based Word infection Version has three levels of difficulty: easy, average and difficult. Also, the theme of the game is in a workplace setting as shown in Fig. 2: job search, job application and job interview were considered as the different stages that one would go through when applying for a job so the vocabulary appropriate for the situation were also identified and validated by language experts. As shown in Fig. 3, the main character of the game is preparing for a job application. In this particular scenario, a story is also being displayed on the screen which is an explanation of what the character is going through and the words which are underlined will be used in the drills that they will take after the story. Meanwhile, the screenshot shown in Fig. 4 is the average level of the game which shows a multiple type of test. Also, it includes a hint which provides help to the student using the game. It provides sample statement using the word that is currently being displayed.  In Fig. 5, the game play is explained and how to earn points for every correct answer. This was implemented so that students would be aware of the challenges that they are about to go through and how they can finish the game successfully.
(4) educational data mining attributes for log analysis consisting of nine attributes were used in order to create a dataset, namely: a) score percentage which refers to percentage of total score per difficulty; b) total hint refers to total number of hints used in the game of the student; c) total hint time refers to the total time a student spent on a hint; d) average answer time refers to the average time a student takes to answer a question, e) tries refer to the total number of tries a student would take the test per difficulty; f) learning gain (pre-test and post-test have been embedded in the game so students would have to take the pre-test after registration and will take the post-test after going through all the modules of the game) is a derived attribute based on the pre-test and post-test score of the student.
(5) satisfaction on the game through usability evaluation which includes five categories: (a) aesthetic refers to the look and feel of the game; (b) functionality refers to how the game supposed to work; (c) usability refers to the user friendliness of the game; (d) satisfaction on the content of the game; and, (e) web behavior refers to the accessibility of the web-based digital game-based learning system. Likert scale from 1 to 3 was utilized and interpreted as 1-Disagree, 2-Neutral, 3-Agree.
The process phase involves generation or collection of logs from the digital game-based learning system. After the students have used the game, dataset has been extracted and it has undergone data mining process which includes (a) pre-processing or cleaning the data; (b) application of clustering to describe the game logs; and (c) performing classification to describe the learning model of students' performance. The collected logs of the selected attributes were analyzed using Rapidminer, a machine learning software, which is able to perform clustering to describe the dataset such as K-means and classification algorithms such as J48 and Naïve Bayes to generate the learning performance model of students which is the expected output of the study.

A. Clustering Analysis
After data cleaning, K-means clustering algorithm was applied to the data set using a machine learning tool, RapidMiner. The student performance value results were clustered per level. The z-scores of the attributes: total score percentage, average answer time, total hint used, total hint time, total tries, and learning gain were calculated. The calculated z-scores were used as input variables to RapidMiner.
Four clusters were used to represent the following: high learning gain, average learning gain, low learning gain, and no learning gain. As shown in Fig. 6, Cluster 0 (BLUE) represents 31% of students which does not utilize the game features such as hint which indicates that the students in Cluster 0 are already proficient in the vocabulary used in the DGBL as indicated also in no learning gain.
Cluster 1 (GREEN) represents 24% of students who got the highest total score percentage, highest hint usage, and highest hint time and yet they have low learning gain which indicates that have been too drained to answer the post-test or gamed the system thus leading to a low learning gain.
In Cluster 2 (YELLOW), is the biggest cluster with 39% of the total participants. They got the second lowest score in total score percentage and average answering time. They also utilized the hint feature of the game longer which implies they took the game seriously yielding an average learning gain.
The last cluster, Cluster 3 (RED) represents only 6% of the students. Despite having the lowest total score percentage, these students also have the highest answering time, number of attempts, and thus learning gain. They may not have used the hint feature of the game; however, they took time in answering the game and patiently repeated the game until they were familiarized with the given words thus yielding a high gain score for learning gain.  In Fig. 7, Cluster 0 (BLUE) represents 18% of students who got the lowest total score percentage and number of tries; International Journal of Machine Learning and Computing, Vol. 10, No. 1, January 2020 however, they answered the test carefully and uses hint wisely which led to a low learning gain. Cluster 1 (GREEN----) is 10% of the total participants who score the highest score percentage, number of tries, and high learning gain. The students in this cluster also had the lowest score for answer time and hint time. Given that the students in this cluster scored the highest number of tries among the clusters, the students may have gamed the system but managed to familiarize themselves with the content thus leading them to score the highest learning gain among the four clusters. Cluster 2 (YELLOW) consists of 34% of students that scored the highest hint usage; however, these students scored low in total percentage score. This indicates that the students in this cluster took advantage of the hints leading them to an average learning gain. Finally, Cluster 3 (RED) which is 38% of students have high total score percentage, low average answer time, low hint usage, low hint time, and low number of attempts. These students also have the lowest learning gain which indicates that the students are already proficient and have advanced skills in vocabulary enrichment.
In Fig. 8, Cluster 0 (BLUE) represents 27% of the students who made use of the hints and taking their time in answering the test which shows the eagerness to learn thus yielding an average learning gain. In Cluster 1 (GREEN) which is 2% of the total students, they have not utilized hints and have not repeated tries resulting to lowest score percentage and no learning gain. This is an indication that these students gamed the system. The third cluster, Cluster 2 (YELLOW) which is composed of 55% of students that are taking their time to answer has yielded a good score percentage. This cluster displayed proficiency by finishing the game with a considerably low number of tries thus yielding a low learning gain. The last cluster, Cluster 3 (RED) which consisted of 16% of the total students got the highest score percentage and have used hints and tries to their advantage which eventually leads to the highest amount of learning gain among the four clusters.

B. Classification Analysis
The set parameters used for classification are: 10-fold cross validation for Naïve Bayes algorithm, J48 Decision Tree Algorithm was processed with a confidence level of 0.25 and a minimal leaf size of four (4). The interpretation of the results of the algorithms used in the study is by Landis and Koch [8] for kappa statistics on inter-rater agreement or reliability, as shown in Table I. The attributes that were used as inputs in the classification process are as follows: score percentage, total hint, total hint time, average answer time, tries, learning gain's verbal interpretation.  Table II shows the results of the accuracy and kappa statistics of J48 and Naïve Bayes for the easy, average, and difficult levels. In the easy level, J48 shows an accuracy of 100% and kappa statistic-rating of 1.0 while the results for Naïve Bayes algorithm shows an accuracy of 88.67% and kappa statistic rating of 0.89 which is still reliable having almost-perfect agreement based on Table I. In the Average level, J48 had a better accuracy with 59.05% and kappa statistic rating of 0.268 over the Naïve Bayes algorithm although both are in the fair agreement. Finally, the difficult level, J48 had a better accuracy and kappa statistic rating having 87.14% accuracy and 0.791 kappa statistic rating over 86.90% accuracy and 0.788 kappa statistic for the Naïve Bayes algorithm.
Even by a small amount, for this study, J48 decision algorithm shows better reliability than the Naïve Bayes algorithm and the easy level and difficult level produced reliable models for the performance of the students with hint as the attribute that is important in giving the students help in the process of answering all the drills in the web-based digital game-base-d learning system.

C. Usability Evaluation
The college students who participated in playing the game answered a usability questionnaire comprised of five criteria: usability, aesthetics, content, feedback and web evaluation as shown in Tables III to VII. Also, questions about the integration of the game theory elements such as rules, control, sensory stimuli, fantasy, challenge, and mystery were mapped in the usability testing. In usability evaluation as shown in Table III, control in the decision-making and actions got the lowest percentage of Cluster 0 71% while rules were found to be clearly and easily understood. This indicates that the game has provided similar rules which can be commonly found in other computer games and unlike other games, students have the feeling of not being quite in control of the game environment. The results of the evaluation of aesthetics being displayed on Table IV shows that fantasy was not fully realized by participants with only 74%; however, the sensory stimuli were found to be effective. This indic-ates that the senses were satisfied but not quite on making the students feel the fantasy brought by the game.
As shown on Table V, the content was found to be helpful and just right for each level of difficulty with-almost half having a hard time in understanding the questions specifically in the difficult level of the game since the students had limited hint thus a rating of 53%; however, this is the goal for the difficult level so that they will be able to take the game seriously and think thoroughly before answering the questions on the game. Participants are not quite convinced on how the sense of mystery has been integrated in the game since only 48% percent of them agreed in the statement; however, participants are challenged and the feedback after each drill was considered helpful as displayed on Table VI.  Navigation was easy; however, the web page's responsiveness and interface have to be improved with both having received an agreement of 68% and 67% respectively as can be seen on Table VII. This indicates that the selected text and graphics has to be further improved.

IV. CONCLUSIONS
There are three evaluation techniques that were employed to test the effectiveness of the design of the web-based digital game-based learning system: (1) In order to evaluate the game theory that was integrated in the design of the game, a usability evaluation using usability, aesthetics, content, feedback, and web behaviour. The results of the usability evaluation show that the design of the developed Web-based Digital Game-Based Learning System was able to incorporate the necessary characteristics of the game theory such as content, challenge, and stimuli; (2) In describing the game patterns and performance of students, four clusters were discovered per level of difficulty: no gain, low gain, average gain, and high gain. In the easy level of the game, cluster of students who took time in answering the game and repeated the game was quite successful and having the highest learning gain. In the average level of the game, cluster of students with the greatest number of tries or repetitions had the highest learning gain, and in the difficult level of the game, cluster of students who used hints and repeated the drills use it to their advantage which eventually leads to the highest amount of learning gain; and, (3) In classification, the attributes that are most relevant in student performance are total hint and number of tries. It shows that students who use hint, as well as those who have a high number of attempts resulted in high learning gain. Students who repeated a level has a chance of being familiarized with the set of questions that appeared in the game thus, resulting to a high learning gain.

V. RECOMMENDATIONS
The study recommends further improvements in the terms of game design and methods. In terms of game content, providing topics and a setting that focuses more on the workplace rather than in preparation for the workplace is recommended. In this w-ay, students will also be able to familiarize themselves with the basic jargon and processes used in the workplace. Also, this could provide students with a larger variation of scenes and views about a particular position in the workplace, thus, promoting motivation for the students as they explore each setting. For the data mining part, a larger population, may it be from the same locale or a different locale is recommended, to be able to explore the wide range of gameplay patterns and performance of students. With this, more valuable and educational models may be produced. Furthermore, a pedagogical agent could boost the game's effectiveness. A pedagogical agent as animated characters that enable and promote learning in computer-based learning environments [9] through incorporation of agent provided hints, feedback, encouragement and motivation to the students [7]. Lastly, future studies can also cover Affect Mining and incorporate it with the gameplay data mining that will be collected. This improvement could be of help in providing correlation to respondent's emotional and behavioral state with his/her performance. Affective computing as concerned with the synthesis of affective states which include emotions, gestures and dynamic interaction at a short period of given time [10].
Copyright © 2020 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0). Ellenita R. Red is an associate professor in the College of Computer and Information Science at Malayan Colleges Laguna, Philippines. She has presented her research works in different international conferences in Thailand, Singapore, Malaysia, Taiwan, Japan and the Philippines and was able to publish several international journal and book publications in the past five years on educational data mining, eLearning, educational technology, and business information systems. She finished BS master in computer engineering from the College Teaching, master in information technology and PhD in information technology management. She is currently involved in educational data mining and development of digital game-based learning system for vocabulary enhancement using different platforms from mobile, PC, and online use.

Aira Jessica B. Corpuz is a graduate of Computer
Science of Malayan Colleges Laguna (MCL) in the College of Computer and Information Science (CCIS). She was born on April 16, 1995 and is currently residing in Santa Rosa, Laguna. She is a persistent student and has the initiative to lead a team. When she was a student, she always makes sure that she finishes the task at hand way ahead of the deadline, and in times of encountering hardships, she always finds ways to learn and excel to the extent of her abilities. Her research interests are machine learning, data mining, digital game-based learning system. She currently works as Software Engineer which was her dream job. . His research interests are machine learning, data mining, digital game-based learning system. He is passionate in developing a game-based learning system wherein he spent sleepless nights just to make sure that the prototype is perfectly running before its implementation. He is currently connected with a leading carrier that supplies global coverage. His tasks are as ABAP developer, support, and QA tester.
Gabriel G. Delgado is a graduate of Computer Science of Malayan Colleges Laguna (MCL) in the College of Computer and Information Science (CCIS). He was born on January 17, 1997 and currently residing in Calamba, Laguna. He makes sure that every task assigned to him is done and never substandard. In times of hardships, he is always willing to break out of his comfort zone to do better and learn from every experience and everything he encounters. He is currently working as a junior research and development engineer.