Bulletin of the Technical Committee on Learning Technology (ISSN: 2306-0212)
There are many Learning Management Systems (LMS) tools like Blackboard, WebCT, EvalTool, Moodle, etc. to support teaching and learning that have produced remarkable results for teachers, students and higher education institutions. There is a remarkable increase in the use of these LMS tools by universities around the globe. Islamic University of Medina (IUM), Saudi Arabia uses two LMS, namely Blackboard and EvalTools, as an online instructional environment. This research was conducted in session 2019-20 to examine the perception of teachers about the effect of EvalTools LMS on teaching and learning in IUM. A five-point Likert scale survey questionnaire that consisted of 30 items was developed to measure the perception of teachers through relevant variables such as efficiency, effectiveness, usage and performance. The findings reveal that most teachers have a positive perception that EvalTools LMS increases the efficiency, effectiveness and performance of the users, while a slight majority of teachers have a negative perception about the ease of usage of this tool.
Keywords: Learning Management System (LMS), eLearning, EvalTools, social media.
In this rapidly changing world driven by globalization, knowledge economy and the advancement of ICT, there is need to produce qualified and skilled professionals. Now, employers hire university graduates who not only have knowledge, but also have the skills to be productive and effective in the workplace. To meet these challenges, there is a shift in educational strategy from the traditional teacher-centered model that stresses on lectures, tests and other methods of assessments towards the technology-supported learning that emphasizes what students know and can do. A Learning Management System (LMS) is one such technology that supports teaching, learning and assessment.
LMS evolved from the Course Management Systems (CMS) that were introduced in the 1990s. The focus of the CMS was on the contents of the courses and their delivery, while the focus of the LMS is on the requirements of the students as well as that of the teachers . These systems support e-learning through the development and access of course content to users . A number of educational institutions across the globe have developed LMSs to let students and professors collaborate beyond the confines of a traditional classroom . They have been shown to be an excellent method for encouraging students’ commitment to educational subjects such as sustainability . Nowadays, LMSs are commonly used in the education sector to create, access, manage, distribute, and retrieve course-related materials. They facilitate novel and innovative methods of teaching and learning by providing various tools for interactive learning; for instance, quizzes, assignment, online assessments, discussions, surveys, chat rooms, blogs, and wikis ,. The majority of students pursuing their degrees through full-time on-campus or distance mode have found collaborative tools of LMS such as chats, wikis and discussion boards as valuable in sharing their learning experiences, especially during group tasks ,. Now, an increased number of teachers and students are adapting to the online teaching and learning environment .
II. RESEARCH DESIGN AND METHODOLOGY
The number of HEIs in Saudi Arabia has grown tremendously in recent years. There are now 29 public universities and 11 private universities in the Kingdom, along with hundreds of community colleges and a rapidly growing number of technical and vocational training schools. Most of these universities and colleges use Blackboard and Moodle as LMS as a supporting aid for teaching and learning.
Islamic University of Medina (IUM) is using two LMS, namely Blackboard and EvalTools as online instructional environments. There are two faculties, namely the Faculty of Engineering (FE) and the Faculty of Computer & Information Systems (FCIS) at IUM that uses only EvalTools as LMS. Previous research shows that work has been conducted on the other LMSs like Moodle and Blackboard to find the impact of these LMS tools on teaching and learning, but no such study has been conducted on the Evaltools LMS. Therefore, the authors found it imperative to contribute to the present body of knowledge by conducting a study to examine the perception of teachers about the effect of the EvalTools LMS on teaching and learning in IUM. Since, EvalTools LMS is one of the recent LMS being increasingly adopted by educational institutions globally, therefore, this study attempts to explore the possibilities of adopting this LMS in Saudi Arabia for effective and efficient teaching, learning and evaluation processes with the help of technology.
EvalTools is an LMS platform developed by MAKTEAM Software, not just for day-to-day classroom activities, but also for assessing learning and teaching delivery in the context of satisfying standards/program accreditation. It is a complete and all-encompassing solution for all day-to-day learning, teaching delivery, and certification preparation. EvalTools was designed and developed in accordance with ABET (Accreditation Board for Engineering and Technology) standards to provide a tool for gathering and evaluating data regarding a program, students’ performance, and learning accomplishments. Furthermore, EvalTools® is useful in offering a method to ease the process of evaluating the assessment findings, as well as identifying the program’s strengths and weaknesses prior to ABET evaluation . It provides an online program assessment toolset that replaces Blackboard functionality for daily classroom lessons and assignment activities, as well as tracking key assignments that are automatically ported into the appropriate Faculty Course Assessment Report (FCAR) document for evaluation purposes . EvalTools® focuses on assessing the efficacy of teaching and learning by allowing faculty to access their teaching/student learning in a formative or summative way utilizing the FCAR approach. Faculty can choose assignments that are linked to a Course Outcome (CO), a Performance Indicator (PI), or a Student Outcome (SO) .
The following are some of the most important elements of EvalTools in terms of outcomes assessment :
- Lessons-a collection of lectures and lessons.
- Assignments-keeping track of assignments to ensure they are achieving course objectives.
- Grade book-keep track of your grades.
- Faculty Course Assessment Report (FCAR).
- Syllabus of the course-a framework for documenting learning activities.
- Rubric-easy to create a rubric for assessing meeting outcomes in a consistent manner.
- A unified storehouse for all artifacts acquired in class — the objective evidence folder.
- Course exit survey-monitoring the results of the end-of-semester course survey in terms of satisfying course objectives.
- Senior departure surveys, alumni surveys, and employer surveys – all configurable to meet the requirements of the university.
- Matrix of curriculum outcomes–allows for quick assessment of program strengths and deficiencies.
This research focuses on studying the perspective of teachers regarding the implementation of EvalTools LMS as they are the main implementors of this tool. However, this may be one of the limitations of this study and can be followed by further studies that will study the perspective of both teachers and students.
The data for this study were collected through a structured questionnaire formulated in English from the teachers of these two faculties. The total population of the teachers in these two faculties was 67 and the sample size was calculated with the help of a sampling calculator by SurveyMonkey . The sampling calculator gave a sample size of 40 with a confidence level of 95 % with a 10% margin of error by taking 67 as the total population. Therefore, in this study, 40 faculty members were chosen as the sample size out of a total population of 67. A simple random sampling technique was used to select the desired number of respondents. These instructors were utilizing the EvalTools LMS for the first time, and had never used any other LMS before.
The items used in the questionnaire for measuring various dimensions were extracted from the literature ,. The questionnaire had 30 items and was based on the 5-point Likert Scale, with responses ranging from “Strongly agree” to “Strongly disagree”.
Reliability of the questionnaire was measured through the Cronbach alpha test, which is the most common reliability test to measure internal consistency, i.e., reliability of the questionnaire . The result of the test is given in Table I, which is more than 0.9, reflecting the high reliability of the measuring instrument.
Moreover, it shows a high level of internal consistency with respect to the specific sample.
To check sample adequacy, Kaiser-Meyer-Olkin (KMO) and Barlett’s test were performed. Table II shows the results where sample adequacy KMO for the overall construct was 0.440 and Barlett’s test of sphericity was significant at 0.000, which indicated the suitability of data for factor analysis . To identify the relevant variables from the questionnaire; Exploratory Factor Analysis (EFA) was performed.
KMO and Bartlett’s Test
Table III presents the component matrix generated by EFA. The component matrix has four constructs/variables which were named as Efficiency, Effectiveness, Performance, and Usage .
As soon as the data was collected, it was carried out for analysis with the help of SPSS 16 and MS Excel 2016 software packages.
The results of this study are discussed below based on constructs such as Efficiency, Performance, Usage, and Effectiveness.
Efficiency is the ratio of the valuable work performed by a machine or its ability to avoid wasting materials, energy, efforts, money, and time in doing something or in producing a desired result. The effect of EvalTools on the efficiency of users was measured by the following four items:
Q3 The Course Info setup section of EvalTools (Course description, course outcomes, Course Outlines, Assessment Methods, etc.) is helpful to students.
Q5 The Assignment section of EvalTools (Home Work, Quizzes, Examinations, etc.) is helpful to me and my students.
Q8 The Grade-book section of EvalTools is helpful in managing the results of students.
Q28 Instructors must spend less time managing their courses with EvalTools.
Fig. 1 gives the response to these items. Most teachers responded highly to Q3 with a mean of 3.93; Q5 with a mean of 3.95 and Q8 with 3.95. In Q28, the mean of 2.85 indicates that the majority of teachers do not believe that using EvalTools has reduced the amount of time they spend maintaining their courses. Overall, results indicate that the majority of teachers are positive towards most of the items; hence efficiency.
Fig. 1 Responses for Efficiency.
Total responses for the Efficiency which were measured through four items i.e., Q3, Q5, Q8 and Q28 were 160 as shown in Fig. 2. Out of these 162 responses, 12 (7%) responses were for total disagree option, 12 (7%) responses were for disagree option, 30 responses (19%) responses were neutral, 69 (43%) responses were for agree option, 37 (23%) responses were for total agree option. This means that 66% of teachers have a positive perception that EvalTools increases efficiency while 24% have a negative perception, other 19% remain neutral. Thus, the majority (66%) of teachers, agree that EvalTools increases the efficiency of users.
Fig. 2. Total responses for Efficiency.
Performance is how well a person, machine, etc. does a piece of work or an activity. The perception of teachers that the performance of users is enhanced by using EvalTools was measured by the following three items:
Q10 I am able to check the performance of my students by using various tools like FCAR etc. in EvalTools throughout the semester.
Q11 Various tools like FCAR, Grade-book, etc. helped me to improve my teaching.
Q19 Quality of teaching improved using the EvalTools software.
Q20 Quality of Learning improved using the EvalTools software.
Q21 Do you think that use of EvalTools has resulted in students getting better grades (Marks) in their courses?
Fig. 3 shows the response to these items. Most teachers responded highly for the Q10 with a mean of 3.45; Q11 with a mean of 3.07; Q20 with a mean of 2.95 and Q21 with 2.67. This indicates that the majority of teachers are positive towards these items hence Performance.
Fig. 3. Responses for Performance.
Total responses for the Performance, which was measured through five items i.e., Q10, Q11, Q19, Q20 and Q21 were 200 as shown in Fig. 4. Out of these responses 19 (9%) responses were for total disagree option, 42 (21%) responses were for disagree option, 73 (36%) responses were neutral, 51 (26%) responses were for agree option, 15 (8%) responses were for strongly agree option. That means that 34% of teachers agree that EvalTools increases performance, while 30% have a negative perception, other 36% remained neutral. Although, the results are slightly positive, an almost equal number of teachers don’t agree that use of EvalTools enhances performance.
Fig. 4. Total responses for Performance.
The Usage construct was used to study the perception of teachers about the ease of use and usability of EvalTools and this construct was measured through the following fourteen items:
Q1 EvalTools is easy to use.
Q2 I have enough information about how to use EvalTools software.
Q4 I used the Lessons section of EvalTools to upload the course material like PowerPoints, videos, etc.
Q6 Students submit their assignments in my courses through the Online Assignment section of EvalTools.
Q7 I send emails to my students through the Class Email tool provided in the EvalTool.
Q9 The Attendance/ Roster section of EvalTools is used by the instructor to take care of the attendance of students.
Q12 I use the discussion forum in the Class Tools section of EvalTools to communicate with other students and the instructor.
Q13 I used all the tools available in the EvalTools.
Q14 EvalTools is compatible with common browsers such as Firefox, Chrome, Internet Explorer, etc.
Q15 EvalTools is compatible with common hardware (pcs, mobile devices, tablets, Laptops, etc.).
Q16 EvalTools is designed in such a way that when a user makes an error, the EvalTools respond with an appropriate error message.
Q17 EvalTools is flexible to enable users to adjust settings to suit themselves, i.e., to customize the interface.
Q22 Do you use survey tool of EvalTools to give feedback about the course.
Q23 Do you think that EvalTools should have alternative Language Support such as Arabic Language Option?
Q29 EvalTools should be used in all universities.
Q30 It should be replaced with some other Learning Management Tool such as Blackboard, Moodle.
Fig. 5. Responses for Usage.
Fig. 5 shows the response of these items. Most of the teachers responded highly to Q4 with a mean of 3.97, Q1, Q2, Q14, Q15, QQ22, Q23 and Q30 have mean greater than 3. It is important to note that Q6, Q7, Q9, Q12, Q13, Q17, Q29 have mean less than 2.5. Overall, these responses reveal that the slight majority of teachers have negative perception towards these items hence Usage.
Total responses for the Usage which was measured through sixteen items were 640 as shown in Fig. 6. Out of these responses, 102 (16%) responses were for total disagree option, 128 (20%) responses were for disagree option, 186 (29%) responses were neutral, 157 (25%) responses were for agree option, 67 (10%) responses were for total agree option.
Fig. 6. Total responses for Usage.
It means that 35% of teachers have a positive perception towards ease of use and usability of EvalTools while 36% have negative perception, other 29% remain neutral. Thus, a slight majority of teachers have shown a negative perception of the ease of use and usability of EvalTools.
Effectiveness is the capability of producing a desired result or the ability to produce desired output. Effectiveness variable/construct was used to study the perception of teachers whether EvalTools is effective or not and this construct was measured through the following five items:
Q18 EvalTools has a help facility and other documentation to support users’ needs.
Q24 EvalTools help instructors perform the necessary activities of the class faster.
Q25 I can easily find the necessary information on EvalTools.
Q26 To manage courses through EvalTool is easy and quick.
Q27 EvalTool is helpful in meeting the course outcomes.
Fig. 7. Responses for Effectiveness.
Fig. 7 shows the response of these items. Most of the teachers responded highly for the Q27 with a mean of 3.58 and Q24, Q25, Q26 with a mean of nearly 3 while Q18 has mean of 2.70. This indicates that the majority of teachers have a positive perception towards these items, hence Effectiveness of EvalTools.
Fig. 8. Total responses for Effectiveness.
Total responses for the Effectiveness which were measured through five items i.e., Q18, Q24, Q25, Q26 and Q27 were 200 as shown in Fig. 8. Out of these responses, 16 (8%) responses were for total disagree option, 44 (22%) responses were for disagree option, 64 responses (32%) responses were neutral, 58 (29%) responses were for agree option, 18 (9%) responses were for total agree option. This means that 38% of teachers have a positive perception towards effectiveness while 30% have a negative perception, other 32% remain neutral. Thus, majority of teachers have shown a positive perception of the effectiveness of EvalTools.
With the increase in internet accessibility, social media has emerged as a new means of communication for people all over the world.
Social media platforms such as Facebook, Twitter, Instagram, LinkedIn, YouTube, and others have become the most popular platforms for supporting online learning, particularly in higher education. As a result of the widespread usage of social networking sites in all fields, their application in the field of education is rising. These sites play a wide range of roles in higher education. Students may communicate with their classmates, exchange files and papers, and professors can post videos and documents to their blogs and wikis. Teachers and students use social media to keep up with the latest news, technological advancements, communicate with friends, and share their thoughts, information, ideas, and media.
Some researchers strongly advise adopting these social media sites as an alternative to LMS , . However, the biggest disadvantage of social media is that it contains several distractions from studying, such as games, video clips, horoscopes, news, advertising, and so on. Teachers have little influence over social networks, since they are run outside of academic institutions. Furthermore, anybody may create a fake profile on these social media platforms, making it impossible to identify students in a huge social network group .
Although social networks have taken over student-to-student contact on LMS, educators and learners still see the benefits of LMS in facilitating student-teacher connection, particularly in terms of assessment and assistance . With the ability to manage content, development, and delivery, training management, tracking, notifications, reporting, scheduling, attendance tracking, and exam management, LMS is not only the foundation of teaching and learning initiatives, but it is also a necessity for efficient teaching and learning management.
Keeping in view the importance of LMS as a teaching and learning aid, this study set out to investigate how faculty perceive the use of EvalTools LMS. The major findings from the results shown in Table IV reveal that Efficiency tops the rank with a mean of 3.67, followed by Effectiveness, which has a mean of 3.00; Performance ranks third with a mean of 3.09, followed by Usage, having the lowest mean of 2.94.
OVERALL DESCRIPTIVE STATISTICS OF CONSTRUCTS
This implies that the majority of teachers believe that EvalTools increases efficiency, performance and effectiveness, despite their negative perceptions about the ease of use and usability of EvalTools. Most of the faculty agree that EvalTools increases efficiency by managing Home Work, Quizzes, Examinations and results, which result in time saving. Teachers can keep track of student performance by using various tools like Faculty Course Assessment Report (FCAR) and Gradebook etc. in EvalTools throughout the semester. It increases the quality of teaching and learning, hence the performance of users. Most teachers view EvalTools as an effective tool for teaching and learning as it is helpful in meeting the course outcomes by providing the above discussed tools. The results showed that a slight majority of teachers have a negative perception of usage of EvalTools LMS. They find that most of the students don’t upload their assignments through the online Assignment section of EvalTools. Most of the teachers don’t use the various tools provided in the EvalTools such as email tool, online Attendance/ Roster tool, discussion forum. The main reason for this is lack of flexibility to enable users to customize the interface. Moreover, too many main menus (14 No) and their sub-menus are confusing and overwhelming for the users. Lack of proper training can also be a factor that teachers have negative perceptions about the usage of this tool.
With advancing technology and the constantly evolving needs of modern learners, the traditional classroom-based approach is fast losing its significance and becoming less efficient. In such a setting, using LMS in education provides a sensible option for educational institutions, allowing instructors to deliver customized information, utilize numerous pedagogical methods, and engage their students far better than before. Although educational institutions can provide training to students in a variety of methods, an LMS is one of the most effective ways to conduct online training. When you use an LMS as an online education platform, you can simply link it with other platforms to encourage students and teachers to study and share their knowledge. The educational benefits of an LMS are strong enough that educational institutions all around the world are taking notice, appreciating, and implementing them.
Keeping in view the importance of the use of technology in learning and teaching pedagogy, this study was conducted to determine the perception of faculty about the technology they are using in their teaching activities, i.e., an LMS. The findings of this study are valid as it has identified important factors including Efficiency, Performance, Usage and Effectiveness that help understand the perception of users towards the Learning Management Tools (LMS) which in this case was EvalTools. These factors were validated using various validation techniques. The collected data were tested for reliability through Chronbach’s Alpha testing method. The results of this study revealed that the overall, majority of teachers have a positive perception that EvalTools LMS increases the efficiency, effectiveness, and performance of users. However, a slight majority of teachers have a negative perception regarding the ease of use of this tool. The findings of this research provide a valid contribution to the existing body of research related to LMS and provide motivation for future research. Future research could be conducted to study the perception of students towards the EvalTools LMS. Further studies can be conducted at other national and international universities that use the LMS.
The researchers wish to extend their sincere gratitude to the Deanship of Scientific Research at the Islamic University of Madinah for the support provided to the Post-Publishing Program.
 S. Iqbal and I. A. Qureshi, “Learning Management Systems (LMS): Inside Matters,” Information Management and Business Review, vol. 3, no. 4, pp. 206-216, 2011.
 T. Martín-blas and A. Serrano-fernández, “The role of new technologies in the learning process: Moodle as a teaching tool in Physics,” Computers & Education, vol. 52, no. 1, p.p 35–44, 2009. DOI: 10.1016/j.compedu.2008.06.005.
 E. Mishchenko, “LMS Technical Analysis: problems of choice,” in E3S Web Conf., Voronezh, Russia, 2021. DOI: 10.1051/e3sconf/202124407005.
 U. Alturki and A. Aldraiweesh, “Application of Learning Management System (LMS) during the COVID-19 Pandemic: A Sustainable Acceptance Model of the Expansion Technology Approach,” Sustainability, vol. 13, pp. 1-16, 2021. DOI: 10.3390/su131910991.
 C. Cigdemoglu, H. O. Arslan, and H. Akay, “A phenomenological study of instructors’ experiences on an open source learning management system,” in Procedia – Social and Behavioral Sciences, vol. 28, pp. 790-795, 2011. DOI: 10.1016/j.sbspro.2011.11.144.
 S. Lonn and S. D. Teasley, “Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems,” Computers & Education, vol. 53, no. 3, p. 686–694, 2009. DOI: 10.1016/j.compedu.2009.04.008.
 M. Akram and A. M. Khan, “Exploring E-Leadership Practices in Implementing Learning Management System in Private Schools of Pakistan,” Pakistan Journal of Distance and Online Learning, vol. 6, no. 1, pp. 43–64, 2020.
 A. Heirdsfield, S. Walker, M. Tambyah, and D. Beutel, “Blackboard as an online learning environment: What do teacher education students and staff think?,” Australian Journal of Teacher Education, vol. 36, no. 7, p.p 1–16, 2011.
 D. A. Falvo and B. F. Johnson, “The use of learning management systems in the United States,” TechTrends, vol. 51, no. 2, pp. 40-45, 2007. DOI: 10.1007/s11528-007-0025-9.
 H. Jenkins, K. Clinton, R. Purushotoma, A. Robinson, and M. Weigel, Confronting the challenges of participatory culture: Media education for the 21st century, Chicago: The MacArthur Foundation, 2009.
 S. Wheeler, P. Yeomans and D. Wheeler, “The good, the bad and the wiki: Evaluating student generated content as a collaborative learning tool,” British Journal of Educational Technology, vol. 39, no. 6, pp. 987-995, 2008. DOI: 10.1111/j.1467-8535.2007.00799.x.
 P. Kaewsaiha, “LMS VS Social Network: Content analysis with the social presence theory approach,” The EUrASEANs: Journal on Global Socio-Economic Dynamics, vol. 5, no. 24, pp. 28-37, 2020.
 P. Kaewsaiha, “Usability of the learning management system and the choices of alternative,” in International Conference on Education, Psychology, and Social Sciences (ICEPS), 2019, pp. 252-259.
 M. Eltayeb, M. Fong, and O. Soysal, “Work in progress: Engaging faculty for program improvement via EvalTools®: A new software model,” in 2012 Frontiers in Education Conference Proceedings, 2012. DOI: 10.1109/FIE.2012.6462443.
 J. K. Estell, J. S. Yoder, B. B. Morrison, and F. Mak, “Improving upon best practices: FCAR 2.0,” in 119th ASEE Annual Conference & Exposition, San Antonio, 2012, pp. 1-47, Available: https://peer.asee.org/improving-upon-best-practices-fcar-2-0
 “Sample size calculator,” Surveymonkey.com, available: https://www.surveymonkey.com/mp/sample-size-calculator/?ut_source=help_center (Accessed Dec. 2, 2017).
 D. Bonett and T. Wright, “Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning,” Journal of Organizational Behavior, vol. 36, no. 1, pp. 3-15, 2014. DOI: 10.1002/job.1960.
 J. F. Hair, W. C. Black, B. J. Babin, and R. E. Anderson, Multivariate Data Analysis, 7th ed., New Jersey: Prentice Hall, 2009.
 E. Galy, C. Downey, and J. Johnson, “The Effect of Using E-Learning Tools in Online and Campus-based Classrooms on Student Performance,” Journal of Information Technology Education: Research, vol. 10, no.1, pp. 209-230, 2011.
 G. Naveh, D. Tubin, and N. Pliskin, “Student LMS use and satisfaction in academic institutions: The organizational perspective,” The Internet and Higher Education, vol. 13, no. 3, pp. 127-133, 2010. DOI: 10.1016/j.iheduc.2010.02.004.
 S. Y. Park, “An Analysis of the Technology Acceptance Model in Understanding University Students’ Behavioral Intention to Use e-Learning,” Journal of Educational Technology & Society, vol. 12, no. 3, pp. 150-162, 2009.
 Makteam, “EvalTools IIS – LMS,” Makteam.com. https://makteam.com/evaltools-iis-lms (Accessed March 21, 2022).
Rafi Ahmad Khan
is working as a faculty member at the Dept. of Management Studies, University of Kashmir. He has Ph. D. in Business Intelligence from University of Kashmir. He has published more than 50 research papers in international journal and have more than 30 presentation papers. He is editorial board member /reviewer of several international journals and is member of CSTA, UACEE, IEDRC, IACSIT and IAENG. His current research interests include Data Analytics, Business Intelligence, Knowledge Management and Machine learning.
is Associate Professor of Information Technology since July 2018 in Faculty of Computer and Information Systems, Islamic University of Madinah. He completed his Masters in Computer Sciences from Iqra University, Lahore, Pakistan. He obtained his MSc Telecommunication Technology in 2007. In 2007, he joined Geotechnical Group, Department of Engineering, and University of Cambridge as Research Ass. (2007- 2009). In 2009, he was awarded a PhD (2009- 2012) scholarship from the Lancaster University, UK and he awarded PhD in 2012.