Usability Effectiveness of the State Polytechnic of Malang’s Learning Management System as e-Learning Media

ABSTRACT


INTRODUCTION
The rapid development of information technology has a broad impact on human life, especially in education, to support a more efficient and effective learning process [1]. During the Corona Virus Disease 2019 (Covid-19) pandemic [2], the concept and mechanism of online learning were carried out to prevent and minimize the spread of the coronavirus. Covid-19 has caused social changes in society. Several new regulations or policies were created by the government, such as learning activities initially carried out directly face-to-face were made online. In Indonesia, online learning stems from the government's policy of requiring social distancing or limiting social interaction to prevent the spread of Covid-19. This policy was welcomed by issuing a circular letter from the Ministry of Education and Culture stating that learning must be done remotely from home.
President Joko Widodo also conveyed the new policy during the inauguration of the Virtual Indonesian Rectors Forum at Bogor Presidential Palace on Saturday, July 04, 2020. He mentioned, "Online lectures have become the new normal, even the next normal. I am sure there will be new normality that is more innovative and productive". Online learning can be interpreted as a system of learning activities carried out without face-to-face meetings but through the help of an Internet network. Online learning is part of electronic learning (e-Learning). E-Learning is a learning process that utilizes Information and Communication Technology (ICT) as its medium [3]. With e-Learning, students' understanding of a material does not depend on the teacher/instructor but can be obtained from electronic media [4]. E-Learning results from systematically integrating various learning components, but it still needs attention regarding the quality, learning resources, and learning interaction (engagement) across time and space. One of the online learning media used is Learning Management System (LMS), an application or software used to manage online learning, which includes several aspects, such as material, placement, management, and assessment [5].
There have been many studies on the usability evaluation of applications and systems. In terms of Usability, Open Learning LMS is still acceptable to students and functions well, with the test results showing the System Usability Scale (SUS) measurement or assessment score at 61.03 [6]. LMS as e-Learning applications or information systems also needs to be evaluated to improve its utilization, according to [6]. Similar research was also conducted on the University of Indonesia (UI) Library Website, with the results of research that the UI Library Website has yet to be well received by 2016 and 2019 students. However, there is a difference in scores between the two generations [7]. In addition, in the Puntukdoro Village Partner Village Development Program Information System research, the system design has a good level of Usability with a SUS score of 80 which is classified as Grade Scale B [8]. Lamandau Polytechnic compared several e-Learnings used with usability results that Edmodo LMS is in the Acceptable category with a SUS score of 70, while Google Classroom and Schoology are in the Almost Acceptable category with SUS scores of 67.6 and 64.4 [9]. In research on the Usability of LMS services at Institute of Informatics and Business (IIB) Darmajaya, it was found that the SUS interpretation from the user's side was said to be quite good. However, lecturers and students could have been more enthusiastic about their experience while using LMS [10].
The State Polytechnic of Malang provides LMS facilities as the primary support for online learning. However, the problems are found in its implementation, such as some lecturers and students needing to start using the LMS with its features because they are pretty complete and cause them to use other e-Learning systems. The reason for choosing this research topic is that the problem is closely related to the current education world, especially at a high education institution. With this research, we also dug into the utilization of LMS as a learning media and the influencing factors to measure the effectiveness of its utilization at the State Polytechnic of Malang. From the measurement results, we can know more about the role of information technology in supporting online learning and the benefits of using LMS as e-Learning media. We raised two research questions. First, how effective is the utilization of LMS as e-Learning media at the State Polytechnic of Malang? Second, how effective is the implementation of LMS as online learning at the State Polytechnic of Malang? We used the SUS approach to answers the questions knowing that it is one of the most widely used questionnaires in measuring the perceived ease of use of interactive systems [11]. Our hypothesis is "The level of effectiveness of online learning using LMS at the State Polytechnic of Malang is low because of the availability of other e-Learning media options that are easier to use, causing its utilization to be less attractive." The rest of the article discusses the research methodology used in this research, followed by results and analysis, and concludes with conclusions.

RESEARCH METHOD
The research method is carried out with a series of stages to observe the problem, which includes collecting references and data and applying relevant techniques to obtain the research results. We used a quantitative descriptive method to evaluate the Usability of LMS as e-Learning media. The data collection technique is a questionnaire, and the System Usability Scale (SUS) is the evaluation method. This research was conducted at the State Polytechnic of Malang, 9 Soekarno Hatta, Jatimulyo, Lowokwaru District, Malang City, East Java 65141, Indonesia. The research itself was held between February 28 to June 04, 2022.

Quantitative Descriptive
Descriptive research seeks to describe a symptom or event that occurs at present [12]. It uses numbers, starting from data collection, interpretation of the data, and the appearance of the results [13]. Based on this understanding, descriptive research is carried out by looking for information related to existing symptoms, explained by the objectives to be achieved, planning how to approach it, and collecting various kinds of data as material for making reports. In this study, we want to know about the activities during online learning using LMS media.

Data Collection
We created a questionnaire as a sheet for data collection purposes. The questionnaire is a set of questions asked to respondents to obtain data to be used or processed further [14]. In addition, they can be used COMPILER to obtain personal information such as respondents' attitudes, opinions, expectations, and desires. Ideally, all respondents are willing to fill in or rather have the motivation to complete the questions in the research questionnaire. The questionnaire is a primary data collection tool with a survey method to obtain respondents' opinions. They can be distributed to respondents in three ways, namely directly by the researcher (independently), sent by post (mail questionnaire), and sent via a computer such as electronic mail (e-mail) or using online form services. There is no specific principle, but the researcher can consider its effectiveness and efficiency in terms of whether it will be sent by post, e-mail, online form service, or directly from the researcher.
This study distributed the questionnaire to predetermined respondents using an online form service, Google Forms. Google Forms is a system with a form template that can be utilized for obtaining users' information. The only requirement to use it is to have a Google account [14]. Therefore, the predetermined respondents are at least 15 people representing each clump or department. This number is determined based on the results of Faulkner's research in 2003 which states that the number of respondents is at least 15 people to provide better results, where more problems are obtained, which is at least 90% of the population [15].

Respondent Profile
The characteristics of respondents in this study are all active students in the State Polytechnic of Malang, all majors, and several study programs. After the questionnaire was distributed for two weeks, from April 18 to May 01, 2022, it was found that 146 respondents filled out the questionnaire. The collected data is shown in Table 1.

Usability Evaluation
According to ISO 9241-210:2019, Usability is the degree to which users feel satisfied, effective, and efficient in using a product to achieve a goal. In addition, the definition of Usability, in general, is an attribute of quality used to evaluate the level of ease of an interface use. Website usability testing aims to assess user experience based on five criteria: Learnability, Efficiency, Memorability, Errors, and Satisfaction [16]. The explanation for each criterion is as follows: a. Learnability: the ease with which users can complete basic tasks the first time they encounter a product display. b. Efficiency: the speed level of users can perform tasks after they learn the product interface. c. Memorability: the ease with which users can learn and reuse a product after not using it for a while. d. Errors: the level of errors made by the user, the severity of those errors, and the ease with which users can recover from those errors. e. Satisfaction: It is related to user pleasure in using a product.
Usability evaluation assesses the extent to which users find it interactive, easy, and comfortable to use a product or system [17]. Several things need to consider in usability evaluation are as follows: a. Usability is an intrinsic, measurable property of all interactive digital technologies. b. Human-computer interaction researchers and interaction design professionals have developed evaluation methods to determine whether an interactive system or device is usable. c. If a system or device is usable, the usability evaluation method also determines its level of Usability using robust, objective, and reliable metrics. d. The human-computer interaction research and practitioner literature thoroughly documented evaluation methods and metrics.
Usability evaluation can be done using two methods, namely qualitative and quantitative. In the quantitative method, evaluation can be done using statistics. The impact of good and bad design can be easily quantified into conversions, completion rates, turnaround times, perceived satisfaction, recommendations, and sales. Usability practitioners and user researchers can quantify the benefits of their efforts to assess the Usability of products that have been created [18].

System Usability Scale (SUS)
SUS was created by John Brooke in 1986 to evaluate various types of systems or products practically. It has been tried, tested, and used for over 30 years and proven reliable for evaluating system usability. The SUS method is one of the most efficient methods to evaluate the Usability of a system or product by statistically collecting valid data and giving a clear and rational value to the website appropriately. Furthermore, compared to other test methods, it is cheaper and faster because it has fewer questions and templates that can be used immediately. It was created to address the need for simplicity and speed in evaluating systems created and can be used in various contexts and types of systems other than websites, such as operating systems, hardware, software, and applications [19].
Three benefits can be obtained by using SUS, namely: (1) it is a scale measurement that is easy to give to users; (2) its results are reliable despite using a small sample size; and (3) its results are valid, which can distinguish between systems that can be used and those that cannot be used effectively. On the other hand, there are several things to consider when using it, namely: (a) the scoring system is quite complex; (b) interference in interpreting the final score as a percentage; (c) the best way to interpret the results involves normalizing the score to produce a percentile ranking; (d) not to be used as diagnosing tool for existing problems in the system, but to classify the easiness use of the system being tested. There are five ways to interpret SUS scores: Grade, Percentiles Range, Adjective, Acceptability, and Net Promoter Score (NPS), commonly known as the SUS Score Grading Scale, as shown in Table 2 [20]. The interpretation of the value of SUS is as follows:

COMPILER
• Grade. This parameter is closely related to the percentile rank. Grade ranges from A, which indicates superior performance, to F, which indicates failing performance, with C indicating average. The grading scale is created on a curve. A more even curve will distribute the grades to fit a standard curve. Hence the normalization process used for percentiles.
• Percentiles Range. SUS scores can be converted into Percentiles Range. Large data sets of SUS scores are normalized for Percentiles Range conversion. Percentiles Range tells the reader how well the SUS score compares to others in the database. For example, the average score (at the 50 th percentile) is 68. It means SUS scores above 68 are above the average, and below 68 are below the average. For example, a SUS score of 75 is at the 73 rd percentile (a score better than 73% of the scores in the database), while a score of 52 is at the 15 th percentile (a score worse than 85% of the scores in the database).
• Adjective. It is a measurement by building ideas using words rather than numbers to describe an experience [21]. 100 SUS scores are associated with Adjective values. This scale contains adjectives consisting of worst imaginable, poor, OK, good, excellent, and best imaginable words that users freely associate with the Usability of a product.
• Acceptability. Another variation in using words to describe SUS is thinking about what is acceptable or not. These terms are defined when the SUS is above average or below average [21]. The acceptable value corresponds to roughly above 70 or above the average of 68, while the not acceptable value is below 50 (closely related to the designation of scores lower than 51.6 with an F value). It defines the range between 50-70 as marginally acceptable, which includes the range from C to D in the grading scale curve.
• Net Promoter Score (NPS). There is consistently a strong correlation between SUS and NPS. The average SUS score explains between 30% and 50% of the variation in users' likelihood to recommend. NPS designates three classes of recommenders based on their responses to an 11-point (0 to 10) likelihood to recommend question. Promoter scores are 9 and 10, Passive scores are 7 and 8, and Detractor scores are six and below. The Promoter is most likely to recommend the product/site/application to a friend, and Detractor is more likely to discourage than recommend. SUS scores must get an average score of at least 81 to achieve Promoter classification, which is a high standard. Detractor is associated with SUS averages of 53 and below, while Passive is the score in between (average 70), indicating the uncertainty of using this approach.

Likert Scale
In response to the difficulty of measuring character and personality traits, Likert 1932 developed a procedure for measuring attitude scales. The Likert scale is a tool in the form of a scale to measure the perceptions of a person or a team regarding responses and judgments about social phenomena [22]. The Likert scale uses a series of questions with five alternative responses: strongly agree, agree, undecided, disagree, and strongly disagree [23]. This research also uses alternative answers: very often, often, rarely, very rarely, and never. There are two forms of questions in the Likert scale, namely the positive form of questions to measure the positive scale with a score of 5; 4; 3; 2; 1, and the negative form of questions to measure the negative scale with a score of 1; 2; 3; 4; 5 [24]. Table 3 shows the scores on the Likert scale according to the explanation above.

RESULTS AND ANALYSIS
This research instrument is a questionnaire that refers to SUS indicators and a Likert scale from 1 (one) to 5 (five). This scale expresses user agreement and disagreement. After obtaining the data, the analysis was conducted using the SUS calculation method. Then the final results were compared and interpreted to the five components of Usability, namely Learnability, Efficiency, Memorability, Errors, and Satisfaction. The Learnability, Efficiency, and Memorability aspects relate to questions 1, 3, 5, 7, and 9. The Errors aspect relates to questions 2, 4, 6, 8, and 10. Then the Satisfaction aspect relates to questions 1 and 9 [25]. The SUS instruments are ten questions with a Likert scale of 1-5 as the answer, as shown in Table 3.

Usability Analysis with SUS
In this research, the SUS calculation uses a positive and negative Likert scale. The positive scale is applied to questions with odd numbers, while the negative scale is applied to questions with even numbers. After the data was collected, the data obtained from the 10 SUS questions was calculated with the following conditions. a. The respondent's answer scale is deducted by 1 point for each answer given by the respondent to questions with odd numbers (1, 3, 5, 7, 9) = respondent's answer scale -1. b. Five points are deducted by the scale of the respondent's answer for each answer given to the question with an even number (2, 4, 6, 8, 10) = 5the value of the respondent. c. After getting the results of subtracting all the questions above, all the results are summed up. d. Then the total number is multiplied by 2.5 to get a result with a maximum value of 100.

Tale 4. Question List
No. Question 1 How often do you use the LMS in a week? 2 Do you agree that LMS is more complicated than other e-Learning media? 3 Do you agree that LMS is easier to use than other e-Learning media? 4 Do you need help from other people or technicians in using the LMS? 5 Are the features available in the LMS working correctly? 6 Are there a lot of inconsistencies (mismatched websites) or bugs or errors in the LMS? 7 Do you agree that most people will quickly understand how to use an LMS? 8 Do you agree that LMS is confusing to use? 9 Do you feel confident and uninhibited while using the LMS? 10 Do you agree that before using LMS, you must familiarize yourself with it and learn many tutorials on its use?
Each respondent's scores are summed up, and then the average is calculated to produce the final SUS score. Then the results are interpreted according to the SUS Value Grading Scale (Table 2) [18]. Finally, respondents provide an answer scale in evaluating Usability or Usability with the SUS method on the State Polytechnic of Malang's LMS with the recapitulation results shown in Table 5.  Respondent  1  2  3  4  5  1  1  5  40  50  50  146  2  9  23  50  52  12  146  3  0  10  27  81  28  146  4  3  18  22  67  36  146  5  1  13  26  93  13  146  6  15  35  62  31  3  146  7  0  12  38  70  26  146  8  1  23  28  65  29  146  9  7  25  62  46  6  146  10  8  49  32  39  18  146 Furthermore, according to the answer data, the SUS calculation is carried out with the provisions of options a, b, and c mentioned above. A recap of the data from the respondents' answers is obtained from the calculation results, shown in Table 6. The respondent's answer data is a recapitulation of the answer data, and then the calculation is carried out according to the SUS provisions. Each question can be explained based on the Nielsen usability component [16], related of respondents' thoughts about the State Polytechnic of Malang's LMS application.

Learnability, Efficiency, and Memorability Aspects
The answers each question from a total of 146 respondents were recapitulated to determine the percentage of each answer calculated using the following formula: Percentage answer (x) = ( ℎ ) × 100 (1) Based on Figure 1, the survey data obtained by 34.2% of respondents answered that they very often and often use LMS within a week. The data shows that students think LMS learning media is needed and can be used easily and comfortably, making them want to use it again. Based on Figure 2, 55.2% responded in the affirmative to Question 3, "Do you agree that the LMS is easy to use compared to other E-Learning media?" The data shows that most students can use the LMS easily. It is related to learnability and efficiency aspects.   Figure 3 shows that most respondents, as much as 63.7% responded in the affirmative to Question 5, namely, "Are the features available on the LMS working properly?" The data shows that most students find that the LMS' features can function so that they can use the LMS comfortably. In other words, most students agree that the LMS can be used efficiently. Figure 4 shows that most respondents, 47.9%, agreed with Question 7, "Do you agree that most people will understand how to use the LMS quickly?" The data shows that most respondents think the LMS is easy to learn from others, closely related to Learnability, Efficiency, and Memorability. It means that the LMS has a display that works efficiently and is easy to remember, making it easy for new visitors to learn it quickly without difficulty. Figure 5 shows that 42.5% of respondents responded hesitantly to Question 9: "Do you feel confident and there are no obstacles while using the LMS?" The data shows that most respondents considered that the LMS was still being used in terms of its ease and comfort when used. In addition, the LMS is also considered that there is still a possibility of an obstacle when used. It is closely related to the efficiency aspect, where the features in the LMS still cannot be ensured to work quickly, so users feel a little hassle or difficulty using it.  Based on the five figures, most respondents agreed in assessing the aspects of Learnability, Efficiency, and Memorability. It shows that most respondents can learn the LMS quickly and well (Learnability), use it efficiently even with no obstacles still in doubt (Efficiency), and remember well without the need to learn again from scratch when visiting the LMS (Memorability). Therefore, the LMS has a simple interface that can be learned quickly and efficiently but needs to be used comfortably due to barriers. Furthermore, because of this simplicity, users will have no difficulty remembering where a feature is located so that they do not have to learn it again when revisiting it.

Errors Aspect
Based on Figure 6, the survey results show that most respondents choose to disagree, namely 52 respondents (35.6%) for Question 2, "Do you agree that the LMS is complicated to use compared to other E-Learning media?" The data shows that the LMS is easy to use without or with a few errors. Based on Figure 7, the survey results show that 67 respondents (45.9%) choose to disagree with Question 4, namely, "Do you need help from other people or technicians in using the LMS?" The data shows that most respondents can learn how to use the LMS independently without the help of others. It means that errors rarely occur on the LMS, especially errors that cannot be resolved independently. Based on Figure 8, the survey results show that most respondents, 62 respondents (42.5%) chose doubtful for Question 6, namely, "Do you feel that there are many things that are inconsistent (website appearance is not harmonious) or bugs or errors on the LMS?" The data shows that the features of the LMS still need to be improved regarding the consistency of the system's functionality. Because the question is general, it needs to be sure what inconsistencies exist in the LMS.    Based on Figure 9, the survey results show that 65 respondents (44.5%) choose to disagree with Question 8, namely, "Do you agree that the LMS is confusing when used?" The data shows that most of them consider that the LMS can be used easily without confusing them so that they can avoid or prevent mistakes. Figure 10 shows the survey results of most respondents; 49 respondents (33.6%) agree with Question 10, namely, "Do you agree that before using the LMS, you need to familiarize yourself first and learn many tutorials on its use?" The data shows that most of them need time to adapt and learn to use the LMS.    Based on the five figures above, most respondents disagree with assessing Errors. However, in Question 6, "Do you feel that there are many inconsistencies (website appearance is not harmonious) as well as bugs or errors on the LMS?" most of them responded undecidedly, and Question 10, "Do you agree that before using the LMS it is necessary to familiarize yourself first and learn many tutorials on its use?" most of them responded agree. The results of the analysis show that a small percentage of respondents found or made a few mistakes and could learn to overcome them quickly from mistakes when visiting the LMS. Although the LMS website has a display and features that function correctly, users still need help with a few errors, so some users feel uncomfortable using it. Figure 11 shows that most respondents choose often and very often, namely 50 respondents (34.2%) for Question 1, namely "How often do you use the LMS in a week?" Most of them think the LMS is needed for online learning, so they want to revisit it. It means that most respondents are satisfied with using the LMS. Based on Figure 12, the survey results show that most respondents, 62 respondents (42.5%) choose doubtful for Question 9, namely, "Do you feel confident and there are no obstacles while using the LMS?" Most of the respondents still doubt the absence of obstacles and feel there is still a possibility of experiencing obstacles when using the LMS. For Question 1, namely "How often do you use the LMS in a week?" Most of the respondents responded very often and often. Then for Question 9, namely "Do you feel confident and there are no obstacles while using the LMS?" Most of the respondents responded doubtfully. Although there are slight differences in responses, it can be said that the LMS still has obstacles when used, but most respondents feel that they still need the LMS to support online learning, so they still visit and use it.    Figure 12. Recapitulation of results from Question 9.

Level of Usability Effectiveness (Usability)
Based on the results of calculating the effectiveness level of Usability using the SUS method, the results are listed in Table 7. The evaluation of the LMS usability receives a score of 54.67. Application usability is measured based on three categories: Acceptability, Grade Scale, and Adjective Rating. Based on Acceptability, the LMS is categorized as marginal, meaning it is still acceptable but minimal. At the same time, the Grade Scale assesses the application from the aspect of quality level. The evaluation results show that the LMS is at Grade Scale D or below the average. Finally, the Adjective Rating assesses the application from aspects that determine the usability rating. The evaluation results show that the LMS is categorized as OK, which means it is a good user experience even though it is at the most marginal level. Although there are several advantages or good values from the usability evaluation research of the LMS using the SUS method, when viewed from the final SUS score, it can be seen that it is below the SUS average score and classified as middle to lower. However, its Usability is still accepted by students. Good scores given by respondents include easy-to-use, easy-to-understand, integrated features, and easy-to-learn. On the other hand, bad scores are obtained from such system inconsistencies, bugs, errors, and time required to familiarize the system. In general, the LMS has low Usability according to the total scores given by respondents on each question as follows.
a. Question 1 gets a total score of 7.45. b. Question 2 gets a total score of 4.40. c. Question 3 gets a total score of 7.17. d. Question 4 gets a total score of 3.03. e. Question 5 gets a total score of 6.78. f. Question 6 gets a total score of 5.48. g. Question 7 gets a total score of 6.88. h. Question 8 has a total score of 3.32. i. Question 9 has a total score of 5.33. j. Question 10 has a total score of 4.83. From the SUS view, the State Polytechnic of Malang's LMS only achieves a score of 54.67, categorized as marginal Acceptability, Grade Scale is D, Adjective Rating is OK, and Net Promoter Score (NPS) is Detractor. Although there are some good scores, it's score is still low in general. However, in the context of Usability, students still accept it, with improvements to be made to improve it. Comparing the SUS results and the hypothesis proposed in this experiment, we found that our hypothesis is only partially accepted in the context of easiness of use and simplicity. However, other platforms of e-Learning media also offer similar capabilities, but it does not inhibit the students use it because they need to access lecture materials and other study activities that can only be accessed through it. Other e-Learning media is the reason that the State Polytechnic of Malang's LMS is less attractive except. Therefore, in conclusion our hypothesis is partially accepted. Overall, the effectiveness of usability on the State Polytechnic of Malang's LMS is still relatively low. Most of the respondents scored well in Learnability, Efficiency, and Memorability aspects, but Errors are still found on the LMS. Based on the SUS score, the respondents said that the LMS is below the average score or the Acceptable level. Therefore, there need some particular improvements to the LMS to give better services.

CONCLUSION
Based on the analysis elaborated above, the State Polytechnic of Malang's LMS has a low usability level, so particular required improvements are needed to make it much better. Fortunately, most of the respondents gave good appreciation in the context of Learnability, Efficiency, and Memorability, even though some Errors are still found and below the Acceptable level. For the Satisfaction aspect, most respondents still need the LMS to support the online learning, making them continue visiting and using it. There needs to be further research on the State Polytechnic of Malang's LMS to identify various problems found in more detail using different methods or instruments. Future research may be focused on the usability effectiveness of the LMS in each department in the State Polytechnic of Malang to find the departments that are needed to be improved in maximizing the use of the LMS.