Development and validation of Digi-MEE Instrument measuring online learning environments in medical education

Objectives: To develop and validate Digital Medical Education Environment (Digi-MEE) Instrument for measuring online learning environment in medical education. Methods: This series of studies involved 696 participants from May 2022 to December 2022. Following scoping review, invited modified e-Delphi experts developed consensus on the components and related items for measuring online learning environments. A panel of content experts and a group of medical students carried out content and response-process validation to determine Content Validity Index (CVI) and Face Validity Index (FVI) respectively. This was followed by exploratory and confirmatory factor analysis and reliability analysis to determine Digi-MEE’s factorial structure and internal consistency using SPSS version 26.0 and AMOS 26.0. Results: Delphi experts agreed upon nine components with 73 items of initial Digi-MEE version. CVI of Digi-MEE 2.0 was more than 0.90. with FVI of Digi-MEE 3.0 of 0.87. Exploratory factor analysis yielded 46 items with 57.18% variance. Confirmatory factor analysis led to the final Digi-MEE version containing 28 items within nine components with acceptable levels of goodness of fit indices. Overall Cronbach alpha of the final Digi-MEE was more than 0.90, and for the nine components ranged between 0.62 and 0.76. Conclusion: Digi-MEE is a promising valid and reliable instrument to evaluate online education environment in medical education. Content, response-process, factorial structure, and internal consistency evidence support the validity of Digi-MEE. Medical schools can use Digi-MEE as an evaluation tool for the continuous quality improvement of online learning environments.


INTRODUCTION
A learning environment provides a setting for students to learn and engage with their teachers, staff, and peers, encompassing physical, social, and psychological aspects and reflects the curriculum being taught. 1 The learning environment, also called the academic environment is crucial for providing quality education as it includes many factors that contribute to effective curriculum delivery, which is essential for training future professionals and student academic performance.Van Vendeloo et al demonstrated that the learning environment plays an indispensable role in promoting student well-being and preventing burnout among Belgian residents. 2he World Federation for Medical Education (WFME) has also recognized the learning environment as a pivotal element to evaluate in medical education programs. 35][6] However, different educational contexts may require a different inventory that is tailored to the specific situation of the institution.Specific tools have been developed to measure the learning environment in a particular discipline, such as the Anatomy Education Environment Measurement Inventory (AEEMI) 7 and the Healthcare Education Micro Learning Environment Measure (HEMLEM). 8Similarly, there is a need to design an inventory specifically for measuring the online learning environment in medical education.
Over the past twenty years, there have been some tools for measuring the quality of online learning environments in general education.However, these tools have limitations.For instance, the Constructivist Online Learning Environment Survey (COLLES) measures university students' perceptions of various factors related to learning 9 but lacks reliability and validity to be used further.The Technology-Rich Outcomes-Focused Learning Environment Inventory (TROFLEI) assesses different dimensions of high school classroom environments, 10 such as student cohesiveness, and computer usage, but is not comprehensive.Distance Education Learning Environment Survey (DELES) 11 and Online Learning Environment Survey (OLLES) 12 measure different domains of post-secondary and tertiary educational institutions' environments, respectively.However, they do not focus on specific attitudes and skills that may be essential for effective online learning.
Recently, Mousavi A et al validated an E-learning educational atmosphere measure (EEAM) to measure online learning based on different factors, such as virtual education status, teaching skills, learner support, and professional ethics. 13However, the study involved a variety of students from various educational backgrounds.Syed emphasizes that it is necessary to develop an inventory that is highly contextualized and specifically targets medical students' needs and requirements. 14Such an inventory will ensure a conducive online learning experience by measuring specific constructs according to what the millennials require for online education.
There is a need to develop consensus on key components of an effective online learning environment for undergraduate medical education, given the overlap in markers discussed above.Additionally, as digital technologies continue to advance, new concepts keep emerging that should be considered when measuring parameters such as computer literacy.Thus, it is necessary to take a systematic approach in developing and validating an instrument that can accurately measure the online learning environment including both synchronous and asynchronous learning environments.This study aims to develop and validate a new inventory named Digital Medical Education Environment (Digi-MEE) instrument to measure the online learning environment.

METHODS
The study was conducted between May 2022 and December 2022, utilizing two main phases described by Yusoff et al. 15 In the first phase, experts (i.e., medical educationists, medical teachers, and instructional designers) provided feedback on identified content and terminology related to the design of the instrument's items and domains via modified e-Delphi study.In the second phase, the instrument was validated to establish its content validity, response process validity, construct validity, and internal consistency through serial crosssectional surveys.

Development of Digi-MEE items:
A scoping review was conducted leading to the generation of nine components and 25 sub-components of online learning environments in medical education. 16Based on the nine components; relevant items were agreed upon in a two-round modified e-Delphi study involving 18 invited experts from May 2022 to August 2022 using an online forum.should be at a minimum of 0.80. 17Based on the I-CVI values, items with I-CVI of 1.0 were accepted, items with I-CVI between 0.79 and 1 were re-discussed, and items with I-CVI less than 0.78 were rejected.I-FVI is the proportion of instrument users (i.e., respondents) giving an item a clarity rating of 3 or 4, while S-FVI is the average of the I-FVI scores for all items on a scale.The acceptable I-FVI and S-FVI value has been set at a minimum of 0.78, and 0.80 respectively. 19Based on the I-FVI values, items with I-FVI of 1.0 were accepted, items with I-FVI between 0.79 and 1 were revised, and items with I-FVI less than 0.78 were rejected.

Validation of Digi-MEE domains and items: Content Validation
For the qualitative analysis, open comments given by the medical students related to each item were analyzed for any revision in sentence structure, format, spelling errors as well as any addition, deletion or shifting of any item to another component as suggested.Assessment of factorial structure and internal consistency: Digi-MEE 3.0 was administered to 230 undergraduate medical students in another private medical school in Pakistan via a cross-sectional survey from October 2022 to November 2022 after obtaining informed consent.Osborne and Costello indicated that sample size calculation for factorial analysis should be either in the ratio of 5:1 for the number of items or greater than 100. 20Participating students rated their experience of their school's online learning platform using Digi-MEE 3.0 in a Likert scale from 1-4 (scale:1= strongly disagree, 2= disagree, 3= agree, and 4= strongly disagree).

Exploratory factor analysis (EFA):
EFA was performed through SPSS version 26.0 to determine the number of domains of the online learning environment and related items for Digi-MEE.Data suitability was determined by the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) and Bartlett's Test of Sphericity.KMO value > 0.7 indicates a good level of factor distinction based on an adequate sample, whereas significant Bartlett's Test (p-value <0.05) demonstrates that the factor analysis is appropriate. 21The factors were extracted through the principal component axis (PCA) to check for total and cumulative variance for the nine extracted components and factor loading of each item.The initial component matrix was then rotated via Varimax with Kaiser normalizations after 28 iterations.For each component, Eigenvalues >1 and item factor loading values >0.40 signify convergent validity. 22Items with factor loading <0.4 were removed.The items were shifted in respective components where they loaded maximally, after being reviewed by two independent researchers (NKN and MSBY) who considered the theoretical meaning of each component.This led to the development of Digi-MEE version 4.0.Confirmatory factor analysis (CFA): CFA was carried out on the data obtained from a cross-sectional survey of 450 undergraduate medical students from a public sector university in Pakistan for their ratings of their medical school's online learning environment using Digi-MEE 4.0 from November 2022 to December 2022.The sample size was determined based on the criteria of 10 participants per item for an adequate sample for CFA. 23ata was analyzed using SPSS version 26.0 and Analysis of Moment Structure (AMOS) version 26.0.Model Chi-square goodness of fit (set at 0.05) demonstrated model fit.Approximate fit indexes were applied to check the available data for measurement model fitness.For absolute fit indexes, Goodness of Fit Index (GFI) (model fit >0.9) and root mean square error of approximation (RMSEA) (model fit=RMSEA<0.08and Root mean squared residual-RMR<0.05)were used.For incremental fit measurement, Comparative fit index (CFI) (model fit >0.9), Normed fit index (NFI) (model fit >0.9), and incremental fit index (IFI) (model fit >0.9), and Tucker Lewis fit index (TFI) (model fit >0.9) were used. 23arsimonious fit was measured through Chi-Square/ Degree of Freedom (Chi Sq/df). 13nternal Consistency: Convergent Validity was determined with the size of factor loading (0.5 or more), as well as composite reliability (CR).Composite Reliability (CR) was calculated on Microsoft Excel using formulae given by Fornell and Larckers. 24Values of CR of more than 0.6 indicate good construct reliability and adequate Convergent Validity. 25ollowing CFA, data was used to perform reliability statistics to assess the internal consistency of final version of Digi-MEE (Digi-MEE 5.0).Cronbach's alpha coefficient was determined as the measured parameter to determine internal consistency.Cronbach's alpha values between 0.7 and 0.9 are considered to represent high internal consistency and values between 0.6 and 0.7 are considered satisfactory. 15

RESULTS
This study involved a total of 696 participants, including 25 experts from eight different countries (Canada, Egypt, Iran, Malaysia, Pakistan, Saudi Arabia, Kuwait, and the USA) and 671 undergraduate medical students from four different institutions in Pakistan.Table-I shows demographic data of participants involved in different phases of the study.

Development of Digi-MEE items: Identification of components, subcomponents, and functional elements:
Literature review led to the identification of nine components and 25 subcomponents with 73 functional elements for effective online learning environments in medical education. 14   shown in Fig- 3. Minimum loading was observed by item LC2 on Learning Characteristics (0.508).Seven items had loadings between 0.5 to 0.59.Twenty-seven items had loadings between 0.6 to 0.69.Seven items out of 46 had loadings >0.69.The maximum loading on the factor was observed by item LC4 on Learning Characteristics (0.749).
There was high correlation and low covariances between the latent factors in the final model (Fig. 3).

DISCUSSION
This study provides four essential pieces of evidence to support the psychometric credential of Digi-MEE to measure the digital learning environment in medical education: content validity, response process validity, internal structure validity and internal consistency.The final validated 28-item Digi-MEE instrument with Likert scale is shown in Table-VII.
Firstly, Digi-MEE instrument demonstrated a high level of content validity as the content was developed through a robust and systematic process based on the evidence and expert consultation. 26The nine components of Digi-MEE are infact reflection of our TELEMED (Technology Enhanced Learning Environments in Medical Education) model. 16These components, namely cognitive enhancement (four items), content curation (three items), cybergogical practices (three items), learner characteristics (four items), digital capability (three items), platform usability (two items), facilitation dynamics (three items), social representation (three items), and institutional support (three items) are well aligned with the issues on online learning environment addressed in the current literature.For example, digital capability represents the much-needed requirement for the medical student to not only develop digital skills but also demonstrate self-management in online environments. 27In contrast, previously available online learning environment inventory focused on development of computer competence only. 9,10revious inventories for digital atmosphere measure social interaction as an essential component. 9,10As an expansion to this platform, the 'social representations' domain of Digi-MEE instrument addresses the need to demonstrate 'ethical and professional practices while interacting in online platforms' for a safe and conducive learning environment. 26Ensuring that the students demonstrate ethical and professional practices in online learning environments can lead to professional digital identity formation of medical students. 28he Digi-MEE instrument presents 'facilitation dynamics' as a set of activities to be ensured by the facilitator in online learning environments that can support learners in their education.Instead of focusing on the facilitator support and teaching skills as mentioned in previous inventories 9,12 , Digi-MEE provides a guideline of validated items for the facilitator to ensure while managing student-centered learning.Similarly, the platform usability and institutional support, vaguely mentioned before 13 , are now clearly presented with items to support students' learning in digital platforms.It has been proven that well designed e-learning platforms influence student satisfaction and improve performance while continuous institutional support can guide and train both learners and facilitators to use these online platforms up to their potential in an effective manner.The continuous integration of online learning in medical schools also necessitates the development of robust institutional policies for online learning and their dissemination to the users of online learning platform highlighted in Digi-MEE instrument. 29he next three components are interlinked and emphasize upon the need to have curated content leading to cognitive enhancement via effective cybergogical practices in online platforms.The content remains one of the important interactions with the students when learning online.
However, with introduction of open educational resources, content is here by proposed to be 'curated' instead of being made from scratch as discussed by the Experts in Content Validity phase. 30The curated content should engage the learner in online platforms to provide meaningful learning and should be organized in a manner for easy comprehension. 31Using this curated content, facilitators can provide activities which allow learners to interact and collaborate with each other to carry out their learning and assessment in online learning platforms.Lastly, 'learner characteristics' of Digi-MEE evaluates students' perception of interest, effort and motivation for online learning which have also been previously used as parameters in other instruments. 32Hence, Digi-MEE presents constructive alignment of the prevailing needs of online learning environments within its nine main components, indicating adequate and relevant coverage of required content for online learning environments in medical education.
Secondly, the response process validity of Digi-MEE was high as the users easily understood it.This is an important finding as this study utilized students as participants who are end users of the finalized Digi-MEE instrument and their input is necessary for questionnaire development at this stage.In addition, the open comments allowed for any needed item revisions and/or removal as suggested by the participants. 19hirdly, Digi-MEE demonstrates a multi-dimensional factor and reached a good model fit as all goodness of fit indices met the acceptable level.EFA of Digi-MEE instrument demonstrated sample adequacy (KMO=0.861,Bartlett's test of sphericity=0.000)with acceptable factor loadings of all items in extracted factors (>0.40). 21iterature debates of varying threshold levels for item factor loadings in EFA depending on the sample size of collected data.Hair et al. proposed a loading threshold of 0.4 for sample size of 200. 33e had invited 230 participants in our study out of which 200 responded to fill the survey.Therefore, a loading threshold of 0.4 applies to our study.Yong & Pearce explain that a high threshold level of 0.5 or more ensures that only items with strong factor loadings are

2
The online platform provides opportunities to promote my independent learning.

3
The content is presented appropriately to enhance my understanding.

4
I understand the orientation given before the task easily.

5
The online learning platform provides clear learning outcomes for the given course I am enrolled in. 6 My online activities with others are monitored in this online learning platform.

7
The online platform allows me to exchange information with my peers/ facilitators easily.

8
The activities on the online platform allow me to interact with others.

9
This online platform provides suitable assessment methods to facilitate my learning.

10
I can communicate and collaborate with my peers/facilitators on this online platform easily.

11
I can see my basic profile information as well as that of my peers/ facilitators on this online platform.
12 I can manage my digital screen time on an online learning platform.
13 I feel I am part of the online learning community.
14 I am provided with timely feedback on my work.

15
I give feedback about courses which I am enrolled in online learning platforms. 16 The online platform encourages me to participate in online learning activities in a professional and ethical manner.
17 The online content can be accessed with ease.18 The platform interface is simple and follows a consistent design.

19
The online platform usage policies are widely disseminated among students.
20 The online platform rules and regulations are informed to me.

21
The institution provides training to me for using online platforms appropriately.
22 The online content is organized in an engaging manner.

23
The facilitator(s) selects the appropriate tool for teaching us online.
24 The facilitator(s) provides positive encouragement to me during classes.

25
I show interest in learning about a given topic in an online learning platform.
26 My learning is supported by this online learning platform.
27 I try my best to put in effort during online activities.

28
The online platform for learning is well accepted by me.
considered to load onto a factor, making it easier to interpret the factor structure and the meaning of each factor. 22However, high threshold may disregard subtle relationships between the variables which we did not want to miss.Having lower thresholds helped us reduce the losing information about the underlying dimensions in the data. 33FA of Digi-MEE depicts minimum discrepancy per degree of freedom (CMIN/DF) ratio of 2.46(<3.00)which represents a satisfactory fit which is close to value of 2.63 in study performed by Butt et al. 34 Further in the same study, the goodness of fit index was 0.911 which is comparable to our model's good for fitness index (GFI).A GFI >0.95 indicates a good model fit, hence our model has an acceptable fit.Another important component in covariance structure modeling is the RMSEA with acceptable values being less than 0.05.In our study, the RMSEA was 0.057 and RMR WAS 0.03 representing constructs one-dimensionality.
Finally, the internal consistency of Digi-MEE was acceptable as the Cronbach alpha and composite reliability values were more than 0.6.Our validated questionnaire demonstrated overall Cronbach's alpha of 0.965, showing excellent internal consistency of Digi-MEE instrument as per criteria defined by George and Mallery's rule of thumb. 35The Cronbach's alpha of the subscale ranged from 0.623 to 0.751, again ranging from acceptable values to relatively high internal consistency (2).When compared with alpha values demonstrated by Rahayu, et al. validation study for Online Classroom Learning Environment Inventory (OCLEI) 36 , our subscale of "social representation" had similar reliability as their scale's "interaction" subscale (α =0.724).However, the platform utility of our scale (α =0.685) was slightly less than what the above study had measured (α =0.744).One of the items out of the two in this subscale relates to access to the platform while the second item relates to platform interface.On the other hand, our subscale of 'Facilitation Dynamics' (α =0.737) is more reliable as compared to OCLEI's subscale of lecturer support (α =0.718).
Major strengths of the study include recommended best practices and guidelines for development and validation of Digi-MEE instrument.The nine domains on online learning environment in medical education were derived from multisource evidence including theory, literature, input from experts and learners.The experts involved in the modified e-Delphi and content validation study were from diverse geographical backgrounds as well as professional backgrounds involved in online education.This diversity of experts makes Digi-MEE a suitable instrument for different institutions all over the world.
Limitations: It includes study sample for factor analysis and reliability analysis from one country only.Future validation studies should be done in different countries.Also, the sample was confined to undergraduate medical students.Further validation must be done before Digi-MEE is used for postgraduate medical students.

CONCLUSION
This study introduced a new measurement tool, namely Digi-MEE, to evaluate the digital learning environment in medical education.The validity of Digi-MEE is demonstrated through its content, response process, factorial structure, and internal consistency.This tool can be used by medical schools to evaluate their digital learning environment for continuous quality improvement.Nevertheless, more validation research should be conducted to verify the validity of Digi-MEE in different educational settings.

Abbreviations: AMOS:
Analysis of Moment Structure.

CVI:
Content validity index.Digi-MEE: Digital Medical Education Environment.

SPSS:
Statistical Package for Social Sciences.
Ethical Approval: The study received approvals from the Human Research Ethics Committee of Universiti Sains Malaysia (USM/JEPeM/21050350) on October 11, 2021, and November 3, 2022, and from the Review Boards of different data collection sites (Letter No. IEC/80-21, dated April 23, 2021, Letter No. MC/DME/609/2022, dated 2 nd September, 2022, Data Collection Approval dated September 16, 2022).Fig.1 presents the flow chart for the development and validation process of Digi-MEE.

:Fig. 1 :
Fig.1: Flow chart for the development and validation of Digi-MEE.
Nine components included content curation, cognitive enhancement, digital capability, learning facilitator, learner characteristics, institutional support, pedagogical practices, technological usability, and social representations.Achieving consensus on identified components and subcomponents: Fifteen out of 18 invited experts participated in the modified e-Delphi and the data about their consensus is presented in Table-III.The names of three main components were revised after consultation (Pedagogical Practices to be changed to Cybergogical Practices, Technological Usability to be changed to Platform Usability and Learning Facilitator to be changed to Facilitation Dynamics).Definitions of three main components (Institutional Support, Cognitive Enhancement and Content Curation) and two subcomponents (Content Selection and Content Organization) were revised and presented in round two.Expert ranking of nine components in Round Two is shown in Fig-2.The component of 'Social representations' Noor-i-Kiran Naeem et al.

Fig. 2 :
Fig.2: Expert ranking for the components of online learning environments in medical education.

Fig. 3 :
Fig.3: Standardized Factor Loadings of Digi-MEE Constructs based on the final model.

Table - I
: Demographic details of the participants involved in different phases of study.

Table -
Out of 73 items, five items having I-CVI of less than 0.78 were removed and 21 items were removed due to similar meaning.Five items were shifted to a different component upon expert suggestions while 17 items were reworded for grammatical inadequacies and clarity as suggested by experts (Table-II).Digi-MEE 2.0 thus had 47 items.Overall, the S-CVI/Ave by score improved from 0.95 to 0.98 in Digi-MeE 2.0, S-CVI/Ave by proportion ranking by experts from 0.95 to 0.98 and S-CVI/UA from 0.69 to 0.79.Table-III: Agreement levels on component definition and appropriateness in Modified Delphi Study.II: Modifications in Digi-MEE questionnaire in different phases of the study.

Table -
IV: Range of Factor Loadings, Eigenvalues and Cronbach's alpha of components of Digi-MEE 3.0.

Table -
VI: Reliability analysis of the 28 items of the Digi-MEE based on the final model.Domains were determined based on basis of expert validation and exploratory factor analysis DCR (Composite Reliability) was calculated based on formula given by Fornell & David 22 www.pjms.org.pk1581 A Reliability analysis: Cronbach's Alpha Coefficient, B

Table VII :
Final Validated 28-item Digi-MEE Instrument with Likert Scale.