Predicting examinee performance based on a fuzzy cloud cognitive diagnosis framework in e-learning environment

Abstract

The score profiles could be used to measure learners’ skills proficiency via cognitive diagnosis models (CDMs) for predicting their performance in the future examination. The prediction results could provide important decision-making supports for personalized e-learning instruction. However, facing the possible complexity of skills, the uncertainty of learners’ skill proficiency and the large-scale volume of score profiles, the existing CDMs have limitations in the measurement mechanisms and diagnostic efficiency. In this paper, we proposed an approach based on a fuzzy cloud cognitive diagnosis framework (FC-CDF) to predicting examinees’ performance in e-learning environment. In this approach, the normal cloud models (NCMs) are utilized innovatively to measure the expectation, degree of variation and variation frequency of learners’ skill proficiency, and each NCM is transformed into an interval fuzzy number to characterize the uncertainty of the skill proficiency for every learner. Combining the educational psychology hypothesis with the parameter estimation method, we could obtain the learners’ skill proficiency level and the slip and guess factors relevant to each test item, based on which the learners’ scores could be predicted in a future test. Finally, the experiments demonstrate that the proposed approach provides good accuracy and significantly reduces execution time for predicting examinee performance, compared with the existing methods.

1 Introduction

1.1 Motivation

Recently, e-learning has become popular worldwide due to the COVID-19 pandemic (Maatuk et al. 2022), and a massive amount of educational data has been collected in e-learning platforms. Assessing the learners’ cognitive abilities based on such data has potentials to help instructors and learners to obtain a comprehensive view of a learner’s learning progress (Mai et al. 2021). The score profiles may be analyzed by cognitive diagnosis models (CDMs) to evaluate the learners’ strengths and weaknesses by identifying the presence or absence of multiple fine-grained skills required for solving problems on a test (Torre 2008). The measurements of learners’ skill proficiencies are helpful in predicting examinee performance in a future test (Ma et al. 2020; Song et al. 2022; Yao et al. 2019). This insight enables the possibility of personalized recommendation (Wan and Niu 2020; Yang et al. 2023; Yun et al. 2022), school precaution (Bernacki et al. 2020; Niyogisubizo et al. 2022), examination paper generation (Wu et al. 2020), collaborative learning team formation (Ma et al. 2023, 2022) and learning path recommendation (Jiang et al. 2022; Liu and Li 2020; Liu et al. 2019b).

Learners’ performance is usually predicted by cognitive diagnosis in the existing research on educational psychology (DiBello et al. 2006; Sessoms and Henson 2018). Researchers and practitioners from both educational psychology and data mining fields have devoted many efforts to CDMs due to their outstanding advantages for analyzing the learners’ cognitive status. Some classical CDMs, e.g., the item response theory (IRT) (Janssen et al. 2000) and the Deterministic Input, Noisy And-gate (DINA) model (Torre 2008), have been applied widely. Among them, IRT models a learner as a single proficiency variable, i.e., latent trait, and the performance of a learner in a test item could be predicted by combing it with some items attributes, e.g., degrees of difficulty and degree of distinction, and the DINA model introduces a Q-matrix (Tatsuoka and Kikumi 1983) to measure learners’ proficiencies from the skill level. The Q-matrix is a matrix to describe the relationship between test items (i.e., questions) and the skills that are usually relevant to the specified knowledge point. Aiming at the shortcomings of classical CDMs, some improved models have been developed, such as HO-DINA (Torre and Douglas 2004), P-DINA (Tu et al. 2010), FuzzyCDF (Liu et al. 2018), R-FuzzyCDF (Li et al. 2017).

However, the above CDMs still have much room to improve the measurement mechanisms and diagnosis efficiency: (1) The DINA model only uses a binary value to measure one learner’s skill proficiency, while FuzzyCDF and R-FuzzyCDF can improve the accuracy of predicting examinee performance by introducing a fuzzy number-based measurement method. However, an integer or a fuzzy real number fails to address the uncertainty of learner’s skill proficiency. In reality, the real performance of learners might be unstable due to the complexities of one skill itself and the diversity of application scenarios. (2) Although recent FuzzyCDF and R-FuzzyCDF significantly improved the accuracy of performance prediction compared with IRT and DINA model and worked well in a small-scale data environment, their diagnosis efficiency is dramatically reduced due to the slow convergence speed and numerous estimated parameters. Particularly, more and more education programs have been moved to online channels due to the pandemic, and the massive amount of educational data could be automatically collected and used for the investigation and prediction of the learners’ learning performance on e-learning platforms. Thus, two key issues are still open for studying CDMs in an e-learning environment: (1) How could we measure a learner’s skills proficiency comprehensively and accurately according to the relationship among learners, test items, and skills? (2) How could we improve the execution efficiency of cognitive diagnosis without a loss of the diagnosis accuracy for supporting large-scale data processing scenarios involving a great deal of learners, items and skills?

To address the above issues, we innovatively introduced the normal cloud models (NCMs) (Peng et al. 2018; Zhang et al. 2007) to model the learners’ skills proficiency. The NCM (Wang et al. 2014; Yang et al. 2018) is an effective tool to realize the conversion between qualitative concepts and quantitative expressions and can be used to measure the fuzziness, the randomness and the relevance of uncertain concepts. At present, it has been applied to many fields, such as intelligent control and information processing (Wang et al. 2014; Yang et al. 2018), system evaluation (Liu et al. 2019a; Ma et al. 2021), recommendation system (Peng et al. 2018; Zhang et al. 2007) and decision-making (Gong et al. 2021; Wang et al. 2021). One NCM consisting of three numerical characteristics (i.e., expectation, entropy and hyper-entropy) is obtained via a reverse cloud generator (RCG) (Li et al. 1995).

Based on the NCMs, an approach via fuzzy cloud-based cognitive diagnosis framework (FC-CDF) is put forward to predict learners’ performance for both subjective and objective test items in an e-learning environment. In this approach, a cognitive cloud is defined for measuring the expectation, degree of variation and variation frequency of learners’ skill proficiency, and each NCM is transformed into an interval fuzzy number to characterize the fuzziness and uncertainty of the skill proficiency for every learner. Similar to FuzzyCDF (Liu et al. 2018), the learners’ cognitive status on items is calculated by combining the educational hypotheses and fuzzy sets theory. The learners’ performance could be predicted in a future examination by considering the slip and guess factors of each test item. Finally, the experiments based on three real-world datasets, a simulation dataset and three extended datasets are analyzed by comparing the other approaches, and the results demonstrate that the proposed approach provides good accuracy for predicting examinee performance and significantly reduces the execution time of cognitive diagnosis.

1.2 Our contributions

The main contributions of this paper are as follows:

• (1) To measure the learners’ real skills proficiency more objectively and comprehensively, the cloud model theory is introduced innovatively into FC-CDF. Different from the existing approaches that measure the learners’ skill proficiency with an integer or a fuzzy real number, FC-CDF establishes an interpretable fuzzy measurement mechanism to characterize the uncertainty of learners’ skills proficiency via two-parameter interval numbers transformed by cognitive clouds. Based on this mechanism, this proposed approach could predict the learners’ performance accurately in future examinations.

• (2) To provide an efficient cognitive diagnosis for supporting the learners’ autonomic learning online in their fragmented time, FC-CDF employs a reverse cloud generation algorithm to calculate the learners’ skill proficiency. Based on it, the process of parameters estimation is simplified and the computation speed of cognitive diagnosis is accelerated significantly, in comparison to the existing approaches. The proposed approach could be applied to a large-scale data environment involving many learners, test items and skills.

The remainder of the paper is organized as follows: Sect. 2 reviews the related work. Section 3 gives the problem statement. Section 4 introduces the preliminaries. Section 5 proposes the framework of fuzzy cloud cognitive diagnosis. Section 6 puts forward the examinee performance prediction via FC-CDF. Section 7 analyzes the experimental results. Finally, Sect. 8 concludes the paper.

2 Related work

2.1 Cognitive diagnosis model

In psychological and educational measurement, the cognitive diagnosis is a diagnostic assessment of an individual’s cognitive process, skill or knowledge structure (Nichols and Joldersma 2008). Researchers proposed the psychometric models with cognitive diagnosis function (i.e., CDMs) based on the cognitive psychology and psychometrics theory. Generally, CDMs could be classified into two categories, i.e., the continuous CDMs and the discrete CDMs. The representative continuous CDM is IRT (Janssen et al. 2000), and the DINA model (Torre 2008) is a classical discrete CDM.

2.1.1 Item response theory

IRT assumes that each learner has a “unique latent trait.” Thus, it measures one learner via a one-dimensional continuous variable to predict her/his performance in test items with different discrimination and difficulty features. The 2-parameter logistic model (2PL) (Janssen et al. 2000) is defined by

$$ P\left( {X_{ij} = 1|\theta } \right) = \frac{1}{1 + \exp (D \times a(\theta - b))} $$

(1)

where P(X_ij = 1|θ) denotes the probability of learner #i with the latent trait #θ answering item #j correctly, a and b are the discrimination and difficulty of test item, respectively, θ represents the latent trait of learner, and D is an empirical scaling constant in logistic cognitive models, which is set as–1.7 (Gregory and Camilli 1994). IRT is built on a rigorous and complex mathematical model, and it has been applied to examinations and traditional intelligence tests. However, the unidimensionality assumption used by IRT is too coarse-grained to accurately measure learners’ proficiency, since learners’ proficiency is related to different skills. To address this issue, the DINA model is proposed to depict learner’s proficiency from the skill level (Torre 2008).

2.1.2 DINA model

Different from IRT, the DINA model introduces a Q-matrix (Tatsuoka and Kikumi 1983), which indicates whether the skills are associated with items, and diagnoses the learner’s cognitive status with a proficiency vector about multi-dimensional skills. Each dimension of this proficiency vector denotes whether a learner masters one specific skill. In this case, the mastery of learner #i on item #j is calculated as follows:

$$ \eta_{ij} = \prod\limits_{k = 1}^{K} {\alpha_{ik}^{{q_{jk} }} } $$

(2)

where α_ik is a binary value indicating whether the learner #i master the skill #k, 1 denotes that #i masters #k; otherwise, 0 is suggested; q_jk denotes whether the skill #k is required to correctly answer the item #j (1 means required and 0 otherwise); and η_ij = 0 means that the learner #i cannot master the item #j; otherwise, η_ij = 1.

Though the DINA model strives to overcome some shortcomings of IRT, there are still some limitations in this model, such as dichotomous description mechanism (Tu et al. 2010); (Cai et al. 2017), underexplored cognitive diagnosis on subjective items (Liu et al. 2018) and high computational cost (Wang et al. 2018). Recently, many efforts have been devoted to improving DINA models for understanding the learner’s learning status comprehensively and providing effectively personalized learning guidance for learners.

2.1.3 Improved DINA model

The improved DINA models could be divided into several categories as follows:

• (1) Improve the dichotomous 0–1 scoring mechanism

  The DINA model only supports the dichotomous 0–1 scoring mechanism, in which a learner gets either a full score or zero in a test item. Tu et al. (2010) developed a P-DINA model to provide a polychotomous scoring mechanism on a scale of zero to a full score. However, the score values easily focus on two extremes (i.e., 0 or a full score) when the P-DINA model is used in data analysis. Thus, Cai et al. (2017) presented an rP-DINA (revised P-DINA) model by reconstructing learner’s ideal score, and this model support all kinds of ideal scores ranging from 0 to full score.

• (2) Adopt the new diagnosis mechanism for subjective test items

  Considering that the traditional DINA model is unable to support the cognitive diagnosis for the subjective items, (Liu et al. 2018) proposed a fuzzy cognitive diagnosis framework (FuzzyCDF). FuzzyCDF measures the skill proficiency of a learner with the degree of membership (i.e., a real number in the range [0, 1]) and adopts the cognitive answer mode via the fuzzy intersection and the fuzzy union operations. In FuzzyCDF, the mastery of a learner on objective items is of conjunctive type and the mastery of a learner on subjective items is of compensatory type. For an objective item, a learner may answer it correctly if he/she possesses all the skills required for it. As for a subjective item, the more skills a learner possesses, the higher the score that the learner gets from it. Based on FuzzyCDF, Li et al. (2017) put forward a revised framework called R-FuzzyCDF by taking into consideration the influence of the importance of every skill. Although R-FuzzyCDF might provide a better accuracy of performance prediction than FuzzyCDF, more parameters being introduced makes the process of cognitive diagnosis more time-consuming than FuzzyCDF.

• (3) Extend to High-order CDMs

  The high-order CDMs are developed to describe the possible structural relationship among multiple latent traits in CDM. Considering that the proficiency of a learner may be affected by one (or more) high-order latent characteristics, De la Torre and Douglas (2004) presented a high-order latent structure model and combined it with the DINA model to design a high-order DINA model (HO-DINA) for reducing the number of estimated parameters. Considering that the HO-DINA model only processes data with a 2-order latent structure, Zhan et al. (2019) put forward a multi-order DINA model by combining a multi-order latent structural model and the DINA model for processing 3-order or higher order latent structure data.

• (4) Improve computational efficiency

  The traditional CDMs usually work well when the small-scale sample data are involved in. However, in the large-scale e-learning scenario, the calculation of traditional CDMs will be significantly time-consuming due to their slow convergence speed. Aiming at this issue, Wang et al. (2018) proposed an approach by integrating Maximum Entropy DINA and Incremental Maximum Entropy DINA to accelerate the calculation speed of the DINA model.

2.1.4 Neural networks-based CDMs

The great success of neural networks in both computer version and natural language processing domains attracts researchers to apply it to the cognitive diagnosis field (Liu 2021).

Aiming at the problem of poor diagnosis performance of IRT on sparse data, (Cheng et al. 2019) presented a general deep item response theory (DIRT) to enhance the robustness of traditional IRT. In DIRT, the neural networks are used to mine the text information of each test item for estimating the parameters in the traditional IRT formula. Wang et al. (2020, 2022) proposed a general neural cognitive diagnosis model (NeuralCDM) by applying the neural networks into cognitive diagnosis. Based on the past performance of learners, NeuralCDM used the neural networks to enhance the model’s learning ability for more accurately modeling learners’ proficiency, items’ factors (e.g., discrimination of the item and the difficulty of the skill required by the item) and their interactions. Gao et al. (2021) combined the graph neural networks with an attention mechanism to model student-exercise-skill hierarchical relations and improve the accuracy of cognitive diagnosis.

However, there remain some limitations for the neural networks-based CDMs due to the poor interpretability and the overfitting problem when regarding to small sample data.

2.2 Predicting examinee performance

2.2.1 Matrix factorization-based methods

The matrix factorization (MF) is widely used in recommendation systems and can also be used to predict examinee performance. For example, the MF techniques could measure the degree of learners’ performance in a low-dimensional space by mapping each learner and each item into a low-dimensional vector. Tscher et al. (2010) modeled learner by singular value decomposition. Thia-Nghe et al. (2010) leveraged MF model to predict learners’ performance to improve the prediction accuracy. Thia-Nghe et al. (2015) put forward a multi-relational factor factorization approach for modeling learners in the intelligent tutoring system and predicting the learners’ performance. However, the traditional MF-based examinee performance prediction methods usually lack a good interpretability. For example, each dimension of a learners’ latent vector could not correspond to a specific skill. Thus, it is hard to explain the relationship between skills and such a latent vector.

2.2.2 Cognitive diagnosis model-based methods

The cognitive diagnosis models (CDMs) usually use learners’ score profiles as input data to model their mastery of skills. Based on the diagnosed skill proficiencies, learners’ performance on the unfinished items could be predicted. Additionally, the accuracy of predicted results could be used to verify the effectiveness of CDMs. In the DINA model (Torre 2008), the mastery of learners on the item and the exception of slip and guess are incorporated to model learners’ real performance in the item. Specifically, the probability of learner #i with skill proficiency α_i answering item #j correctly could be modeled as:

$$ P_{j} \left( {\alpha_{i} } \right) = P\left( {X_{ij} = 1|\alpha_{i} } \right) = g_{j}^{{1 - \eta_{ij} }} \left( {1 - s_{j} } \right)^{{\eta_{ij} }} ,\;P_{j} \left( {\alpha_{i} } \right) = P\left( {X_{ij} = 1|\alpha_{i} } \right) = g_{j}^{{1 - \eta_{ij} }} \left( {1 - s_{j} } \right)^{{\eta_{ij} }} $$

(3)

where s_j and g_j denote the slip and guess factors of item #j, respectively; η_ij is the mastery of learner #i on item #j; and X_ij is the normalized score of learner #i on item #j.

To predict the learners’ performance on both objective and subjective items, FuzzyCDF (Liu et al. 2018) employs the following equation:

$$ P\left( {X_{ij} = 1|\eta_{ij} ,s_{j} ,g_{j} } \right) = \left( {1 - s_{j} } \right)\eta_{ij} + g_{j} \left( {1 - \eta_{ij} } \right) $$

(4)

where (1-s_j)η_ij denotes the probability that learner #i with all necessary skills answers item #j correctly; g_j(1- η_ij) is the probability that learner #i who does not master all the skills required by item #j answers it correctly.

Different from the MF-based methods, the CDM-based methods enhance the interpretability of predicted results by measuring the learners’ cognitive status according to the relationships among skills, items and score profiles. However, the diagnostic efficiency of the existing CDMs is usually time-consuming since they need to spend considerable time in the process of parameter estimation. Moreover, the existing CDMs ignore the uncertainty of learners’ skill proficiency. It might bring some errors to the cognitive diagnosis, and consequently, these errors result in the biases of predicted results.

3 Problem statement

Suppose there is a mobile learning platform for “Java programming” course. After learners have completed a learning task about one knowledge point (i.e., one skill), this platform would provide them with some related exercises (i.e., test items) to check whether they have mastered the skill relevant to this knowledge point. Moreover, this platform also allows learners to spend their fragmented time on practicing or testing online freely and independently. Once a learner completes a test, the platform should recommend the new exercises for this learner’s next round of practice according to his/her latest score profiles. In this case, the platform is required to measure the skill proficiency of learners via cognitive diagnosis quickly and accurately.

Suppose two learners u₁ and u₂ learned three skills k₁ ~ k₃, e.g., “Array Programming,” “Switch Programming” and “Loop Programming,” and finished 12 related exercises in three days. These exercises are designed with different difficulties. For example, one exercise requires learners to add codes to complete an algorithm or figure out the execution result of a given code segment. Obviously, to get high scores on these exercises, one learner should grasp the skills’ essence. One learner should not only understand the theoretical basis related to the skills, but also use them flexibly to solve practical problems by programming. An example of cognitive diagnosis is shown in Fig. 1.

Fig. 1

An example of cognitive diagnosis

In Fig. 1, the score profiles of u₁ and u₂ and the corresponding Q-matrix are the input data. The Q-matrix given by the experts describes the relationship between exercises and skills. q_jk representing the element on the jth row and kth column of the matrix indicates whether skill #k is required to correctly answer exercise #j. q_jk = 1 means skill #k is required; otherwise, q_jk = 0. After the input data are processed by one CDM (e.g., DINA model or FuzzyCDF), the output, i.e., the diagnosis results describing learners’ mastery on every skill, is obtained. In this example, all CDMs are implemented by Python 3.6 on a Linux server with a 2.3 GHz Intel Xeon CPU and 16 GB memory.

From Fig. 1, different CDMs generate different results with different processing efficiency, and the following analysis could be provided:

• 1) For u₂, DINA measures u₂’s proficiency on k₃ with a fixed integer 0; that is, u₂ did not master k₃ totally. However, the score profiles show that u₂ gave the right responses to some exercises, i.e., e₁, e₂, e₅ and e₇, related to k_3.

• 2) For u₁, DINA and FuzzyCDF measure u₁’s proficiency on k₂ with the number 1, which means that u₁ mastered k₂ totally. Nevertheless, actually u₁ gave the wrong responses to some exercises, i.e., e₅, e₇ and e₁₁, related to k₂.

• 3) DINA diagnoses two users’ skills proficiency on 3 skills with the expectation–maximization algorithm and takes 0.001 s. However, when more skills are involved in, the execution time of DINA would increase dramatically since its time complexity is exponential with the growing number of skills. Our experiments show that DINA takes 72.83 s when a real-world dataset Math2, including 16 skills, is used as the input data. It is foreseen that the DINA’s performance is bound to be unbearable when 30 or 40 skills from the “Java programming” course are used.

• 4) Although FuzzyCDF could model the learners’ cognitive status more precisely than DINA, it needs more time to estimate the parameters than DINA does. Based on the u₁ and u₂’s score profiles, FuzzyCDF takes 21 s to diagnose their skills proficiency. However, our experiments show that the FuzzyCDF’s execution time increases to 9,729 s when the Math2 dataset, including 16 skills, is used as the input data. Obviously, it is unacceptable.

According to the above analysis, we could find two key issues are still underexplored: (1) The existing CDMs, e.g., DINA and FuzzyCDF, are not able to measure the uncertainty of skills proficiency for learners objectively. It should be helpful to predict learners’ performance accurately if a reasonable mechanism could be used to measure the characteristic of fluctuation. (2) The execution efficiency of the existing CDMs needs to be significantly improved, especially when the input data cover many skills. It is necessary to explore an efficient and concise CDM to diagnose the learners’ skill proficiency with high diagnosis accuracy.

In summary, a problem of cognitive diagnosis in an e-learning environment could be defined as follows: Given the learners’ score profiles and the Q-matrix, how could we comprehensively measure the learners’ skill proficiency with the consideration of uncertainty, and how could we conduct an efficient cognitive diagnosis to support the large-scale data processing scenarios involving a great deal of learners, exercises and skills?

In dealing with this problem, the following ideas are proposed: (1) To describe the uncertain feature of learners’ skill proficiency, we represent learners’ skill proficiency in the form of interval numbers, which is shown in Fig. 1. (2) To improve the efficiency of cognitive diagnosis, we introduce the cloud model theory to calculate learners’ skill proficiency and its uncertainty, which could simplify the parameter estimation process.

For easy reading, the key definitions used in the following sections are given in Table 1.

Table 1 Key definitions

4 Preliminaries

4.1 Definition of the cognitive cloud

Definition 1. (Cognitive cloud). Define domain X = “score profiles of learner U_i (i = 1, 2, …, I) on the item set P = {P₁, P₂, …, P_J} which requires skill V_k (k = 1, 2, …, K),” domain T = “mastery of learner U_i (i = 1, 2, …, I) on skill V_k (k = 1, 2, …, K)” and $\mu_{T} \left( x \right) \in \left[ {0,1} \right]$ is the certainty degree of $x \in X$ belonging to T (i.e., the value of $\mu_{T} \left( x \right)$ represents the degree to which the score x is close to the learner’s real skill proficiency). The cognitive cloud is defined as the distribution of $\mu_{T} \left( x \right)$ on qualitative concept X, and it can be characterized by the following three numerical characteristics:

• (1) Cognitive Expectation $E_{{\text{x}}}^{{\text{C}}}$: it represents the expectation of a learner’s skill proficiency, which can be regarded as the most representative and typical sample of the qualitative concept.

• (2) Cognitive Entropy $E_{{\text{n}}}^{{\text{C}}}$: it represents the fluctuation range of a learner’s skill proficiency, which reflects the uncertainty of cognitive expectation.

• (3) Cognitive Hyper-entropy $H_{e}^{C}$: it denotes the change frequency of a learner’s skill proficiency, which is the uncertain degree of cognitive entropy.

Thus, a cognitive cloud is expressed as: $C^{C}$ = ($E_{x}^{C}$, $E_{n}^{C}$, $H_{e}^{C}$). The cognitive cloud of learner #i on skill #k is defined as $C^{{{\text{C}}ik}}$. To generate $C^{{{\text{C}}ik}}$, the related score profiles need to be firstly extracted from X. The score profiles of learner #i on the item set requiring skill #k are described as $X_{ik}^{^{\prime}} = \left\{ {\left. {x_{ik1}^{^{\prime}} ,x_{ik2}^{^{\prime}} , \ldots ,x_{ikN}^{^{\prime}} } \right\}} \right.$, in which N is the number of the test items requiring skill #k. The three numerical characteristics of $C^{{{\text{C}}ik}}$ are obtained via RCG (Li et al. 1995) as follows.

$$ \left\{ \begin{array}{l} E_{x}^{{{\text{C}}ik}} = \frac{1}{N} \times \sum\limits_{m = 1}^{N} {x_{ikm}^{^{\prime}} } \hfill \\ E_{n}^{{{\text{C}}ik}} = \sqrt {\frac{\pi }{2}} \times \frac{1}{N}\sum\limits_{m = 1}^{N} {\left| {x_{ikm}^{^{\prime}} - E_{x}^{{{\text{C}}ik}} } \right|} \hfill \\ H_{e}^{{{\text{C}}ik}} = \sqrt {\left| {\frac{1}{N - 1}\sum\limits_{m = 1}^{N} {\left( {x_{ikm}^{^{\prime}} - E_{x}^{{{\text{C}}ik}} } \right)^{2} } - \left( {E_{n}^{{{\text{C}}ik}} } \right)^{2} } \right|} \hfill \\ \end{array} \right. $$

(5)

4.2 Case analysis of cognitive cloud

Suppose a learner’s performance is normalized to [0, 1], where 0 means that the learner gives a totally wrong response to the item, and 1 otherwise. Now, there are two learners #a and #b. The performance of the former on skill #k is [0.90, 0.80, 0.85, 0.90, 0.82], and the performance of the latter on skill #k is [1.00, 0.00, 0.70, 0.60, 0.80]. According to Eq. (5), the cognitive clouds of learners #a and #b on skill #k can be computed as C^Cak = (0.8540, 0.0461, 0.0069) and C^Cbk = (0.6200, 0.3208, 0.1976), respectively. Taking the three numerical characteristics of each cognitive cloud as the input, the scatter diagram with 1,500 cloud drops generated by the forward normal cloud generator (Li et al. 2009) is displayed in Fig. 2, where the abscissa denotes learner’s performance on skill #k, and the ordinate represents the membership degree. Additionally, considering the characteristics of cognitive diagnosis analysis, the abscissa of cloud drops will be intercepted within [0, 1] in practice since a learner’s performance is normalized to [0, 1].

Fig. 2

Cognitive cloud scatter diagram of learner #a and learner #b on skill #k

Figure 2a shows that the cloud drops are densely distributed. Most cloud drops are in a small interval abscissa ranging around $E_{{\text{x}}}^{{{\text{C}}ak}}$ and have higher membership degree due to his good and stable performance on skill #k. For learner #b, Fig. 2b shows that the cloud drop distribution is scattered in the range of abscissa [0, 1] and with a higher membership degree. The reason is that learner #b ever obtained not only a full score but also a zero on skill #k. Figure 2b clearly indicates the uncertainty and fluctuation for the proficiency of learner #b on skill #k. Overall, the cognitive cloud can comprehensively and objectively characterize the learners’ real skill proficiency.

4.3 Fuzzy cognitive interval numbers

Definition 2. (Fuzzy cognitive interval numbers). Suppose b^L and b^H are two real numbers, if B = [b^L, b^H], 0≦b^L≦b^H≦1, B can be called fuzzy cognitive interval numbers, where b^L is the lower limit of the fuzzy interval of learner’s cognitive status, and b^H denotes the higher limit of the fuzzy interval of learner’s cognitive status (i.e., how much a learner masters a skill or an item). The fuzzy cognitive interval number B is a real number when b^L = b^H.

5 Fuzzy cloud cognitive diagnosis framework

By incorporating the cognitive cloud and fuzzy cognitive interval numbers, the fuzzy cloud cognitive diagnosis framework (FC-CDF) measures the learners’ skill proficiency objectively and comprehensively for predicting learners’ performance, shown in Fig. 3.

Fig. 3

4-tiers fuzzy cognitive diagnosis framework

The input data of FC-CDF include the learners’ score matrix, i.e., $X$ and a matrix that describes the relationship between skills and test items, i.e., Q. The diagnosis process of FC-CDF is mainly divided into 4 steps as follows:

• (1) Employ cognitive clouds to model the proficiency of each learner on each skill. For example, $C^{Ci1} = \left( {E_{x}^{Ci1} ,E_{n}^{Ci1} ,H_{e}^{Ci1} } \right)$ represents the cognitive cloud of learner #i on skill #1.

• (2) Transform each cognitive cloud into the fuzzy interval numbers of learners’ proficiency on the skill level by incorporating the three numerical characteristics of cognitive clouds. The transformation results are affected by a parameter w, which is introduced in Subsect. 6.1 and discussed in Subsect. 7.2 in detail.

• (3) Combine the ideas of “fuzzy intersection” and “fuzzy union” to measure learners’ mastery of the items in the form of fuzzy interval numbers.

• (4) Calculate the interval numbers of learners’ performance in each item, and convert these interval numbers into learners’ final performance prediction results. The calculation results are affected by two parameters, i.e., the slip factor s and the guess factor g, which are discussed in Subsect. 6.3 in detail.

6 Examinee performance prediction via FC-CDF

In this section, an approach via FC-CDF is proposed to predict examinees’ performance in an e-learning environment by introducing the cloud model theory.

6.1 Diagnose learner’s skill proficiency

By introducing cognitive clouds, the learner’s skill proficiency is depicted with three numerical characteristics. Among these characteristics, the cognitive expectation denotes the normal level of one learner’s skill proficiency, and the cognitive entropy and the cognitive hyper-entropy depict the uncertainty of one learner’s skill proficiency. Therefore, the learner’s skill proficiency could be depicted comprehensively if these numerical characteristics are integrated into a measurement mechanism. In FC-CDF, the fuzzy interval numbers are introduced. By transforming the cognitive cloud’s three numerical characteristics into fuzzy interval numbers, the measurement mechanism of FC-CDF could not only consider the normal level of learners’ skill proficiency, but also investigate the uncertainty of it. To be specific, the learners’ skill proficiency level matrix is defined as $A = \left[ {\left[ {\alpha_{ik}^{{\text{L}}} ,\alpha_{ik}^{{\text{H}}} } \right]} \right]_{I \times K}$, where $\alpha_{ik}^{{\text{L}}}$ and $\alpha_{ik}^{{\text{H}}}$ denote the lower and upper limits of the skill proficiency level of learner #i on skill #k, respectively.$\alpha_{ik}^{{\text{L}}} ,\alpha_{ik}^{{\text{H}}} \in \left[ {0,1} \right]$. Referring to (Ma et al. 2016), $\alpha_{ik}^{{\text{L}}}$ and $\alpha_{ik}^{{\text{H}}}$ could be calculated as follows: 1) If there are only two items in the item set requiring skill #k, i.e., a learner’s score profiles on skill #k obtain two values r₁ and r₂, then $\alpha_{ik}^{{\text{L}}} = r_{1}$, $\alpha_{ik}^{{\text{H}}} = r_{2}$, r₁ ≤ r₂. 2) When more than two items require skill #k, $\alpha_{ik}^{{\text{L}}}$ and $\alpha_{ik}^{{\text{H}}}$ are computed as follows:

$$ \alpha_{ik}^{{\text{L}}} = \max \left\{ {0,E_{x}^{{{\text{C}}ik}} - E_{n}^{{{\text{C}}ik}} - w \times H_{e}^{{{\text{C}}ik}} } \right\} $$

(6)

$$ \alpha_{ik}^{{\text{H}}} = \min \left\{ {1,E_{x}^{{{\text{C}}ik}} + E_{n}^{{{\text{C}}ik}} + w \times H_{e}^{{{\text{C}}ik}} } \right\} $$

(7)

In Eqs. (6) and (7), $E_{x}^{{{\text{C}}ik}}$, $E_{n}^{{{\text{C}}ik}}$ and $H_{e}^{{{\text{C}}ik}}$ denote the cognitive expectation, the cognitive entropy and the cognitive hyper-entropy of learner #i on skill #k, respectively, and w is the influence coefficient of the hyper-entropy of the cognitive cloud. The value of w is discussed in Subsect. 7.2 in detail.

6.2 Diagnose learner’s mastery of item

As mentioned before, the learner’s proficiency on the skills is measured in the form of fuzzy interval numbers since the skill proficiency is fuzzy and uncertain. Similarly, the learners’ mastery of the items is also fluctuating in practice. Inspired by FuzzyCDF (Liu et al. 2018), FC-CDF also assumes as follows: (1) The learners’ mastery of the objective and subjective items is conjunctive; namely, one learner could answer one objective item correctly only if he/she masters all the skills required by the objective item; (2) the learners’ mastery on the subjective items is compensatory; namely, the more skills required by the subjective items one learner has mastered, the higher score that he/she would get for this subjective item. Then, the fuzzy intersection and fuzzy union of fuzzy theory (Klir and Yuan 1995) are introduced to transform the conjunctive and compensatory assumptions into mathematical formulas. To be specific, one learner’s mastery of one subjective/objective item is the fuzzy intersection/union of his/her skills proficiency required by this item. Thus, the mastery of learners on the items could be defined as follows:

$$ \eta_{ij}^{{\text{L}}} = \bigcap\limits_{{1 \le k \le K,q_{jk} = 1}} {\alpha_{k}^{{\text{L}}} \left( i \right)} ,\;\eta_{ij}^{{\text{H}}} = \bigcap\limits_{{1 \le k \le K,q_{jk} = 1}} {\alpha_{k}^{{\text{H}}} \left( i \right)} $$

(8)

$$ \eta_{ij}^{{\text{L}}} = \bigcup\limits_{{1 \le k \le K,q_{jk} = 1}} {\alpha_{k}^{{\text{L}}} \left( i \right)} ,\;\eta_{ij}^{{\text{H}}} = \bigcup\limits_{{1 \le k \le K,q_{jk} = 1}} {\alpha_{k}^{{\text{H}}} \left( i \right)} , $$

(9)

In the above equations, $\cap$ and $\cup$ represent the fuzzy intersection and union operators of the fuzzy theory, respectively. They are defined as follows:

$$ \left( {A \cap B} \right)\left( x \right) = \min \left( {A\left( x \right),B\left( x \right)} \right) $$

(10)

$$ \left( {A \cup B} \right)\left( x \right) = \max \left( {A\left( x \right),B\left( x \right)} \right) $$

(11)

Based on the fuzzy intersection/union operation, one learner’s mastery of one objective/subjective item could be the minimum/maximum of his/her skills proficiency required by the item.

6.3 Estimate item parameters

Intuitively, the learners with good item mastery give a correct response to the item and the ones with bad item mastery give a wrong response with a high probability. Nevertheless, it is possible that the learners with good item mastery might give a wrong response to the item by carelessness, and those with bad item mastery could correctly answer the item by a lucky guess. Thus, the existing research, such as the DINA model and FuzzyCDF, introduces the slip factor (s) and the guess factor (g) shown in Eqs. (3) and (4), respectively. The expectation–maximization (EM) algorithm and the Markov chain Monte Carlo (MCMC) algorithm (Chakraborty 2002) are often used in CDMs for parameter estimation, and MCMC performs better for multi-parameter estimation than EM does. In this section, an effective training algorithm using MCMC is proposed to estimate the slip and guess factors of each test item and the variance of the normalized score of learners on subjective test items for FC-CDF. By referring to the parameter settings of HO-DINA (Torre and Douglas 2004), the prior distribution of the parameters in FC-CDF is given as follows:

$$ \begin{gathered} s_{j} \sim{\text{Beta}}\left( {m_{s} ,n_{s} ,\min_{s} ,\max_{s} } \right), \hfill \\ g_{j} \sim{\text{Beta}}\left( {m_{g} ,n_{g} ,\min_{g} ,\max_{g} } \right), \hfill \\ \frac{1}{{\sigma^{2} }}\sim{\Gamma }\left( {x_{\sigma } ,y_{\sigma } } \right), \hfill \\ \end{gathered} $$

(12)

where Beta(m, n, min, max) is a four-parameter Beta distribution, which has two shape parameters m and n and is supported on the range [min, max]; ${\Gamma }(x_{\sigma } ,y_{\sigma } )$ is a Gamma distribution which has shape parameter x_σ and scaling parameter y_σ. Then, the joint posterior distribution of s, g and σ² given the score matrix (i.e., X matrix) is as follows:

$$ P\left( {s,g,\sigma^{2} |X} \right) \propto L\left( {s,g,\sigma^{2} } \right)P\left( s \right)P\left( g \right)P\left( {\sigma^{2} } \right) $$

(13)

where L is the joint likelihood function of FC-CDF as follows:

$$ L\left( {s,g,\sigma^{2} } \right) = L_{{\text{o}}} \left( {s,g,\sigma^{2} } \right)L_{{\text{s}}} \left( {s,g,\sigma^{2} } \right) $$

(14)

where L_o and L_s denote the joint likelihood functions of objective and subjective items, respectively. Based on Eqs. (12)–(14), L_o and L_s are defined by

$$ L_{{\text{o}}} \left( {s,g,\sigma^{2} } \right) = \mathop \prod \limits_{i}^{I} \mathop \prod \limits_{j}^{{P_{{\text{o}}} }} \left( {F_{ij} } \right)^{{X_{ij} }} \left( {1 - F_{ij} } \right)^{{1 - X_{ij} }} $$

(15)

$$ L_{{\text{s}}} \left( {s,g,\sigma^{2} } \right) = \mathop \prod \limits_{i}^{I} \mathop \prod \limits_{j}^{{P_{{\text{s}}} }} N\left( {X_{ij} |F_{ij} ,\sigma^{2} } \right) $$

(16)

In Eqs. (18) and (19), σ² is the variance of the normalized score of learners on subjective test items, and $N\left( { \cdot |\mu ,\sigma^{2} } \right)$ denotes the probability density function of a Gaussian distribution with mean μ and variance σ². The full conditional distributions of each parameter are as follows:

$$ P\left( {s,g|X,\sigma^{2} } \right) \propto L\left( {s,g,\sigma^{2} } \right)P\left( s \right)P\left( g \right) $$

(17)

$$ P\left( {\sigma^{2} |X,s,g} \right) \propto L\left( {s,g,\sigma^{2} } \right)P\left( {\sigma^{2} } \right) $$

(18)

Finally, a Metropolis–Hastings (M-H)-based MCMC method for parameter estimation is proposed by Algorithm 1. First, each parameter is initialized according to the initial distribution settings. For each iteration, a uniformly random sample of each parameter is drawn within a predefined interval specified by $\delta_{s} ,\delta_{g}$ and $\delta_{{\sigma^{2} }}$, and then, the full conditional probability of item slip factor s, item guess factor g and the variance of normalized scores of subjective items σ² are computed. Next, the acceptance probability of samples can also be calculated based on the M-H algorithm. Each sample will accept the transfer if the acceptance probability is greater than the random number generated from the uniform distribution U(0, 1); otherwise, it will not accept the transfer. In this way, the parameters could be estimated after W iterations of sampling. In addition, to simplify the calculation, $P\left( s \right),P\left( g \right)$ and $P\left( {\sigma^{2} } \right)$ are equal to the probability density functions of their corresponding distributions.

6.4 Predict learner’s performance

Considering that learners’ mastery of the skills is fluctuant and uncertain, the learners’ performance in the items should be also fluctuant and uncertain. Inspired by FuzzyCDF (Liu et al. 2018), the learners’ performance in both objective and subjective test items can be predicted based on fuzzy interval numbers by

$$ F_{ij}^{{\text{L}}} = \left( {1 - s_{j} } \right)\eta_{ij}^{{\text{L}}} + g_{j} \left( {1 - \eta_{ij}^{{\text{L}}} } \right) $$

(19)

$$ F_{ij}^{{\text{H}}} = \left( {1 - s_{j} } \right)\eta_{ij}^{{\text{H}}} + g_{j} \left( {1 - \eta_{ij}^{{\text{H}}} } \right) $$

(20)

$$ F_{ij} = \frac{{F_{ij}^{{\text{L}}} + F_{ij}^{{\text{H}}} }}{2} $$

(21)

where $F_{ij}^{{\text{L}}}$ and $F_{ij}^{{\text{H}}}$ denote the lower limit and the upper limits of learner #i’s performance on item #j, respectively; F_ij is the predicted score of learner #i on item #j.

6.5 Deal with data deficiency problem

In practice, when the learners just start to learn a course, they have only completed a handful of test items. In this case, only a small amount of data about learners’ score profiles are available, which will lead to the limited accuracy of cognitive diagnosis and obviously influence the performance of prediction. Some experiments have confirmed it. For example, in the real-world dataset Math2, which is introduced in Sect. 7.1, skill #2 is only required by item #2, i.e., there is only one score profile to depict learner’s cognitive status on skill #2. Then, if one learner’s response to item #2 is correct/wrong, his/her cognitive cloud on skill #2 would be [1, 0, 0]/[0, 0, 0] in FC-CDF or his skill proficiency on this skill would be 1/0 in other CDMs. Obviously, the limited sample data increase difficulty to cognitive diagnosis. Aiming at the possible data deficiency problem, two strategies are proposed as follows:

• 1) If the number of learner #i’s score profiles on skill #k is less than 4, i.e., 0 < $\left| {X_{ik}^{^{\prime}} } \right|$< 4, the whole score profiles of learner #i will be regarded as input data to compute the cognitive cloud $C^{{{\text{C}}ik}}$.

• 2) If learner #i has not answered the item, which requires skill #k, i.e., $\left| {X_{ik}^{^{\prime}} } \right|$= 0, aiming at the possible data deficiency, FC-CDF would combine PMF to guarantee the performance of examinee performance prediction. PMF was proposed by Ruslan et al. (Mnih and Salakhutdinov, 2007) to deal with the prediction tasks in the large, sparse and imbalanced datasets. By integrating the PMF method, the prediction result is computed as follows:

  $$ F_{ij}^{*} = \left( {1 - \lambda } \right)\frac{{F_{ij}^{{\text{L}}} + F_{ij}^{{\text{H}}} }}{2} + \lambda F_{ij}^{{{\text{PMF}}}} $$

where $F_{ij}^{{{\text{PMF}}}}$ denotes the performance of learner #i on item #j predicted via PMF; $\lambda$ is the adjustment coefficient. $\lambda$ = 1 when $\left| {X_{ik}^{^{\prime}} } \right|$= 0, otherwise = 0.

7 Experiments

To verify the effectiveness of our FC-CDF, we conduct experiments to answer the following research questions (RQs):

RQ1: What is an appropriate value of the coefficient w when we diagnose learner’s skill proficiency via cognitive clouds?

RQ2: How about the performance of FC-CDF on score prediction compared with the existing research?

RQ3: How about the performance of FC-CDF on execution efficiency compared with the existing research?

All procedures were implemented by Python 3.6 on a Linux server with a 2.3 GHz Intel Xeon CPU and 16 GB memory.

7.1 Experimental setup

7.1.1 Experimental datasets

The experiments use three real-world datasets, a simulation dataset and three extended datasets to verify the prediction performance and execution efficiency of FC-CDF. These datasets are introduced as follows:

• (1) The real-world datasets: They are FrcSub (DeCarlo 2011), Math1 and Math2. FrcSub records the score profiles committed by junior high school students about fraction subtraction objective test items. Math1 and Math2 are actually two independent subsets of Math2015,¹ and widely used in existing research, e.g., FuzzyCDF (Liu et al. 2018) and R-FuzzyCDF (Li et al. 2017). Math1 and Math2 are collected from two final mathematical examinations, including both objective and subjective test items, for high school students. Each dataset contains a normalized score matrix X and a Q-matrix.

• (2) The Simulation dataset: To verify the performance of FC-CDF in large-scale data environment, Simulation dataset² is generated. In the dataset, there are 1,000 learners, 100 items including 80 objective items and 20 subjective items, and 10 skills. All the learners answered all the items, and their score profiles obey the standard norm distribution. Each objective item requires 2 to 3 skills randomly, and each subjective item requires 3 to 5 skills randomly. Every skill is required by more than 8 items.

• (3) Three extended datasets (i.e., Vir_FrcSub, Vir_Math1, Vir_Math2): The three extended datasets³ are generated based on FrcSub, Math1 and Math2, respectively, to analyze the FC-CDF’s performance in a large-scale data environment. The data of every extended dataset contain two parts, i.e., the original real-world dataset and the simulated data. The extended data are constructed prudently. Taking Vir_Math1 as an example, the Q-matrix of Vir_Math1 is the same as Math1’s, and the size of the simulated data part of Vir_Math1’s score matrix is the same as that of Math1. In the simulated data part of Vir_Math1’s score matrix, for the objective items, the score profiles of each objective item are the same as Math1’s. As for subjective items, suppose the subjective item score matrix of Math1 and Vir_Math1 is defined as $R_{{\text{s}}} = \left[ {r_{ij}^{{\text{s}}} } \right]_{{I \times J_{{\text{s}}} }}$ and $VR_{{\text{s}}} = \left[ {vr_{ij}^{{\text{s}}} } \right]_{{I \times J_{{\text{s}}} }}$, respectively, where J_s denotes the number of subjective test items, and $r_{ij}^{{\text{s}}}$ and $vr_{ij}^{{\text{s}}}$ are the performance of learner #i on subjective test item #j in Math1 and Vir_Math1, respectively, $vr_{ij}^{{\text{s}}} \sim U\left( {r_{ij}^{{\text{s}}} - 0.1,r_{ij}^{{\text{s}}} + 0.1} \right)$.

The preview of the Q-matrix and the brief summary of all datasets are shown in Fig. 4 and Table 2, respectively. Figure 4 demonstrates those test items related to every skill. According to Fig. 4, these datasets fall into 2 categories, i.e., the symmetric datasets and asymmetric datasets. In the symmetric datasets (e.g., FrcSub, the simulation dataset), every skill is required by more than 1 test item. In the asymmetric datasets (e.g., Math1, Math2, Vir_Math1, Vir_Math2), some skills might be required by only 1 test items. For example, in Math1 and Vir_Math1, skill #4 is required by only item #11.

Fig. 4

Preview of Q-matrix of all datasets

Table 2 Summary of datasets information

7.1.2 Baseline approach and parameter settings

The baseline approaches and their corresponding parameter settings are shown as follows:

• (1) PMF (Mnih and Salakhutdinov 2007): It maps the learners and items into low-dimensional latent vectors for performance prediction. The dimension of the latent vector is set as 10.

• (2) DINA (Torre 2008; Torre and Douglas 2004): It is one of the most representative CDMs which uses binary vectors with multi-dimension to represent learners’ skill proficiency. It could predict learners’ performance by combing the slip and guess factors of items. In this paper, we use two methods to estimate the parameters of the DINA model. The DINA model using the MCMC method for parameter estimation is denoted as MCMC-DINA, and the DINA model with the EM method for parameter estimation is denoted as EM-DINA.

• (3) FuzzyCDF (Liu et al. 2018): It is the first approach that applies fuzzy theory to cognitive diagnosis and address the problem of conducting cognitive diagnosis on subject items. Its parameter setting follows the original paper.

• (4) R-FuzzyCDF (Li et al. 2017): It improves the performance of FuzzyCDF by introducing the importance of skills. The parameter setting of R-FuzzyCDF follows its original paper.

• (5) NeuralCDM (Wang et al. 2020) It is one of the most representative CDMs that apply neural networks to cognitive diagnosis. Its parameter setting follows the original paper.

For our FC-CDF, the hyperparameters of Algorithm1 are set by referring to HO-DINA (Torre and Douglas 2004) as follows:

$$ m_{s} = 1,n_{s} = 2,\min_{s} = 0,\max_{s} = 0.6,\;m_{g} = 1,n_{g} = 2,\min_{g} = 0,\max_{g} = 0.6,x_{\sigma } = 4,y_{\sigma } = 6,\;w = 0.1 $$

Referring to FuzzyCDF (Liu et al. 2018), the number of iterations of Algorithm 1 is set to 5000 and the parameters are estimated based on the last 2,500 samples to guarantee the convergence of the Markov chain. Moreover, the setting of w will be discussed in Subsect. 7.2.

7.1.3 Evaluation metrics

The root-mean-square error (RMSE) and the mean absolute error (MAE) are used as the metrics as follows:

$$ {\text{RMSE}} = \sqrt {\frac{1}{N}\mathop \sum \limits_{i = 1}^{N} \left( {F_{{U_{i} }} - X_{{U_{i} }} } \right)^{2} } $$

(22)

$$ {\text{MAE}} = \frac{1}{N}\mathop \sum \limits_{i = 1}^{N} \left| {F_{{U_{i} }} - X_{{U_{i} }} } \right| $$

(23)

where N is the number of learners in the testing dataset; $F_{{U_{i} }}$ denotes the predicted performance of learner U_i; and $X_{{U_{i} }}$ represents the real performance of learner U_i.

The learners’ performance in items is predicted by Eqs. (14)–(18) after obtaining the slip and guess factors of each item through the model training. Note that the predicted results are continuous, for objective items, they are supposed to be discretized by

$$ F_{ij}^{*} = \left\{ {\begin{array}{*{20}l} {1 \quad if\, F_{ij} \ge {\text{TH}}_{F} } \\ {0\quad {\text{ otherwise}}} \\ \end{array} } \right. $$

where $F_{ij}^{*}$ is the discrete predicted score of learner #i on objective item #j; TH_F is the threshold, which is usually set as 0.5.

7.2 Value of w (RQ1)

In FC-CDF, w is a trade-off parameter to balance the influence of the cognitive hyper-entropy on learners’ skill proficiency. The experiments use FrcSub and simulation dataset to find a reasonable value of w, which is set from the range [0.0, 1.0], and the performance of FC-CDF with different w is verified via a fivefold cross-validation method. The results are shown in Fig. 5.

Fig. 5

Prediction performance with different values of w

As shown in Fig. 5, FC-CDF performs the best in FrcSub and the simulation dataset when w is set as 0.1. Moreover, with the increase of w, the performance of FC-CDF decreases, especially in the small-scale dataset (i.e., FrcSub). The reason is that large w would amplify the fuzziness and uncertainty of cognitive cloud, which leads to inevitable errors in cognitive diagnosis. Based on the above analysis, 0.1 is used as the suggested value of w.

7.3 Prediction performance analysis (RQ2)

7.3.1 Performance analysis in real-world and simulation datasets

All approaches are verified by a fivefold cross-validation method for a fair comparison, i.e., 80% of the score profiles of each learner are selected randomly as the training set, and the rest 20% for testing. Moreover, we also conduct a one-sample t-test to compare our FC-CDF with the strongest baseline approaches. The results are shown in Fig. 6 and Table 3. Considering that the performance of MCMC-DINA is like that of EM-DINA, only EM-DINA is displayed in Fig. 6 for easy reading.

Fig. 6

Performance comparison on each item

Table 3 Performance comparison

From Fig. 6 and Table 3, the analysis is given as follows:

• (1) Compared with the classic approaches, i.e., PMF and DINA, FC-CDF performs optimally in all cases. By introducing cognitive cloud and interval numbers, FC-CDF could depict learners’ cognitive status comprehensively and accurately predict their performance on the items.

• (2) Compared with the fuzzy theory-based approaches, i.e., FuzzyCDF and R-FuzzyCDF, our FC-CDF outperforms them in the symmetric datasets (i.e., FrcSub and simulation datasets). For the asymmetric datasets (i.e., Math1 and Math2), the cognitive clouds in FC-CDF suffer from data deficiency problems on some items because some skills are checked by only 1 test item. Thus, the performance of FC-CDF is close to that of FuzzyCDF. However, as mentioned before, since FuzzyCDF needs to estimate more parameters than FC-CDF, the subsequent experiments confirm that the execution time of FuzzyCDF is one hundred times more than that of FC-CDF.

• (3) Compared with the neural networks-based approach, i.e., NeuralCDM, there is a gap between NeuralCDM and FC-CDF in the real-world datasets. The performance of NeuralCDM declines because that neural networks need more training data, while the small scale of a dataset (e.g., FrcSub) may cause the overfitting for NeuralCDM. It should be noted that, the cross-entropy was chosen as the loss function in the original paper of NeuralCDM. Obviously, it is not a perfect choice to train a model for performance prediction tasks when dealing with the datasets consisting of both objective and subjective items. NeuralCDM might perform better if tailored loss functions are adopted.

• (4) In the FrcSub dataset, the p value < 0.05 indicates the improvement of our FC-CDF is statistically significant. While in other datasets, the differences between FC-CDF and FuzzyCDF are not statistically significant. However, as mentioned before, FuzzyCDF consuming a lot of time to estimate parameters sacrifices efficiency for performance improvement. It is difficult to apply FuzzyCDF to real e-learning applications, while our FC-CDF could provide both competitive prediction accuracy and satisfactory diagnosis efficiency. The efficiency analysis is discussed in Subsect. 7.4 in detail.

7.3.2 Performance analysis in extended datasets

The following experiments use Vir_FrcSub, Vir_Math1 and Vir_Math2 datasets, and FC-CDF is compared with six approaches including PMF, MCMC-DINA, EM-DINA, FuzzyCDF, R-FuzzyCDF and NeuralCDM. Different percentages of the simulated data are mixed with the corresponding real-world data. The performance of all the approaches is verified by a fivefold cross-validation method to guarantee the fairness of the comparison. The experimental result is shown in Fig. 7, where the x-axis represents the percentage of simulated data, and the y-axis denotes the average RMSE or MAE values of the prediction performance in the test dataset.

Fig. 7

Performance comparison in extended datasets

Obviously, Fig. 7 demonstrates similar results to Fig. 6. FC-CDF outperforms other approaches in the symmetric datasets (i.e., Vir_FrcSub), while for the asymmetric datasets (i.e., Vir_Math1 and Vir_Math2), the predicted results of FuzzyCDF and R-FuzzyCDF are a bit better than that of FC-CDF. However, FuzzyCDF and R-FuzzyCDF need more training time than FC-CDF. For instance, compared with FuzzyCDF and R-FuzzyCDF on Vir_Math1, when the percent of added simulation data reaches 90%, the RMSE value of FC-CDF declines by 1.14% and 0.79%, respectively, and the training time of FuzzyCDF and R-FuzzyCDF are 144.29 and 223.99 times longer than FC-CDF’s, and more analysis about the execution time will be mentioned in the next section.

7.4 Execution time analysis (RQ3)

To support the learners’ autonomic learning online in their fragmented time, the e-learning platform is required to diagnose the learners’ cognitive status quickly for recommending suitable exercises to them. In this section, the experiments are used to analyze the execution efficiency of FC-CDF. The experiments are executed on a Linux server with a 2.3 GHz Intel Xeon CPU and 16 GB memory. This server only runs our experimental program.

7.4.1 Execution time in real-world and simulation datasets

FC-CDF is compared with the six approaches including PMF, MCMC-DINA, EM-DINA, FuzzyCDF, R-FuzzyCDF and NeuralCDM. The training time of every approach in three real-world datasets and a simulation dataset is recorded, and the number of iterations of MCMC sampling is set to 5000. Moreover, we introduce a Tesla-T4 16G GPU to boost the training speed of NeuralCDM, the number of iterations of the training process is set as 5 for NeuralCDM. Table 4 shows the results.

Table 4 Execution time analysis in real-world and simulation datasets (seconds)

From Table 4, the analysis is given as follows:

• (1) FC-CDF outperforms the other MCMC-based baseline approaches (e.g., R-FuzzyCDF, MCMC-DINA). More importantly, FC-CDF performs much better than FuzzyCDF. In the asymmetric datasets (i.e., Math1, Math2), the execution time of FuzzyCDF is 102.96 and 138.16 times as long as FC-CDF’s, respectively, and in the symmetric datasets (i.e., FrcSub, Simulation), FC-CDF still beats FuzzyCDF. The reason is that FC-CDF only uses the MCMC algorithm to estimate the item parameters (i.e., the slip and guess factors), while for other approaches, MCMC is utilized to estimate more parameters. For example, R-FuzzyCDF estimates the learners’ traits and skills’ difficulties via MCMC, and MCMC-DINA estimates the learners’ skills proficiency via MCMC.

• (2) Although we used a GPU to boost the training speed for NeuralCDM, the execution time of NeuralCDM is 20.33 and 19.65 times as long as FC-CDF’s in Math1 and Math2 datasets, respectively.

• (3) In terms of execution time, although the gap between PMF and our FC-CDF is small, the lack of interpretability and limited performance on score prediction make PMF hard to compete with our FC-CDF.

• (4) EM-DINA is obviously better than FC-CDF in FrcSub, Math1 and simulation datasets, while FC-CDF outperforms EM-DINA in Math2 dataset. The reason is that the EM-DINA’s complexity is exponential with the number of skills. In the FrcSub, Math1 and simulation datasets, the number of skills is 8, 11 and 10, respectively. Among these datasets, EM-DINA could enumerate at most 2¹¹ possible patterns of skill proficiency for one learner and find the most suitable skill proficiency for one learner. However, the number of skills is 16 in the Math2 dataset. In this case, EM-DINA should enumerate 2¹⁶ possible patterns of skill proficiency for each learner, which is more time-consuming than the performance on the former datasets.

Taking EM-DINA’s performance in Math1 and Math2 as an example, when the number of skills increases from 11 to 16, the execution time of EM-DINA in Math2 is 12.49 times as long as that in Math1. Even there are fewer learners in Math2 than Math1. As for FC-CDF, its execution time does not grow fast with the increasing skill numbers like EM-DINA. It could be foreseen that FC-CDF would outperform EM-DINA in terms of efficiency when the number of skills is more than a certain number. To verify this, a series of simulation datasets are constructed. In these datasets, the number of learners and items are set as 1000 and 100, respectively; the number of skills is set from the range [10, 22]; and the training time of FC-CDF and EM-DINA in every simulation dataset is recorded. Considering that the specific values of the score matrix and Q-matrix make no difference to the training time, their values are randomly generated in these datasets. The results are shown in Table 5 and Fig. 8.

Table 5 Execution time analysis in simulation datasets (seconds)

Fig. 8

Execution time comparison in simulation datasets

From Table 5 and Fig. 8, it could be observed that there is a surge in EM-DINA when the number of skills increases from 18 to 22, while our FC-CDF performs stably. In practice, a course (e.g., “Java Programming”) usually covers more than 20 skills, even 30 skills. In this case, the execution time of EM-DINA would be unacceptable. Compared with the baseline approaches, FC-CDF is a preferable cognitive diagnosis framework for practical e-learning scenarios.

7.4.2 Execution time in extended datasets

In the following experiments, the extended datasets (i.e., Vir_FrcSub, Vir_Math1 and Vir_Math2) are used to verify the execution efficiency of FC-CDF in larger-scale data. Different percentages from 10 to 100% of the simulation dataset are added. The execution time of training each approach in each extended dataset is recorded, and the number of iterations of MCMC sampling is set to 5000. The results are shown in Table 6.

Table 6 Execution time analysis in extended datasets (seconds)

Obviously, compared with other MCMC-based approaches (i.e., MCMC-DINA, FuzzyCDF and R-FuzzyCDF), FC-CDF significantly improves the execution efficiency of the training process. For example, in Vir_Math2, when the percent of added simulation data reaches 100%, the execution time of MCMC-DINA, FuzzyCDF and R-FuzzyCDF is 47.59, 144.25 and 229.90 times longer than that of FC-CDF, respectively. More importantly, the gap between FC-CDF and EM-DINA is getting smaller with the increase in learners in the Vir_Math2 dataset. When the percent of added extension data is more than 60%, FC-CDF beats EM-DINA because the EM-DINA’s complexity is exponential with the number of skills.

7.5 Experimental summary

In experiments, the prediction performance and execution efficiency of FC-CDF are compared with those of the other six approaches. By introducing cognitive cloud and fuzzy interval numbers, FC-CDF could improve the prediction accuracy in the symmetric datasets (i.e., FrcSub and simulation dataset) and obtains satisfactory prediction accuracy in the asymmetric datasets (i.e., Math1, Math2, Vir_Math1 and Vir_Math2). Most importantly, FC-CDF greatly reduces the training time of the cognitive diagnosis by employing an efficient RCG algorithm, which quickly recommends suitable exercises to learners in a new round of practice according to their latest score profiles. Although all the real-world data used in the experiments are only related to mathematics, it is worth mentioning that the proposed approach is easily generalized to other subjects or courses. In summary, FC-CDF provides good prediction accuracy and decreases the execution time in a large-scale data environment involving in many learners, test items and skills, which strongly supports the learners’ autonomic learning online in their fragmented time.

8 Conclusion and discussion

The current CDMs employ a simple measurement mechanism to measure the learners’ skill proficiency with low computational efficiency in a large-scale data environment. This paper proposed a new cognitive diagnosis framework FC-CDF, which takes advantage of both the cloud model theory and the fuzzy interval number. The learners’ skills proficiency is modeled as cognitive clouds via RCG, and the learners’ skills proficiency is measured by transforming each cognitive cloud into fuzzy interval numbers. Then, the mastery of learners on the items is simulated by mapping conjunctive and compensatory interactions into the fuzzy set operations. Based on it, the learners’ performance could be predicted by considering the slip and guess factors of each item. Finally, aiming at the data deficiency problem, PMF is integrated into FC-CDF to deal with the examinee performance prediction task. The experimental results demonstrate that the proposed approach via FC-CDF significantly reduces the execution time of cognitive diagnosis and provides good accuracy of learners’ performance prediction.

There are still some open issues, which can be explored in the future: (1) In a big data environment, the neural networks could be utilized to mine the latent attributes of learners and test items for more accurate cognitive modeling. Based on the neural networks, it is possible to incorporate the cloud model theory and fuzzy interval numbers to measure the fuzziness and uncertainty of cognitive status more exactly. (2) Based on the cognitive status and predicted scores via FC-CDF, the personalized learning resources recommendation, collaborative learning team formation and learning path recommendation could be achieved.