1.1 Revised Bloom’s Taxonomy

Multiple taxonomies have been developed to provide structured frameworks for articulating the mental processes elicited by assessment tasks and to guide their design, classification, and analysis (Irvine, 2021). Bloom’s (1956) pioneering cognitive taxonomy was originally intended to assist teachers in formulating lesson plans, learning objectives, and assessment instruments. The taxonomy conceptualized cognitive complexity as a cumulative hierarchy consisting of six levels, ranging from simple to complex, knowledge, comprehension, application, analysis, synthesis, and evaluation, with each level building upon the preceding one (Krathwohl, 2002). Bloom’s framework played a key role in standardizing the design of cognitive demand within item development across institutions. It has since been widely applied across educational levels for various purposes, including the creation of test items, learning objectives, instructional guidance, and assessment types (Krathwohl, 2002).

However, the taxonomy has faced criticism for its unidimensional structure, broad generalizations of cognitive dimensions, and lack of clear guidance for determining cognitive levels (Krathwohl, 2002). For instance, the overlap between knowledge and comprehension complicates item classification (Lemons & Lemons, 2013; Marzano & Kendall, 2007). Additionally, its rigid hierarchical order is problematic. For example, some evaluations can occur without synthesis skills, despite evaluation being positioned at the highest level (Crowe et al., 2008). These critiques prompted revisions to the arrangement of upper cognitive levels (Kim et al., 2012), culminating in the development of a revised version.

The Revised Bloom’s Taxonomy (RBT) was created in 2001 when Anderson and Krathwohl established a two-dimensional framework comprising the cognitive and knowledge dimensions. Similar to the original taxonomy, the cognitive dimension is arranged hierarchically, progressing from simple to complex but expressed in verb form: remembering, understanding, applying, analyzing, evaluating, and creating. The knowledge dimension is arranged from concrete to abstract: factual, conceptual, procedural, and meta-cognitive. This version emphasized classifying educational objectives as two-dimensional guidelines (Krathwohl, 2002) that could enhance teaching quality by helping teachers understand the curriculum, plan instruction, and design assessments that align with the objectives better (Anderson & Krathwohl, 2001).

The hierarchical order of the six levels is distinguished by degrees of difficulty (Marzano & Kendall, 2007). In this version, the levels were renamed using verbs to reflect the active nature cognitive processes, and the top two levels were changed from synthesis and evaluation to evaluating and creating. The change highlighted that while typically requires both critical and creative thinking, evaluation may not always involve synthesis skills (Crowe et al., 2008).

RBT features prominently in Malaysia’s educational policies and official documents. It is the recommended framework for developing assessment items and instruments in the Malaysia Education Blueprint (MOE, 2013). Its six levels describe student performance and inform performance indicators in the Dokumen Standard Kurikulum dan Pentaksiran (Standard Curriculum and Assessment Document; MOE, 2017). Additionally, this taxonomy distinguishes between lower-order thinking skills (LOTS) and higher-order thinking skills (HOTS), with the top four levels, applying, analyzing, evaluating, and creating, representing HOTS (MOE, 2016). The latest guidebook for creating HOTS-level items, Panduan Pembinaan Item Kemahiran Berfikir Aras Tinggi (Guide to Developing Higher-Order Thinking Items), uses these top four levels to guide item development (Lembaga Peperiksaan, 2023). Given its prominence in policy documents, it is unsurprising that teachers are more familiar with this framework. However, such widespread adoption may lead teachers to regard it as the primary or even exclusive framework for instructional practices. Numerous local studies on item classification and development have also favored the use of RBT (Abu Hassan et al., 2020; Aisyah et al., 2019; Hassan et al., 2017; Jamalludin & Zamzuri, 2018; Singh & Shaari, 2019; Tee et al., 2012; Zaiful Shah & Zakaria, 2024; Zulkifli & Abidin, 2022). The taxonomy’s verb lists are particularly appealing to teachers, as they offer practical guidance for lesson planning and assessment design. Consequently, RBT continues to be widely preferred and extensively used.

This version has been adopted and implemented across diverse countries, regardless of sociological and political contexts, and across all educational levels and subjects. RBT has been applied to analyze and classify science curricula objectives and standards in different countries, such as South Korea and Singapore (Lee et al., 2015), Mainland China, Taiwan, Macao, and Hong Kong (Wei & Ou, 2019), and chemistry curricula in Czechia, Finland, and Türkiye (Elmas et al., 2020). Internationally, RBT is widely applied in the creation and classification of assessment items across education systems (Lemons & Lemons, 2013; Motlhabane, 2017; Phillips et al., 2013; Zulkifli & Abidin, 2022).

Nonetheless, RBT continues to face several challenges. Marzano and Kendall (2007) argued that it lacks comprehensive tools necessary to assess higher-order thinking. For example, some verbs may not align with the intended cognitive demands, leading to misinterpretation (Zulkifli & Abidin, 2022). Teachers have commonly used RBT to classify cognitive demand due to its detailed verb lists that aid in item development and categorization (Lemons & Lemons, 2013). However, this reliance can be limiting, especially when inappropriate verbs are used to represent the intended cognitive processes (Thompson, 2008). Therefore, the ability to evaluate the relevance of verbs is essential for accurately understanding cognitive demand (Sangodiah et al., 2014). Teachers must independently evaluate whether the selected verbs reflect the intended level of thinking, as there is no direct association between verbs and cognitive processes (Larsen et al., 2022). Understanding the taxonomy’s structure alone is insufficient; teachers must also grasp the intent behind each question. This is particularly important in crafting MCQ s that assess more than simple recall (Thompson, 2008).

The limitations that led to the development of the RBT prompted the emergence of alternative taxonomies, such as the Structure of the Observed Learning Outcome (SOLO) taxonomy (Biggs & Collis, 1982), Marzano and Kendall’s (2007) New Taxonomy, and Webb’s (1997, 1999) Depth of Knowledge. Research has shown that frameworks like SOLO provide a more detailed structure for developing and assessing students’ critical thinking (Lee et al., 2017). Unlike RBT, the SOLO taxonomy provides a clearer means of evaluating different levels of cognitive processing, making it a valuable tool for both assessment and instruction.

1.2 The SOLO Taxonomy

The SOLO taxonomy (Biggs & Collis, 1982) emerged as a neo-Piagetian model (Piaget, 1950), due to its constructivist roots and its description of learning as a developmental progression. The SOLO taxonomy identified four meaningful levels of cognitive demand or capability: unistructural, multistructural, relational, and extended abstract. This framework considers at least four structural and psychological factors in determining the learner’s cognitive development or the cognitive demand of tasks or products (Hattie & Purdie, 1998).

In terms of psychological processes, increasing levels within the SOLO taxonomy require increasing demand on working memory or attention span: Unistructural tasks require attention to one element, multistructural to two or more elements, relational to the connection among two or more elements, and extended abstract to generalization beyond given information. The SOLO taxonomy also addresses two opposing cognitive needs: consistency and closure. The drive for closure may lead a learner to ignore important information in order to reach an answer more quickly, resulting in a lower-level response. In contrast, a drive for consistency may prompt greater effort to ensure a response aligns with the demands of a task or prompt.

In parallel, structural characteristics of a task influence the outcome produced by the learner. The size of the task (e.g., list three causes of air pollution) determines the cognitive demand and shapes the parameters of the response. The qualitative shift from listing objects, unistructural or multistructural, requires learners to focus on how the various elements in a task space relate to one another. In this sense, the SOLO taxonomy presents a progressive hierarchy that connects task demand with psychological resources, motives, and individual goals. It suggests that cognitive development is dynamic and context-dependent, progressing at different rates across individuals (Biggs & Collis, 1982).

The SOLO taxonomy shifted the focus from cognitive development alone to the complexity of learning outcomes, using students’ tangible and measurable responses to evaluate their development (Biggs & Collis, 1989). In educational settings, particularly assessment, the SOLO taxonomy is well-suited for describing learning complexity as it progresses from surface to deep levels across quantitatively and qualitatively distinct levels.

The SOLO taxonomy has been described as “a language for describing the level and quality of learning both within and across the curriculum” (Biggs & Collis, 1989, p. 151). It serves as a framework for assessing student responses and classifying them based on its four-level structure. This feature helps teachers to enhance student learning by analyzing their responses. For example, science teachers use the SOLO taxonomy to identify and improve problem-solving skills (Collis & Davey, 1986), while English teachers apply it to foster understanding and critical thinking (Ibrahim Badr, 2020). In mathematics, the framework helps describe students’ abilities in mathematical representations (Trapsilasiwi et al., 2020). In higher education, it has served as a tool for scaffolding, assessing, and fostering deep learning (Boulton-Lewis, 1995; Lucander et al., 2010). Beyond classroom teaching, the SOLO framework has been used to classify students’ postings in asynchronous online discussions, helping educators evaluate understanding and engagement (Holmes, 2005). Additionally, it has informed the design of teacher training interventions aimed at improving teaching effectiveness and student outcomes (Hattie, Biggs, & Purdie, 1996; Hattie & Brown, 2004).

Although less widely used in Malaysia (Weay et al., 2016), the SOLO taxonomy has been applied in several studies to analyze student responses in across a variety of subjects and educational levels (Abdul Aziz, 2022; Idek, 2016, 2020; Lim et al., 2010; Lim & Wun, 2011; Mohd Nor & Idris, 2010; Weay et al., 2016), as well as to improve student learning (Idek, 2020; Prakash et al., 2010; Tan, 2006). Nonetheless, after extensive searches across major academic databases, including Web of Science, Scopus, Education Research Complete, and Google Scholar, very few studies were found in Malaysia that specifically applied the SOLO taxonomy for item classification.

1.3 The Relationship between RBT and the SOLO Taxonomy

Both RBT and the SOLO taxonomy follow a hierarchical structure that reflects the principle of growth in learning, from simple to complex. Generally, both frameworks are applicable across all educational levels and subjects, and are useful for planning learning objectives, guiding instructional strategies, and identifying the cognitive demands of assessment tasks. They provide a basis for aligning assessments with intended learning outcomes.

However, empirical studies (see review in Hattie & Purdie, 1998) have indicated that results often differ when applying these taxonomies in cognitive classification (Weay et al., 2016). RBT appears more suitable for thinking about curriculum and learning objectives prior to the assessment stage (Adams, 2015; Anderson & Krathwohl, 2001). Unfortunately, RBT is not designed to track student learning growth, which is essential in science education (Lee et al., 2017). Additionally, some science teachers struggle to recall or explain the higher cognitive levels in the taxonomy (Hassan et al., 2017). Research has also shown that teachers often face challenges with item classification due difficulties interpreting the underlying cognitive processes (Hassan et al., 2017; Omar et al., 2012; Zulkifli & Abidin, 2022). Hence, despite the longevity of RBT in Malaysia, it remains poorly understood by many teachers.

In contrast, the SOLO taxonomy can be applicable in both curriculum design and assessment design and plays a distinct role in evaluating learning outcomes and student understanding (Caridade & Pereira, 2023; Hattie & Brown, 2004). The SOLO taxonomy allows for the independent analysis of student responses, separate from design tasks (Gillies, 2020; Hattie & Purdie, 1998). It is not limited to designing tasks or test items; rather, it is particularly effective in analyzing learning progress, especially in open-ended contexts (Biggs & Collis, 1982). The ability to classify task demand independently from student response makes the SOLO taxonomy especially useful for tracking student progress and learning growth (Biggs & Tang, 2011). Furthermore, its simplicity enables teachers to the levels to provide descriptive feedback and to help students construct more complex and relational responses over time (Chan et al., 2002; Pegg & Tall, 2005).

Although RBT is the required taxonomy for test development in Malaysia, there are good reasons to believe that the SOLO taxonomy might be easier for teachers to use. In New Zealand, high school mathematics and English teachers were able to apply the SOLO taxonomy effectively, after brief training, to identify the cognitive demands of MCQ test items (Hattie & Brown, 2004). Based on these findings, this study examined how accurately Malaysian science teachers could classify the cognitive demand of MCQ s using both RBT and the SOLO taxonomy. Given the novelty of the SOLO taxonomy relative to the RBT, we expected that classification accuracy to be higher with RBT. However, given the simplicity of the SOLO taxonomy, we also expected that basic training would enable comparable levels of classification accuracy. Thus, the study evaluated both taxonomies.

2.1 Research Aims

Given the documented difficulties teachers face in using the RBT to classify science test items, this study was designed to address three key research questions:

1. To what extent do teachers’ ratings of cognitive demand agree when using RBT versus the SOLO taxonomy?

2. How closely do teachers’ classifications align with those made by the research team?

3. Do RBT and SOLO classifications correspond with each other when grouped into broader categories of lower- and higher-order thinking?

2.2 Hypotheses

The study tests the following hypotheses:

• H1. Science teachers will demonstrate moderate to high agreement and inter-rater reliability when classifying the cognitive demands of science MCQ s using the RBT. SOLO taxonomy classifications will yield lower agreement, reflecting its relative unfamiliarity among teachers.

• H2. Science teachers’ classifications will show strong alignment with those of the research team.

• H3. When levels are grouped into lower- and higher-order thinking, there will be a statistically significant relationship between RBT and SOLO taxonomy classifications.

By applying both RBT and the SOLO taxonomy, this study introduces a novel dual-framework approach to evaluating cognitive demand in Malaysian science education, offering new possibilities for improving item design and assessment validity.

3.1 Participants

A convenience sample was recruited from a restricted online Telegram group for Malaysian science teachers, used to share information and teaching materials. Following ethics approval from the institutional board, workshops were conducted via Zoom. Recruitment was limited to teachers with at least 5 years of experience teaching science at the lower secondary level, to ensure sufficient expertise. In total, seven science teachers, all with experience in writing school-level assessment items, were recruited. While all seven participated in the RBT session, only four were available for the SOLO taxonomy session. Only one participant had national-level item-writing experience. Two participants taught in high-performing schools; the others taught in national secondary schools across Malaysia. All panelists held at least a bachelor’s degree; one was pursuing a master’s. All had a minimum of 5 years of teaching experience and had taught science as their primary subject. Session 1 included all seven teachers; Session 2 include four who had also participated in the first session.

3.2 Materials

The study used 30 MCQ s from the 2013 Penilaian Menengah Rendah (Lower Secondary Assessment) (Lembaga Peperiksaan, 2013) and 2019 Penilaian Tingkatan Tiga (Form Three Assessment) (Lembaga Peperiksaan, 2019a). These were the final standardized science assessments before the Penilaian Menengah Rendah was discontinued in 2014 (Kang, 2014) and the Penilaian Tingkatan Tiga was postponed in 2020 and 2021 due to the COVID-19 pandemic (BERNAMA, 2020). Both assessments were developed by the Lembaga Peperiksaan (Examination Syndicate) (Lembaga Peperiksaan, 2019b), making them reliable and valid measures of science knowledge at the end of lower secondary education (Form 3, age 15).

A single set of 30 MCQ s was used to create two versions of an ordered item booklet (OIB). To reflect thematic organization of the curriculum, the items were grouped into four overarching science themes: (1) maintenance and continuity of life (10 items), (2) exploration of elements in nature (11 items), (3) energy and sustainability of life (5 items), and (4) exploration of Earth and space (4 items). Each theme drew on content from learning areas across Forms 1 to 3 (ages 13 to 15). In total, seven items were from the Form 1 syllabus, 15 from Form 2, and eight from Form 3. The assessment items used in this study were drawn from the Ordered Item Booklet (Version 6), which contains a selection of science multiple-choice questions aligned with curriculum standards. The full dataset is openly available and was archived by the author (Sirri, 2025) in the University of Auckland’s Figshare repository.

The two booklets were then structured according to the two theoretical taxonomies. In the RBT version, items were categorized into three difficulty levels: low (k = 14), moderate (k = 12), and high (k = 4). In contrast, the SOLO taxonomy version categorized items by structural complexity into four levels: unistructural (k = 14), multistructural (k = 1), relational (k = 13), and extended abstract (k = 2). The content remained identical across both versions. However, to align with the different theoretical perspectives on taxonomic difficulty, seven items appeared in a slightly different sequence in the SOLO taxonomy booklet (e.g., Item 9 in the RBT version appeared as Item 8 in the SOLO version). Prior to the sessions, expert teachers cross-checked the item order, and minor adjustments were made.

3.3 The Bookmark Procedure

This study employed the bookmark procedure (Mitzel, Lewis, & Green, 2001) for standard setting. The procedure was implemented across two separate sessions, each comprising three structured rounds: individual, small panel, and full panel. In each round, panelists set cut scores to mark transition between taxonomic or difficulty levels and recorded the item numbers on a form.

In the individual round, panelists worked independently and recorded their placements without discussion. In Session 1, seven panelists were split into two groups of three and four. In Session 2, four panelists were divided into two pairs. In the small group round, panelists compared their bookmark placements, discussed their reasoning, and worked towards consensus. The discussion helped clarify discrepancies and refine understanding of item classifications. Finally, in the full panel round, all panelists in the session finalized their placements. Items with disagreement were revisited and justifications provided. This consensus-building process ensured that all viewpoints were considered.

Session 1 used RBT classifications based on difficulty level (low, moderate, and high), while Session 2 used the SOLO taxonomy, which included four levels of structural complexity.

3.4 Protocol

The bookmark procedure was piloted with five qualified Malaysian science teachers. After a brief presentation on the procedure and the two cognitive taxonomies, panelists rated 30 MCQ s from the OIB using a Google Form. The briefing slides received positive feedback, and the form was considered user friendly. Panelists found the timing appropriate but recommended more active facilitation, especially in Round 3.

To avoid confusion between the two taxonomies, sessions were held one week apart, using the same group of panelists. Session 1 used the RBT version, and Session 2 used the SOLO taxonomy version, each with its corresponding OIB. This within-subject design enabled direct comparison of how the same items were classified under both frameworks while controlling for content differences. Each session included a 20-minute taxonomy briefing and three rounds of the bookmark procedure. The full procedure lasted approximately 2 hours pers session, with the lead author facilitating discussions in Rounds 2 and 3.

3.5 Interrater Reliability

Consensus on bookmark placements was defined as at least 67% of panelists assessing the same item to the same level transition (Karantonis & Sireci, 2006). The kappa coefficient (κ) (Cohen, 1960) was calculated to account for the possibility of chance agreement. This was particularly relevant due to the small number of categories used for rating. As the study involved more than two panelists and multiple categories, Fleiss generalized kappa was used to assess interrater reliability (Zapf et al., 2016). Interpretation of κ values followed McHugh (2012): values between 0.60 and 0.80 indicate moderate agreement, while as values above 0.80 indicated strong agreement.

3.6 Cross-Taxonomy Comparison

To compare classifications across two taxonomies, item ratings were grouped into lower- and higher-order categories. A chi-square test was used to evaluate the relationship between the RBT and SOLO taxonomy classifications. Although traditional guidelines recommend a minimum expected cell frequency of five, this threshold is considered arbitrary (Howell, 2011). Simulations by Campbell (2007) showed that chi-square tests remain valid with expected frequencies as low as one. Therefore, this test was considered appropriate given the small sample size and categorical nature of the data.

3.7 Qualitative Insights

Panelist discussions during Rounds 2 and 3 were recorded and transcribed to gain qualitative insight into their reasoning and negotiation processes. Although formal thematic analysis was not conducted, a qualitative content review was performed. Transcripts were manually reviewed, and representative excerpts were selected to illustrate key themes, and shifts in consensus across rounds.

4.1 First Session: RBT

In this session, panelists showed weak to moderate agreement during the transition from individual scoring to small group discussions (see Table 1). In Round 1, individual bookmark placements showed weak agreement across all taxonomic levels (< 67% agreement, κ < 0.60), except for the low-difficulty category. Individual accuracy compared to the research team ranged from 20% to 93% (M = 61%, SD = 27%), indicating high variability in teachers understanding of the difficulty levels within RBT. In Round 2, two small groups of three or four panelists also showed weak agreement across all levels, except for high-difficulty items, which reached perfect agreement. In Round 3, the full panel achieved 100% consensus on difficulty classifications: 13 items were designated as low-difficulty, 11 as moderate, and six as high. This perfect agreement shows the importance group discussions in developing a shared understanding of the difficulty levels defined by the RBT.

View Full Size

Inter-rater agreement on RBT item difficulty classifications by bookmarking round

Citation: Asia-Pacific Science Education 11, 2 (2025) ; 10.1163/23641177-bja10101

• Download Figure
• Download figure as PowerPoint slide

During the full panel discussion, determining cut scores for low- and moderate-difficulty items took longer than for high-difficulty items. This supports findings by Perez et al. (2012), who reported that distinguishing lower cognitive levels, often associated with low- and moderate difficulty items, is more challenging than identifying high-difficulty ones. For example, panelists debated Item 10: Group 1 classified it as low-difficulty, while Group 2 considered it to be of moderate-difficulty. Group 2 argued that the item required “double loops of thinking,” indicating higher cognitive demands, whereas Group 1 maintained that it involved only remembering and understanding, consistent with the second level of the taxonomy. In contrast, agreement on Item 27 was reached quickly. Both groups identified it as requiring inferencing, which aligned with its classification as a high-difficulty item.

4.2 Second Session: the SOLO Taxonomy

The second session r low initial agreement in the individual ratings (see Table 2). In Round 1 there was minimal consensus and weak chance-adjusted agreement across levels (< 50% agreement, κ > 0.40). Individual accuracy compared to the research team ranged from 43% to 87% (M = 62%, SD = 20%), again showing variability in teachers’ interpretations of the cognitive demands as outlined in the SOLO taxonomy. Round 2 showed modest improvement in consensus for most levels (< 60% agreement, κ < 0.50), although divergence was observed for extended abstract items; one group did not classify items at this level. Agreement coefficients were strong for unistructural and multistructural levels (κ > 0.80), but weak for the relational level (κ < 0.60). By Round 3, the full panel achieved unanimous consensus on all SOLO taxonomy classifications: 7 items were classified as unistructural, 7 as multistructural, twelve as relational, and 4 as extended abstract.

View Full Size

Inter-rater agreement on SOLO taxonomy cognitive demand classifications by bookmarking round

Citation: Asia-Pacific Science Education 11, 2 (2025) ; 10.1163/23641177-bja10101

• Download Figure
• Download figure as PowerPoint slide

The increased agreement between individual and group rounds likely resulted from the collaborative nature of the discussions, which helped panelists align their classifications (Geisinger & McCormick, 2010), despite their limited prior experience with the SOLO taxonomy. For example, during the full panel discussion in Round 3, Group 2 initially did not classify any items at the extended abstract level. However, after hearing Group 1’s justification for classifying an item involving growth curve analysis as higher-order thinking, Group 2 maintained their view that it required analytical skills based on textbook descriptions, but not cross-disciplinary abstraction. The final classification reflected Group 2’s interpretation. In contrast, for Item 27, Group 1 explained that the task required students to engage in problem solving by understanding the effects of temperature on matter. This led Group 2 to reconsider and ultimately agree that the task involved predictive reasoning. As a result, Items 27 to 30 were classified at the extended abstract level.

4.3 Comparing Classifications of Teacher Panels with the Research Team

The final classifications from the full panels were compared to those of the research team using Cohen’s kappa (Table 3). Agreement levels were strong when using the RBT (90% agreement, κ > 0.80), but only moderate with the SOLO taxonomy (77% agreement, κ < 0.80). The most notable discrepancy appeared with multistructural items (Items 8 to 12), where agreement was minimal (< 30% agreement), and the chance-adjusted agreement was not statistically meaningful, suggesting ratings may have occurred by chance. While the teacher panel consistently classified these items as multistructural, the research team identified them as requiring only unistructural responses, involving simple factual recall of a single aspect. During discussions, panelists often referred to the cognitive processes involved but appeared to based judgements on the verb used in the question, consistent with the RBT procedure. For example, in evaluating Item 14, a Group 1 panelist cited a local textbook to justify assigning the item to the “understanding” level, explaining that this level involves interpreting or extending basic information with prompts such as huraikan (elaborate), bandingkan (compare), nyatakan (state), and bezakan (differentiate).

View Full Size

Comparison of agreement between teacher panel and research team ratings by taxonomy and cognitive level

Citation: Asia-Pacific Science Education 11, 2 (2025) ; 10.1163/23641177-bja10101

• Download Figure
• Download figure as PowerPoint slide

Notably, one panelist asked whether the structural complexity of the answer choices should also be considered in classification. This indicated awareness of an additional dimension beyond verb choice. This moment reflects a broader interpretive challenge: the SOLO taxonomy requires judgements about the depth of understanding demonstrated in a response, a perspective that may be unfamiliar for teachers more accustomed to the verb-driven classification style of RBT. The structural focus of the SOLO taxonomy introduces interpretive challenges, particularly for teachers with limited exposure to the framework. Making abstract judgements about depth of knowledge, rather than simply matching cognitive levels to the verbs, can be difficult. Nonetheless, this outcome might have been anticipated given, the relative unfamiliarity of the SOLO taxonomy in Malaysia, where it has not been widely adopted or understood, especially in the context of item classification.

4.4 Comparing the Classification of LOTS vs HOTS between Teacher Panels and the Research Team

Further analysis examined the relationship between the two taxonomies in classifying items as lower-order thinking skills (LOTS) or higher-order thinking skills (HOTS). Four teachers participated in both taxonomy sessions. To simplify analysis, items were grouped into two categories: lower order vs higher order. In the RBT, low-difficulty items align with LOTS, while moderate- and high-difficulty items correspond to HOTS. Similarly, in the SOLO taxonomy, unistructural and multistructural items reflect surface-level learning (LOTS), whereas relational and extended abstract items indicate deep-level learning (HOTS).

Table 4 shows the number of items classified at LOTS and HOTS by both science teacher panels and the research team. A chi-square test of independence showed a significant association between the two taxonomies in the teacher panel classifications (χ²₍₁₎ = 22.57, p < .001). The teacher panels also demonstrated strong alignment with the research team’s classification (χ²₍₁₎ = 26.12, p < .001). These results suggest that, despite different theoretical structures, both taxonomies measure cognitive complexity in a broadly similar way, with LOTS corresponding to surface-level learning and HOTS to deep-level learning.

View Full Size

Comparison of item classifications by the full teacher panel and the research team using RBT and SOLO taxonomies

Citation: Asia-Pacific Science Education 11, 2 (2025) ; 10.1163/23641177-bja10101

• Download Figure
• Download figure as PowerPoint slide

For instance, while classifying Item 22, one panelist remarked:

Yes, this item is moderately difficult, but we felt it was difficult because we thought it might be challenging for our students.

Despite repeated prompts to focus on cognitive processes, teachers were often influenced by contextual factors, such as school setting and student access to resources. One Group 2 panelist noted:

I felt that we cannot generalize our school experiences to all students. Like me, teaching in an island, it’s easy to find a shell for the experiment, but in a rural or mountainous areas, this is not feasible.

These context-driven perspectives, shaped by teachers’ backgrounds and school environments (Engelhard Jr., 2011), reflected subjective interpretations that occasionally led them to misclassification. Although these views were grounded in real classroom experience, they sometimes resulted in compromised the validity of assessment classification. While item classification should ideally be systematic and unbiased (Xu & Brown, 2016), it is often affected by human factors, including professional experience, context, and interpretation (Cicchetti & Feinstein, 1990). The inconsistencies observed in this study underscore the need for structured training and clearly defined guidelines to reduce subjectivity and improve reliability of item classification.

5.1 Teachers’ Ratings of Item Cognitive Demands

The lack of consensus among panelists highlights the influence of professional judgement and personal interpretation. Rather than relying solely on taxonomy criteria, teachers frequently drew on intuitive understandings of their students’ abilities. This reliance on informal reasoning led to divergent decisions and idiosyncratic interpretations, as illustrated in the results section.

The disagreements, often rooted in differing teaching backgrounds and interpretations of cognitive demand (Cicchetti & Feinstein, 1990; Engelhard Jr., 2011), were gradually mediated through structured discussion and collaborative reflection. These interactions fostered shared understanding and clarified ambiguities in the taxonomies, highlighting the value of dialogic engagement in professional learning (Richardson, 1998).

Teachers showed more consistent classifications with RBT than with the SOLO taxonomy. This pattern likely reflects long-standing exposure to RBT through policy documents, training materials, and daily teaching practice. Familiarity created a shared heuristic and a common language for judging item intent, narrowing variation across raters. By contrast, classifications using the SOLO taxonomy were more variable. The SOLO taxonomy requires raters to evaluate the structural complexity of student responses rather than labeling processes with verbs. This shift is conceptually demanding and particularly challenging to apply to MCQ s, which lack extended responses.

Greater consistency, however, should not be interpreted as greater validity. It may instead reflect alignment with a familiar framework rather than a more accurate construct representation. Effective classification depends not only on theoretical familiarity, but also on teachers’ content knowledge and awareness of national assessment standards and practices. This broader pedagogical assessment knowledge, which combines theory, subject matter expertise, and awareness of testing conventions, is essential for valid classification, but cannot be assumed, even among experienced educators or university-level instructors. In Malaysia, where assessment responsibilities are decentralized and validated items banks are limited, these challenges are acute (Koh, 2011). As Brookhart (2010) argued, the diagnostic value of assessment depends on the quality of its tools, and poorly constructed items can risk distorting student outcomes and compromise instructional decision-making. These findings reinforce the need for structured, sustained training that integrates theory, subject matter knowledge, and contextual expertise in assessment design and evaluation.

5.2 Comparing RBT and the SOLO Taxonomy

RBT assumes that both question and answer reflect the same cognitive level. In contrast, the SOLO taxonomy allows for variations in student responses within the same task. For example, a relational-level question may elicit either multistructural or relational levels, depending on the learner. This flexibility captures student thinking more precisely and offers richer, individualized feedback (Biggs & Collis, 1982). While RBT is widely used for designing assessment items, the SOLO taxonomy provides a framework for evaluating the depth of student responses after assessments are administered.

In this study, analysis focused on how teachers classified science MCQ items using both taxonomies. Given the absence of student response data, judgments depended on item structure and answer choices. Teachers appeared influenced by how options were framed, as these often signaled recall or reasoning demands. In MCQ design, such cues can subtly shape teachers’ perceptions of cognitive complexity.

The common reliance on action verbs in RBT also contributed to misinterpretations. Although convenient, verbs can be ambiguous (Mohamed et al., 2019) and imprecise (Zulkifli & Abidin, 2022). Accurate classification requires analyzing the task demands and reasoning involved. Over emphasis on surface features risks misalignment between test items and intended learning outcomes.

Unsurprisingly, teachers working with the SOLO taxonomy for the first time, struggled at the individual item level, especially at the multistructural tasks. At the aggregate LOTS vs HOTS level, however, performance improved. This suggest that while the SOLO taxonomy offers greater conceptual depth, significant professional development would be required for accurate application in Malaysia.

At this broader level, science teachers’ judgements were consistent with the research team’s ratings. This finding suggests that teachers can reliably classify items into LOTS and HOTS, even if they struggled with finer distinctions. Given the demands of classroom practice, such aggregate categories may be sufficient for item writing and classification. Expecting teachers to consistently apply more fine-grained distinctions may be unrealistic without specialized training.