Cognitive Level of Program for International Student Assessment (PISA) Questions Based on the Revised Bloom’s Taxonomy

This study aimed to determine the cognitive level of the content in the Program for International Student Assessment (PISA) questions. This research method described the cognitive level of scientific literacy PISA questions. The cognitive level used in this study was based on the revised Bloom’s taxonomy by Anderson and Krathwohl, which includes C1 (remembering), C2 (understanding), C3 (applying), C4 (analyzing), C5 (evaluating), and C6 (creating). The PISA questions were arranged according to a predetermined theoretical framework: competence, context, knowledge, and attitude. The subjects of this research were scientific PISA questions released in 2006 and 2015, with 45 questions about the content of living systems of biological materials. The analysis of the cognitive level was presented in the form of a table and then made into a diagram so that a conclusion could be made. The results of this study indicated that PISA questions were dispersed across several cognitive levels according to the revised Bloom’s taxonomy, with the most dominant percentage of 38.24% being the cognitive analysis level (C4) in 2006 PISA and 36.36% in 2015 PISA. This study only analyzes PISA questions in the years that only focus on scientific literacy, so it needs to be analyzed in other years as well; then, it was hoped that teachers could make PISA model assessment questions with a higher cognitive level and increase the questions of students’ conceptual knowledge.

Introduction

Education in Indonesia is faced with various external challenges, among them related to the issue of educational development at an international level. That is to say, Indonesia has consistently been ranked very low at events in the Trends in International Mathematics and Science Study (TIMSS) and as well as the Program for International Student Assessment (PISA) (Ministry of Education and Culture, 2016a, 2016b). Indonesia’s low achievement in participating in the TIMSS and PISA events is due to the many test materials not included in the Indonesian curriculum (Ministry of Education and Culture, 2016a, 2016b).

Regarding the issue of the development of education in Indonesia at an international level, efforts have been made to improve the 2013 curriculum, specifically on content and assessment standards (Isbandiyah & Sanusi, 2019). Improvements to the content standards according to Regulations of the Minister of Education Number 21 of 2016 are reducing inappropriate materials, deepening and expanding materials that are suitable for students, and enriching with materials that stimulate students to think critically and analytically in accordance with international standards. Meanwhile, improvements to assessment standards according to the Regulations of the Minister of Education Number 23 of 2016 are introducing, studying, adapting, and applying international standard models of assessment in the hopes of helping students improve higher-order thinking skills. The PISA assessment was designed by the Organization for Economic Co-Operation and Development (OECD) to assess the ability of 15 years students to use their knowledge and skills to deal with situations in society of the three literacy levels, namely math, reading, and science literacy (OECD, 2013).

Scientific literacy measured in the PISA assessment is necessary to welcome the 21st century. Scientific literacy is the ability of students to use scientific knowledge, to identify questions, to understand and make conclusions in accordance with the facts that exist in nature and the changes that occur in nature due to human activity (OECD, 2007). Currently and in the future, individuals with high-order thinking skills, such as the ability to reason, think creatively, make a decision, and solve a problem, are highly needed (Setiawanet al., 2014).

The results of the PISA study showed the positions of Indonesia’s scientific literacy achievements; in the year 2000, Indonesia was ranked 38^th out of 41 participating countries. In 2003, it was ranked 38^th out of 40 countries. In 2006, it ranked 50^th out of 57 participants. In 2009, it ranked 64^th out of 65. In 2012, it ranked 64^th out of 65 countries; in 2015, it was ranked 62 out of 72 countries (Susantiet al., 2016). Then, in 2018, it ranked 74^th out of 79 participating countries (OECD, 2019).

Based on these results, it can be concluded that the scientific literacy ability of students in Indonesia is still relatively low at the international level. The PISA questions on scientific literacy were developed based on a framework consisting of four aspects such as context, knowledge, competence, and attitude. PISA questions have some characteristics that consist of 1) related to context, 2) demand reasoning ability, 3) opinion building, and 4) creative thinking skills to solve a case (Wardhani & Rumiati, 2011).

PISA questions in each literacy level are based on literacy ability. The 2006 PISA’s skill levels consist of levels one to six. The 2015 PISA, which focuses on scientific literacy, stated that the PISA ability level consists of six levels: levels one and two are in the lower category, and levels three and four are in the medium category. Levels five and six are in the higher category (OECD, 2016). The skill level of PISA 2015 refers to the Depth of Knowledge according to Webb, consists of level one (recall and reproduction), level two (using skills and conceptual knowledge), level three (strategic thinking), and level four (extended thinking) (OECD, 2016).

Students must use their thinking skills well in order to solve PISA questions. These thinking skills can be classified into two categories: higher-order thinking skills and lower-order thinking skills. In education, thinking skills are the orientation of the cognitive domain (Gunawan & Palupi, 2016). The cognitive domain was organized by an educational psychologist named Benjamin Bloom with a conceptual framework known as Bloom’s taxonomy, which was then revised in 2000 by Anderson and his colleagues; therefore, it is known as the revised Bloom’s taxonomy (Anderson & Krathwohl, 2010).

Bloom’s taxonomy consists of three domains, which include: 1) the cognitive domain that emphasizes the intellectual aspects, namely knowledge and thinking skills; 2) the affective domain that emphasizes behavioral aspects related to emotions, such as values, interests, and attitudes; and 3) the psychomotor domain which emphasizes the manipulative functions and motor skills (Utari, 2011). The cognitive domain of the revised version of Bloom’s taxonomy consists of two dimensions: the knowledge dimension and the cognitive process dimension. The knowledge dimension consists of factual, conceptual, procedural, and metacognitive knowledge. In contrast, the dimension of cognitive processes is composed of low to high levels, namely from the C1 (remembering), C2 (understanding), C3 (applying), C4 (analyzing), C5 (evaluate), and C6 (create). Levels C1 to C3 are in the Lower Order Thinking Skills (LOTS) category, while levels C4 to C6 are in the Higher Order Thinking Skills (HOTS) category (Anderson & Krathwohl, 2010). Essential competencies, competency achievement indicators, and learning objectives consist of cognitive processes targeted for using Operational Verbs (KKO) and types of knowledge based on the revised Bloom’s taxonomy (Setiawatiet al., 2019). This revised Bloom’s Taxonomy is used as a reference in preparing the curriculum, setting educational objectives, and making assessments by teachers (Gunawan & Palupi, 2016).

Low achievement among Indonesian students in participating in PISA studies is caused by several factors, such as the assessment instruments presented by teachers in schools, which were less related to the life contexts faced by students, the unavailability to facilitate students to think at higher levels and to express opinions (Wardhani & Rumiati, 2011).

Based on the revised Bloom’s taxonomy, PISA questions are related to contexts covering almost all cognitive levels. It can be proven from the results of research conducted by Wasis (2013) that the cognitive level of the 2006 PISA questions of physics content consists of the level of remembering (C1) to evaluating (C5). Furthermore, looking at the results of Rianavita’s (2016) research that analyzed the 2006 PISA questions of chemistry content, it was found that the PISA questions were at various Bloom’s cognitive levels and the more dominant questions reached the higher cognitive level, namely the level of analyzing (C4), evaluating (C5), and create (C6). Based on the above background, this study aimed to determine the dispersion of cognitive levels based on Bloom’s taxonomy on the PISA questions of science literacy for biology content.

Materials and Method

This research was conducted using a qualitative descriptive method. Descriptive research is intended to picture or describe a phenomenon that is currently happening vividly. Qualitative research is a research method that provides an overview and explains it based on reality to find the meaning of a phenomenon. In addition, it is non-experimental, which means that this research does not require research variables (Sugiyono, 2018). Therefore, qualitative descriptive research is a method used to obtain data by describing a particular phenomenon according to its reality or its true nature.

The data source in this study was PISA questions on the content of living systems of biological materials in the years 2006 and 2015. The questions were analyzed according to the cognitive levels (C1–C6) based on the revised Bloom’s taxonomy and then presented in a table. The data obtained is processed by calculating the number and percentage of cognitive levels based on Revised Bloom’s Taxonomy using the following formula:(1)P(%)=NNi×100where:

P: Percentage value,

N_i: Number of questions categorized in the cognitive process level based on the revised Bloom’s taxonomy,

N: Total number of questions.

Results

The 2006 and 2015 PISA questions were analyzed based on Bloom’s taxonomy, which was located at several cognitive levels based on the revised Bloom’s taxonomy. Grouping the questions in categories using Operational Verbs showed the thinking process done by students to solve problematic questions. The analysis of the knowledge dimension category used was adjusted with the knowledge that was used to solve a question. The results of grouping cognitive levels and knowledge dimension questions from PISA for the years 2006 and 2015 were based on Revised Bloom’s Taxonomy, which can be seen in Tables I and II.

Table I. Results of Cognitive Level and Knowledge Dimension Analysis of the 2006 PISA Questions Based on the Revised Bloom’s Taxonomy

Topic	No	Cognitive level
Topic 1: Diversity	1.	C2, interpret the food web to determine which animal has three food sources
	2.	C4, examine food webs to determine which food webs are most affected due to the extinction of the leaf moth
Topic 2: Cloning	1.	C4, conclude sheep that are identical to the Dolly’s sheep
	2.	C4, conclude the meaning of “one very small piece” of the mammary gland
	3.	C2, classify statements regarding scientific cloning
Topic 3: Diary Semmelwei’s	1.	C4, find the right reasons to prove that puerperal fever is not caused by an earthquake based on data
	2.	C4, find Semmelwei’s idea that appeared to reduce the percentage of deaths from puerperal fever based on the case
	3.	C4, find the reason for washing bed sheets at high temperatures can reduce death rates due to puerperal fever
	4.	C4, discover the cause of bacteria that causes antibiotics to become less effective over time
Topic 4: Cow cloning	1.	C4, find the main idea that has been proven through experiments conducted by research teams regarding clones on calves
	2.	C2, classify statements that includes to the same category of genetic material (identical)
Topic 5: Ozon	1.	C1, recalling diseases caused by exposure to UV rays from the sun
Topic 6 : Corn	1.	C3, carry out the process of aerobic respiration in the animal body
	2.	C3, carry out photosynthesis mechanism in plants
Topic 7: Suitable for drinking	1.	C1, identify diseases caused by drinking polluted water
Topic 8: Tooth damage	1.	C3, implement the bacteria’s mechanism in tooth decay
	2.	C2, interpreting a graph of tooth decay and sugar consumption in a country
Topic 9: Mousepox	1.	C4, find the cause of the virus infecting other species
	2.	C4, examine the effect of mouse extinction on food webs in an ecosystem
Topic 10: Stickleback fish habits	1.	C4, find questions in accordance to experimental data
	2.	C5, check over whether the conclusions made are in accordance with the results of the experiments carried out
	3.	C4, conclude the correct reaction of the fish corresponding to the color of its belly
Topic 11: Tobacco cigarette	1.	C1, recall the function of the lungs for the body
	2.	C1, identify diseases caused by smoking tobacco
	3.	C5, assess the best research that can be used to test the effectiveness of tobacco cigarettes
Topic 12: Evolution	1.	C2, interpret pictures that proves horses evolved over time
	2.	C5, assess the appropriate research plan that proves horses have evolved
	3.	C1, recall the definition of the theory of evolution
Topic 13: Bread dough	1.	C3, using the fermentation mechanism in making bread
	2.	C5, assessing experiments that can be used as comparison to prove that the yeast caused the dough to reduce
Topic 14: Major operation	1.	C2, explain the need for adding sugar through an IV tube in patients undergoing major surgery
	2.	C4, examine the systems in the human body that are disrupted caused by anesthetics
	3.	C5, check over the conclusion that corresponds to the condition of the organ transplant graph
	4.	C2, explain the reasons why the operating room must be sterile

Table II. Results of the Cognitive Level and Knowledge Dimension Analysis of the 2015 PISA Questions Based on the Revised Bloom’s Taxonomy

Topic	No	Cognitive level
Topic 15: The collapse of the bees	1.	C4, conclude the impact of bee extinction on bird population
	2.	C6, make a hypothesis regarding the causes of hives that are not exposed to insecticides but experience death in bees
	3.	C3, implement the virus mechanism in infecting bees
Topic 16: Adjustable glasses	1.	C2, explain the benefits of adjusting the human eye lens
Topic 17: Running sweat	1.	C4, examine the role of drinking water while running in a certain air temperature and humidity
	2.	C2, explain the reason for the increase in sweat volume when the temperature increases also
Topic 18: Bird migration	1.	C4, find the right reasons for birds that migrate in large groups undergo behavioral evolution
	2.	C4, find the cause of errors in data collection on the number of birds that migrate
	3.	C5, check over statements in accordance to plover birds migration map
Topic 19: Sustainable fish farming	1.	C6, designing the location of organisms in a tank according to their function and in order that the sole fish get nutrients but the water returned to the sea does not contain nutrients
	2.	C2, explain the causal relationship of adding swamp grass organisms into the water

In this study, the PISA questions that were released in 2006 and 2015 were analyzed because, in those years, the PISA assessment focused on scientific literacy. PISA questions relate to the context because PISA tests students’ ability to use their knowledge in real life. According to the results of the analysis carried out, it can be seen that the PISA questions about Biology content were located at several cognitive levels according to the revised Bloom’s taxonomy. The distribution of the 2006 PISA cognitive level according to their topic can be seen in Table III.

Table III. Distribution of the Cognitive Levels in Each Topic of the 2006 PISA Questions

Topic No.	Number of questions	The cognitive domain of the Revised Bloom’s taxonomy (%)
		C1	C2	C3	C4	C5	C6
1	2	–	2.94	–	2.94	–	–
2	3	–	2.94	–	5.88	–	–
3	4	–	–	–	11.76	–	–
4	2	–	2.94	–	2.94	–	–
5	1	2.94	–	–	–	–	–
6	2	–	–	5.88	–	–	–
7	1	2.94	–	–	–	–	–
8	2	–	2.94	2.94	–	–	–
9	2	–	–	–	5.88	–	–
10	3	–	–	–	5.88	2.94	–
11	3	5.88	–	–	–	2.94	–
12	3	2.94	2.94	–	–	2.94	–
13	2	–	–	2.94	–	2.94	–
14	4	–	5.88	–	2.94	2.94	–
Total	34	14.71	20.59	11.76	38.24	17.65	0

In PISA 2006 there were cognitive levels of C1 (remembering), C2 (understanding), C3 (applying), C4 (analyzing), and C5 (evaluating). The distribution of the percentage of the overall cognitive level can be seen in Fig. 1.

Fig. 1. Cognitive level distribution of the 2006 PISA.

Furthermore, the distribution of the 2015 PISA cognitive level according to its topic can be seen in Table IV.

Table IV. Distribution of the Cognitive Levels in Each Topic of the 2015 PISA Question

Topic No.	Number of questions	The cognitive domain of Revised Bloom taxonomy (%)
		C1	C2	C3	C4	C5	C6
15	15	–	–	9.09	9.09	–	9.09
16	16	–	9.09	–	–	–	–
17	17	–	–	9.09	9.09	–	–
18	18	–	–	–	–	9.09	–
19	19	–	9.09	–	–	–	9.09
Total	11	0	18.18	18.18	18.18	9.09	18.18

In the PISA 2015 there were cognitive levels of C2 (understanding), C3 (applying), C4 (analyzing), C5 (evaluating), and C6 (creating). The distribution of the percentages of the overall cognitive level is also shown in Fig. 2.

Fig. 2. Cognitive level distribution of the 2015 PISA.

Discussion

In this study, the 2006 and 2015 PISA questions were analyzed because, in those years, the PISA assessment was focused on scientific literacy. The PISA questions analyzed were more dominantly taken from the 2006 PISA than the 2015 PISA. This is because the 2006 PISA questions were still paper-based, making them easier to access, while the 2015 PISA questions were computer-based and difficult to access. The questions tested at the 2015 PISA were not taken from the PISA questions in the previous years, but the questions were developed in order to adapt to the use of computers.

Each topic in the 2006 and 2015 PISA questions consists of one or more questions and one or more content. However, not all content was in that one topic. PISA technology system content was only found in the 2006 PISA, while in 2015, there was no technology system content. Living system contents were the most dominant content tested for PISA in 2006 and 2015; however, in 2015, if more questions were collected, according to researchers, there is a possibility that the highest percentage of content would not be about living system content.

The PISA questions were related to context because they test students’ ability to use their knowledge in real life. According to the revised Bloom’s taxonomy, questions related to context were one of the characteristics of the HOTS questions. Based on the results of the analysis, it can be seen that the PISA questions for Biology content were dispersed at several cognitive levels and used several types of knowledge based on the revised Bloom’s taxonomy.

Based on Tables I and II, from the 14 topics of the 2006 PISA, there were 4 topics and 5 questions that were part of the remembering cognitive level (C1), which was 14.71%, while in PISA 2015, there were no questions that fell into this category. The topics were topic 5 (ozone), topic 7 (suitable for drinking), topic 11 (tobacco smoking), and topic 12 (major surgery). Remembering is the simplest cognitive level that involves retrieving knowledge from long-term memory (Anderson & Krathwohl, 2010).

One example of a PISA question with Operational Verbs that function to recall is question number 1, topic 11 (tobacco cigarettes). There was talk about the dangers of smoking for humans, and the question was, “What is the function of the lungs?”. Students only need to recall what the teacher taught or what they have read in their textbooks to answer the lung function. According to Anderson and Krathwohl (2010), recalling retrieves the knowledge needed from long-term memory when the question requires it. Moreover, another Operational Verb question appeared to identify, and an example was on topic number 7’s question (suitable for drinking). It was a “yes/no” question about a disease caused by drinking polluted water. The question was answered by involving the cognitive process of identifying and retrieving information from long-term memory about several diseases and their causes. Students need to recognize among the many diseases caused by drinking polluted water. According to Widodo (2005), identifying or recognizing is retrieving information stored in long-term memory that is the same as the information required or that was just received. The C1 cognitive level was the most basic or simple category; hence, it is important to use it to solve more complex questions. Students do not need to describe the information in the case, but only by remembering the materials that had been studied previously, then students could answer the question. The other Operational Verbs that fell into this category were mentioning, defining, and showing (Anderson & Krathwohl, 2010; Utari, 2011).

The next cognitive level is the cognitive understanding (C2). Based on Tables I and II from the 14 topics of the 2006 PISA questions, there were 6 topics and 7 questions (20.59%), while in the 2015 PISA, out of 5 topics, there were 3 topics and 3 questions (27.27%) in this category. The topics were topic 1 (diversity), topic 2 (cloning), topic 4 (calf clone), topic 8 (tooth decay), topic 12 (evolution), topic 14 (major surgery), topic 16 (adjusted glasses), topic 17 (running sweat), and topic 19 (sustainable fish farming). Understanding (C2) is a process of building meaning from educational passages, whether oral, written, or from a graph, delivered through lessons by teachers, books, or computer screens (Anderson & Krathwohl, 2010). Operational Verbs that appeared were interpreting, classifying, and explaining.

One example of the PISA questions included in this category was question number 1 on topic 1 (diversity) with the Operational Verb interpreting. There was a case about food webs in an ecosystem, and the question was, “What animal has three food sources?”. Questions were answered only by interpreting the food web by looking at the arrows and then understanding the meaning of the arrows as animals being eaten (food sources). Thus, it can be determined which animal has three sources of food, which is the animal with the three arrows positioned near it. This question can be answered without reading a long passage before the image display, but it can be understood only by changing the meaning of the arrow into another form, namely the food source. Then, the question will be answered. This question belongs to the C2 cognitive level because it cannot be answered by only remembering (C1); students need to construct the meaning conveyed in the displayed food web.

According to Gunawan and Palupi (2016), interpreting is an activity in which students change information from one form to another, for example, changing words into other words, pictures into words or vice versa, and numbers into pictures or vice versa.

There was also an Operational Verb, explained in question number 1, topic 14 (major surgery). The question was, “Why is it necessary to add sugar to the IV tube for patients who have had major surgery?”. Questions can be answered when students understand that patients who have had major surgery need food as an energy source for their bodies. Students use the causes and effects model, in which patients who have just finished major surgery need a nutritional energy source for their bodies. Therefore, it was necessary to add sugar because sugar is a source of nutrition for the body. According to Widodo (2005), explaining is the process of building and using a cause-and-effect model in a system.

Then, an Operational Verb classification was shown in several questions, for example, question number 2, topic 4 (calf clone). Several statements had a ‘yes/no’ question. The question stated that the result of the clone was “five calves that have the same genetic material” or can be said to be identical. Three statements were of something identical, which was that: the same genes, the same gender, the same hair. Identical is a category or concept, and there are several characteristics that must be met in order to classify the “statement” as an identical category. According to Widodo (2005), classifying occurs when students identify that something (object or phenomenon) falls into a certain category. Several examples of Operational Verbs fall into this category, namely giving examples, concluding, and summarizing (Anderson & Krathwohl, 2010; Utari, 2011). The questions in this category need to be answered by understanding the meaning of readings, pictures, and graphs and then explaining them using their own words. Questions at this level were not classified as C1 because they are new information received, and students need to interpret the new information. A person is said to understand if he can use his words to explain knowledge in detail.

The next cognitive level is the applying cognitive level (C3). Based on Tables I and II from the 14 topics of the 2006 PISA, there were three topics and 4 questions (11.76%), while in 2015, out of 5 topics, there was 1 topic and 1 question (9.09%) included in the cognitive level of applying (C3). The topics were topic 6 (corn), topic 8 (tooth decay), topic 13 (bread dough), and topic 16 (collapse of bee colony). According to Anderson and Krathwohl (2010), applying involves using certain procedures or stages to practice questions or solve problems. Many Operational Verbs were in this category, and they were adjusted to the knowledge used. The questions on these topics use the Operational Verb to operate or to implement.

One example of a problem with implementing a procedure was the question on topic 6 (corn), number 3. A quote from a case said, “…plants absorb it and turn it into oxygen...” and the question asked, “Fill in the blanks with the name of the substance produced other than oxygen.” Students understand the quote and then implement the process of photosynthesis in order for the questions to be answered. Photosynthesis is a process or step for plants to make food (glucose). Photosynthesis is the same procedure, and if there were a question asking for photosynthesis results, the answer would remain the same. When familiar questions about implementing the same procedures are asked, the answer is always the same. So, questions like these that use Operational Verbs were associated with procedural knowledge used in techniques and algorithms. However, when the question was to solve a problem (not familiar), students had to modify the procedure by implementing several concepts so that the problem could be solved. The operational verb to implement has the same meaning as execute. According to Widodo (2005), executing is a routine procedure in the form of stages in a certain order that has been studied previously.

The next cognitive level is the cognitive level of analyzing (C4). Based on Tables I and II, the percentage of cognitive level (C4) on the 2006 and 2015 PISA questions was 38.24% and 36.36%. In the 2006 PISA, there were 9 topics and 13 questions, while in the 2015 PISA, there were 3 topics and 4 questions included in this category. The topics included were topic 1 (diversity), topic 2 (cloning), topic 3 (Semmelwei’s diary), topic 4 (calf clone), topic 9 (Mousepox), topic 10 (habits of stickleback fish), topic 14 (major surgery), topic 16 (the collapse of bee colony), topic 17 (running sweat), and topic 18 (bird migration). Analyzing is the process of separating materials into small parts and then determining the relationship between these parts and the overall structure (Anderson & Krathwohl, 2010).

The operational verbs in this PISA questions category were finding, concluding, and examining. One example of an operational verb used was examining, stated in question number 2, topic 9 (Mousepox). The question asked, “What might happen if the mouse goes extinct?”. This question was answered by examining the information contained in the case. Students will describe the information and then relate the information together to determine the effect of the extinction of mic on living things in the ecosystem. This is in accordance with Isbandiyah and Sanusi (2019), who stated that analyzing is students’ ability to specify aspects/elements, describe, organize, compare, and find implied meanings. The same goes for the operational verbs to find and to conclude that were stated in different questions when they are trying to describe back the information to find important elements, then connecting them with other elements so that they can conclude the overall information obtained that belonged to the analysis category. According to Anderson and Krathwohl (2010), analyzing includes learning to find important pieces of information (distinguishing), determining how to organize these pieces of information (organizing), and determining the purpose behind the information (attributing). Many operational verbs were used in each cognitive process; sometimes, they were used to determine cognitive levels. However, there were times when teachers made it confusing because they used the same operational verb at different cognitive levels. The use of operational verbs was adjusted to the nature of the cognitive process itself. According to Isbandiyah and Sanusi (2019), operational verbs are strongly influenced by how thought processes go when needed to answer the questions given. The C4 level questions were preceded by a long passage case or stimuli containing information that can be used to solve the problem. When analyzing, students go through several cognitive processes, and this category was considered the beginning of students for higher-order thinking. This is in line with the results of Rosdiana (2018), where the HOTS questions at the cognitive level of analyzing (C4) lead students to use more than one cognitive aspect where students must know what is informed first then understand the question and then connect the information and then conclude to answer the question.

The next cognitive level is the cognitive level of evaluating (C5). Based on Tables I and II, from 15 topics from the 2006 PISA questions, there were 5 topics and 5 questions (14.71%), while in PISA 2015, out of 5 topics, 1 topic and 1 question (9.09%) were included in this category. The topics included were topic 10 (habit of the stickleback fish), topic 11 (tobacco smoking), topic 12 (evolution), topic 13 (bread dough), topic 14 (major surgery), and topic 18 (bird migration). Evaluating (C5) is making decisions based on existing criteria and standards (Anderson & Krathwohl, 2010). The operational verbs that appeared on the PISA questions were checking and assessing. One example of a question that includes the operational verb checking was question number 2, topic 10 (the habit of stickleback fish). There was a wax model that showed the color of the female Stickleback fish and a graph that showed the reaction of a male fish when it met the red, silver flat belly versus the round silver belly of the fish models. The question asked the “appropriate conclusion according to the information on the wax model and the graph.” This question was answered by students having to check several conclusions to determine whether they correspond with the conditions of the graph/data or not. According to Anderson and Krathwohl (2010), the checking process occurs when students test whether a conclusion is in accordance with the existing data. This question can only be answered by understanding the graph and then assessing whether a conclusion is in accordance with the condition of the graph. When looking at the level of cognitive processes, students only need to go through the cognitive process (C2) without the need to apply (C3) or analyze (C4), which means that revised Bloom’s taxonomy was arranged hierarchically from a lower cognitive process to a higher cognitive processes, but that does not mean that when students are at a higher cognitive level, they must pass all previous cognitive levels. This is in accordance with Widodo’s (2005) statement that the newly revised Bloom’s taxonomy is more flexible, meaning that to carry out higher cognitive processes, it is not required to master lower cognitive processes. Most cognitive processes require decision-making. However, not all decisions made were evaluative; for example, to decide whether an example fits its category (understanding) or not; therefore, a decision to use one certain procedure to solve a problem is not enough (apply) (Anderson & Krathwohl, 2010). To make a decision or judgment at this evaluation, the cognitive level is based on existing criteria and standards (Widodo, 2005).

One example of a question that uses the operational verb term to assess was question number 2, topic 13 (bread dough). In the question, several experiments were designed, and then students assessed the most appropriate experiment to compare and prove that the rise in bread dough was caused by yeast. Students make decisions about whether a product (of the experiment that has been designed) was effective or not in solving problems with positive and negative criteria; for example, “it is better to compare one experiment with two because both experiments were closed, but the other used yeast and the other does not. Yeast does not require oxygen to work, so it is better to choose experiments one and two so that the problem can be solved”. According to Anderson & Krathwohl (2010), assessing is the same as criticizing, which is more or less the process of assessing a product based on external criteria.

The last cognitive level is the cognitive level of creation (C6). Based on Tables I and II, there was no cognitive level of creation at the 2006 PISA 2006, while in PISA 2015, there were 2 topics and 2 questions from the cognitive level of creation (C6), which was 18.18% in topic 16 (the collapse of bee colonies), and topic 19 (sustainable fish farming). Creating (C6) is combining several elements into something new and coherent or creating an original product (Anderson & Krathwohl, 2010). The operational verbs used on these questions were planning/designing and making hypotheses.

One example of a question that uses operational verbs to design was question number 1 on topic number 19 (sustainable fish farming). There was a case about how to cultivate Sole fish sustainably, and the question asked was to “put every organism in a suitable tank to ensure that the Sole fish gets food and the salt water that is returned to the sea has zero nutrients.” Students must solve the problem of ensuring that only fish get food and that seawater does not contain nutrients. Hence, students must design the tank by placing the organisms according to their function so that the problem can be solved. According to Anderson and Krathwohl (2010), designing is planning a method or strategy to solve a problem in accordance with the criteria of the problem.

Then, one example of the operational verb of the term making a hypothesis was stated in question number 2 about the collapse of the bee colony. In the question, there was a problem where it said that “on the 20^th day, bee hives that are not exposed to insecticides experience bee deaths”. The question was, “What caused the bee colony collapse that was not exposed to insecticide (to control the hive) on day 20?”. This question was answered by making a hypothesis about the cause of the collapse of the bees that were not exposed to insecticides, which creates the possibility that the practitioner did not properly maintain the hive control; hence, they were contaminated by exposure to imidacloprid. The hypotheses can be used to solve the problem of hives being more controlled and not experiencing bee deaths. Making a hypothesis involves describing a problem so that various possible hypotheses can be formulated to solve the problem (Widodo, 2005). According to Bloom’s revised taxonomy, the cognitive level of creation (C6) is the highest and is the last level of higher-order thinking.

Based on the results of the analysis, PISA questions were dispersed across various cognitive levels, one of which was the cognitive levels of remembering (C1), understanding (C2), applying (C3), analyzing (C4), evaluating (C5), and creating (C6). The same thing was also found in the results of research conducted by Wasis (2013), who analyzed the cognitive level of the 2009 PISA questions on physics, stating that the PISA questions of physics content consist of cognitive levels C1 to C6 based on the revised Bloom’s taxonomy.

However, according to the revised Bloom’s taxonomy, PISA questions were indeed dominated by HOTS questions. Based on Figs. 1 and 2, the percentage of cognitive levels C4 to C6 (HOTS) in the 2006 PISA was 52.94%, while in the 2015 PISA was 63.64%. The 2006 and 2015 PISA LOTS questions were 47.06% and 36.36%. This is in line with the results of research conducted by Dongoran (2015), which analyzed the cognitive level of the 2009 PISA questions content of living systems of biological material based on the revised Bloom’s taxonomy that resulted in the most dominant cognitive level questions were won by the HOTS question of 59.99%. On the other hand, the LOTS questions were only 40.01%. Based on the analysis results, there was no C6 cognitive level in the 2006 PISA question and no C1 cognitive level in the 2015 PISA questions about biological material. However, that does not mean that scientific literacy, which includes physics and chemistry, does not have a C6 cognitive level (creating). This can be seen from Rianavita’s (2016) analysis on 24 questions of the 2006 PISA for chemistry, with 6 questions, including the C6 cognitive level (creating). This shows that the PISA questions of scientific literacy cover all cognitive levels based on Bloom’s taxonomy, which were dispersed over several learning materials (physics, chemistry, biology).

The PISA questions were dominant at a higher cognitive level because the PISA assessment was an international standard assessment that demanded analytical thinking skills from students. The PISA assessment was intended to see the ability of students aged 15 years or near the end of the school year to use their knowledge for real life, meaning that PISA assesses students’ scientific literacy abilities. Individuals with high-order thinking skills, for example, those who can reason, think creatively, make decisions, and solve problems, are highly needed currently and in the future. According to Rustaman (2006), PISA assesses students’ scientific literacy abilities, which includes applying knowledge and skills and analyzing, reasoning, and effectively communicating when faced with problems that must be solved and interpreted in various situations. Assessment questions that require the ability to reason are at a higher cognitive level, which consists of the cognitive level of analyzing (C4) until creating (C6).

PISA questions that require higher-order thinking skills were supported by the characteristics of the questions related to context. Cases or passages preceded the HOTS questions before solving the questions. Students usually use the information in the case or passage to solve the question. According to Isbandiyah and Sanusi (2019), one of the characteristics of the HOTS question is that it is preceded by an interesting and contextual case so that it stimulates students to look for relationships between information, capture ideas or main topic in the case, examine ideas and information critically, or interpret a new situation of the case presented. This aligns with PISA, which presents questions preceded by contextual cases related to personal, social, and global contexts. These cases can be in the form of short writing or text that is accompanied by tables, charts, graphs, photos, or diagrams and require students to use the information provided to answer questions (OECD, 2007).

PISA questions were more dominant at a higher cognitive level. However, it is possible that PISA questions were also at a cognitive level that stimulates students of the lower cognitive level, like remembering (C1) until applying (C3). A contextual case preceded some questions, but they do not require higher cognitive processes to solve them. According to researchers, one of the consequences of Indonesia’s low achievement in the PISA events was the low interest of students in reading to solve the questions of PISA. Long texts make students skip the questions or just copy what was in the case or passage even though the questions were not necessarily HOTS questions. This is in line with the research of Indasyafira (2019) and Putri (2019), which stated that PISA questions preceded by a long and contextual case or passage make students answer questions only by copying what was in the passage. Some of them do not even fill out the answer sheets. The ability of students to answer PISA questions of scientific literacy was related to students’ low reading interest. According to Rustaman (2006), students’ reading literacy contributes to scientific literacy because most scientific literacy questions are presented in the form of reading (text) accompanied by several questions to be answered based on the understanding of the text.

Conclusion

Based on the results of the research, it can be concluded that the PISA questions on the content of living systems of biological material cover several cognitive levels based on the revised Bloom’s taxonomy that includes in 2006 where the percentage of remembering level (C1) was 14.71%, understanding level (C2) was 20.59%, applying level (C3) was 11.76%, analyzing level (C4) was 38.24%, and evaluating level (C5) was 14.71%. In PISA 2015, the percentage of the cognitive level of understanding (C2) was 27.27%, the applying level (C3) was 9.09%, the analyzing level (C4) was 36.36%, and the creating level (C6) was 18.18%. The most dominant cognitive level was the higher cognitive levels, namely C4 to C6, in the 2006 PISA, which was 52.94%; meanwhile, in the 2015 PISA, it was 63.64%. A suggestion was also made in this study, which stated that the translation of the questions into the Indonesian language should correspond to terms that Indonesian students are more familiar with. For future researchers, it is better to analyze PISA questions in a year that does not only focus on scientific literacy. Furthermore, the results of the analysis showed that the dominant PISA questions were HOTS questions. Therefore, it is better for the teachers who want to make PISA model assessment questions to adapt to the cognitive demands of PISA in order for students to be trained in working on PISA model questions.