Assessing Conceptual Knowledge Using Three Concept Map Scoring Methods

Background: Conceptual understanding is a prerequisite for engineering competence. Concept maps may be effective tools for assessing conceptual knowledge, yet further work is needed to examine scoring methods. Purpose: Our purpose was to evaluate the efficacy of three concept map scoring methods. Traditional scoring requires judges to count concept map components. Holistic scoring requires judges to analytically evaluate concept maps using a rubric. Categorical scoring requires judges to categorize concepts according to relevant categories before quantification of overall complexity. Design/Method: In this study, concept maps were collected from 72 undergraduates before and after a capstone design course. Judges analyzed the concept maps using three methods. Using statistical analyses, we examined the comparability, interrater reliability, and convergent/divergent validity of scoring methods. Results: While traditional scoring allowed judges to score concept maps relatively quickly, holistic scoring allowed judges to better capture changes in knowledge structure, because ranking of data was required without assuming hierarchical concept arrangement. Only categorical scoring provided insight into content and structure of student knowledge. For all methods, interrater reliability was acceptable, and convergent/divergent validity was established. Conclusions: Due to its applicability to concept maps of varying structures, holistic scoring is best if more than one judge is available; traditional scoring is appropriate if there are time constraints or if multiple methods are used. Categorical scoring has potential as a supplemental or stand-alone method, depending on the needs of the instructor or researcher.