Sorting Out Card Sorting

	[Home] [Abstract] [TOC] [Chapter 1] [Chapter 2] [Chapter 3] [Chapter 4] [Chapter 5] [Appendix A] [References]

	Download PDF
	CHAPTER IV – ANALYSIS OF DATA This chapter presents a comprehensive, replicable, extensible tool (see Table 1: Twelve Categories of Card Sorting Characteristics), designed for practitioners to use in two ways:  to review data gleaned from the guidelines and case reports presented in this study in order to assist them with choosing criteria for their own card sorting exercise; and to extend the data set by coding their own card sorting exercises and observations into the table. Table 1 is intended to represent one proposed model for the identification of criteria that practitioners should consider when designing a card sorting exercise. The researcher hopes that practitioners may discover literature in the references that warrants further study. The aggregated observations of practitioners may provide discursive rationale for decisions made in card sorting design. Contradictory observations may prompt reconsiderations in methodology design or promote considerations of further empirical research. Practitioners and researchers may extend the data set to include other literature, or incorporate their own observations and property values for analysis. Table 1 is the report of the data analysis of fourteen articles – seven articles are classified as practitioner guidelines and seven are classified as case reports that describe a single study. This literature under review represents the boundaries of this analysis and as such, there is no intent to project this data as representative of all card sorting methods used in information design and testing. The reader should assume that these presumptions exist throughout the reporting of the analysis of data. A conceptual analysis process (Palmquist, et al. 2005) is conducted to code these fourteen references, and the coding results are entered into a spreadsheet. The results of the coding process are interpreted using a constant comparative method, which seeks to categorize the properties of the phenomena while concurrently generating theory (Glaser & Strauss, 1967, p. 102-103). Forming Categories A preliminary discussion on the process and value of the identification of categories is important for a number of reasons. According to Coxon (1999), “The two most basic principles about category formations are (1) that they provide maximum information with the least cognitive effort and (2) that the perceived world comes as structured information rather than as arbitrary or unpredictable attributes” (Coxon, 1999, p. 13). Categories are often identified by first identifying a “prototypical instance” of the phenomena to serve as a foundational representation of the properties or attributes of the category (Coxon, 1999, p. 13). The essential criterion for the formulation of categories is the presence of a “similarity of meaning” in the semantics of the language. This does not imply that the “same” meaning is sought, which would “reduce the semantic task to finding synonyms” (Coxon, 1999, p.14; attributed to Miller, 1969). Categorizing the properties of a phenomenon is intended to provide a basis for comparing both the “maximization and minimization of similarities and differences” discovered within the data (Glaser and Strauss, 1967, p.55). This process may reveal interrelationships within or between categories, or may generate new categories. The identification of minimized differences (similarities) within a category tends to establish a “probability of a theoretical prediction” (Glaser and Strauss, 1967, p. 55). Maximized differences within a category may help to identify ranges of values, causes of outcome, variations in approach, degrees of consensus, or other quantitative or qualitative insight that furthers the formulation of substantive theory (Glaser and Strauss, 1967, p. 32, 56). Definition of Twelve Card Sorting Categories The content analysis process begins with a thorough reading of four of the references annotated in the Review of Literature for this study. These core references include Akerelrea and Zimmerman (2002), Deaton (2002), Mauer and Warfel (2004), and Ahlstrom and Allendoerfer (2004). The first three references are classified as practitioner guidelines; Ahlstrom and Allendoerfer (2004) is classified as a case report. These references serve as “prototypical instances” (Coxon, 1999) or “theoretical samplings” (Glaser & Strauss, 1967) of the literature. As the literature is studied, potential groupings of characteristics are noted on a sheet of paper with notations made on the printed literature. A set of five or six colored highlighter markers are instrumental for locating the notations in the literature. These characteristic groupings identified ten of the eventual twelve categories. Spreadsheet columns were created and the references were coded in the random order that resulted from the continuous shuffling of the printed versions of the literature. The initial twelve categories identified at the completion of the conceptual analysis are: Sort Type. This category identifies the description of the card sort as either open sort or closed sort. Information Domain Defined. If identified in the literature, this category indicates the type of information resource referred to in the guideline or case report reference. Group or Individual Sort. This category identifies whether the literature discusses using groups or individuals for the sorting exercise. Where practitioners offered perspectives on advantages or disadvantages of the two designs, these notations were assigned a positive, neutral, or negative property. Reconciling Categories. This category was defined at the end of the content analysis in response to the researcher’s determination that the method of sorting analysis did not adequately address this essential component of card sorting exercise design. Reconciling Categories from an open card sort should be viewed as a potential property of Sorting Analysis. Sorting Analysis. This category identifies the whether the method used to aggregate the results of the open sort is quantitative or qualitative. If identified in the literature, practitioner perspectives on the benefits or drawbacks of the method are assigned properties of positive, neutral, or negative. Number of Cards Sorted. Where reported, the number of cards sorted or recommended numeric range of cards to sort is recorded. Number of Participants. Where reported, the number of participants involved in the sort, or the recommended numeric range of participants to include is recorded. Minutes for Card Sorting Exercise. When identified, the typical time or range of time needed for the card sort is noted. How to Define Target Audiences. This category briefly identifies methods used, or recommendations for identifying the primary users of the information domain. How to Select Participants. This category briefly identifies methodologies and recommendations used to select participants for the card sorting exercise. Content Selection Process. This category briefly identifies methodologies and recommendations for selecting the content to include on the cards to be sorted. Term for Content Sorted. This category does not truly represent a characteristic of the design of the card sorting exercise. It is included to demonstrate semantically the range of concepts included in the content selected for the card sort. Table 1: Twelve Categories of Card Sorting Characteristics is shown here. This represents a condensed hypertext tool that is intended to display the results of the conceptual analysis. The condensed hypertext tool is prefaced by a list of the references cited in Table 1. Following Table 1, a narrative analysis of the data is provided. References cited in Table 1, including analysis notations: Akerelrea & Zimmerman, 2002, analysis 1. Fuccella & Pizzolato, 1998, analysis 2. Kidwell & Martin, 2001, analysis 3. McGeorge & Rugg, 2003, analysis 4. Ahlstrom & Allendoerfer, 2004, analysis 5. Faiks & Hyland, 2000, analysis 6 Fuccella, 1997, analysis 7. Dearholt et al., 1986, analysis 8. Mauer and Warfel, 2004, analysis 9. Deaton, 2002, analysis 10. Martin, 1999, analysis 11. Nielsen & Sano, 1994, analysis 12. Hahsler & Simon, 2001, analysis 13. Robertson, 2002, analysis 14. Discussion of Category # 1: Sort Type Open Sorting The data analysis begins by identifying the card sort as either open or closed. Open sorting is defined as “[a methodology] in which subjects can determine their own groupings by first sorting the cards and then labeling the resulting piles” (Deaton, 2002, p.4). The predominance of literature reviewed describes open sort methods, where no pre-existing categories are provided and limited instructions are offered on how to group the cards. Most practitioners recommend providing simple instructions that allow the participants considerable flexibility. For example, the following approaches are described: “It is important to highlight to the users that they should organize the cards in a way that works for them” (Robertson, 2002). See Table 1, analysis 14. “The users were asked to sit down at a table and sort the cards into piles according to similarity” (Nielsen & Sano, 1994). See Table 1, analysis 12. “Ask each participant to arrange the cards into logical groups. Explain that the groups should contain topics that seem to the participant to be related” (Martin, 1999). See Table 1, analysis 11. “Instruct the subjects to sort the cards into at least two groups” (Deaton, 2002). See Table 1, analysis 10. “[Participants are instructed to] sort the cards into groups that make sense to you” (Mauer & Warfel, 2004). See Table 1, analysis 9. “[Users] are instructed to organize the cards in any way that is meaningful to them. Users can create any number of groups and any group can have any number of cards in it” (Fuccella, 1997). See Table 1, analysis 7. “[Users] were instructed to sort the cards by placing similar cards into piles. Users were asked to try not to make piles of a very few or a great many cards but were given no other instructions” (Faiks & Hyland, 2000). See Table 1, analysis 6. “We asked the participants to choose their own group names, allowed them to use as many groups as they wanted, and told them they should create an ‘I don’t know’ group if necessary” (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. “The participant arranges cards representing content into groups of items that he or she sees as interrelated” (Kidwell & Martin, 2001). See Table 1, analysis 3. “Users are given the stack of cards (arranged randomly) and are instructed to organize the cards in any way that is meaningful to them” (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. “Ask participants to lay the cards out in front of them on the table, arrange the cards into groups or piles that make sense to them. Stress that there are no right or wrong answers, number of piles, or number of cards required” (Akerelrea & Zimmerman, 2002). See Table 1, analysis 1. The spreadsheet representation of the coded data in this study does not specify properties of the open sort category or identify a category for instructions to participants. However, this narrative analysis suggests that by providing minimal instructions given to participants related sorting criteria, practitioners generally adhere to the conceptual definition of “open sorting” used in this study. Two exceptions to this generalization in participant instructions are noted – McGeorge and Rugg (2003) and Dearholt et al. (1986). McGeorge and Rugg (2003) suggest that “it is usually advisable to tell respondents explicitly not to lump two sorting criteria together into one sort. For example, ‘big and expensive’ should be sorted once for ‘big’ and once for ‘expensive’” (McGeorge and Rugg, 2003). In this referenced study, the information domain is a collection of scientific journals and the participants are a librarian and an experienced researcher. To minimize introducing bias to the sort, McGeorge and Rugg (2003) suggest providing sorting instructions using examples that are greatly distanced from the information domain (McGeorge & Rugg, 2003). See Table 1, analysis 4. Dearholt et al. (1986) instructed the participants to “sort the cards into piles based on function. They were told to first select the cards (commands) from the deck that they were definitely familiar with and sort them into as many piles as they wished according to function” (Dearholt et al. 1986). See Table 1, analysis 8. In this referenced study, the information domain is a collection of UNIX commands and the participants are experienced UNIX administrators. Closed Sorting This study does not reveal substantial data on the use of closed card sorting methodologies in information design. Closed sorting is defined as “[a methodology] in which the groupings are defined by the researcher and the subject is putting object cards into the defined groups” (Deaton, 2002, p.4). Closed sorting is typically used for testing proposed or existing designs, or for testing information categories and labels that emerge from an open sort exercise (Mauer & Warfel, 2004, p.2) (Boutelle & Sinha, 2004, p.350) (Deaton, 2002). Discussion of Category # 2: Information Domain Defined The information domains in the comparative studies, where identified, are either public Internet or corporate intranet sites with broadly diverse audiences and information items. The information domain, the intended audience, and the participants selected in the studies conducted by McGeorge and Rugg (2003) and Dearholt et al. (1986) are substantially dissimilar and represent exceptions to the comparative studies. In both cases where exceptions are noted, the participants are highly familiar with the information domain and the information sorted represents relatively narrow and specialized topics. Thus, for insight into the design of these exception studies, the participant instructions should be viewed within the context of the information domain, and the participant knowledge of the information domain. Discussion of Category # 3: Individual or Group Sort Practitioner perceptions vary on the value of having participants sort information individually or in groups. The perspectives offered by practitioners in the Individual or Group Sort category may be interrelated with the How to Select Participants and How to Define Target Audiences category, as noted in this summary. The Individual or Group Sort category has properties of positive, neutral, and negative. The more salient points are listed below. Positive Perspectives on Individual Sorting “Each participant completed the exercise in an individual session to assure independence of grouping strategies” (Kidwell & Martin, 2001). See Table 1, analysis 3. “Scheduling individuals can be easier than scheduling groups of people” (Mauer & Warfel, 2002). See Table 1, analysis 9. Negative Perspectives on Individual Sorting “Individuals can find it difficult to sort larger numbers of cards, providing less valuable input” (Mauer & Warfel, 2002). See Table 1, analysis 9. Positive Perspectives on Group Sorting “A benefit of group sorts is that they typically provide richer data than individual sorts. Whereas individuals need to be prompted to ‘think aloud,’ groups tend to discuss their decisions aloud openly” (Mauer & Warfel, 2004). See Table 1, analysis 9. “Sometimes, having a group of users get together and do the sorting collectively can produce valuable information not only with the results of the sorting, but in the conversations carried on while the sorting process” (Deaton, 2002). See Table 1, analysis 10. “The use of a group format also has considerable benefits. Often, the participants will bring to the session quite different opinions. Through the discussion and eventual resolution of these differences, it becomes possible to identify a workable structure” (Robertson, 2002). See Table 1, analysis 14. Negative Perceptions of Group Sorting “If the participants worked as a group, deciding the categories through a consensus process, individual approaches to the information organization may be lost” (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. “In a multiple participant situation, participants may influence one another's number of card groups or sorting criteria" (Martin, 1999). See Table 1, analysis 11. Other than one notation on the difficulties individuals may face in sorting large numbers of cards, there is not substantial data that suggest practitioners have a negative perception of Individual Sorting. The positive practitioner perspectives on the benefits of Group Sorting may be interrelated to the design of these studies, where the selection of participants for a group sort represents various audience definitions and the practitioners’ desire to observe and record participant interaction during the sorting session. Substantial data exist that support the identification of additional properties within this category, or the creation of a “super-category” of Card Sorting Session with Individual and Group Sort as properties of the category. Although not specifically coded in this analysis, there are significant variations in the practitioner approaches to the monitoring of the card sorting session. Discussion of Category # 4: Process of Reconciling Categories A significant amount of data exists that describes methodologies used to reconcile information categories that emerge from an open sort. Although the description of closed sorting suggests that it may be used for this purpose, the data do not support the reconciliation of categories as a property of closed sorting. However, every article reviewed addresses the topic of category reconciliation from open sorting. As a result, a category of Reconciling Categories is created. The method used for the reconciliation of categories that emerge from an open sort is directly related to the method of data analysis used. However, hybrid qualitative and quantitative methods of category reconciliation are mentioned. In several cases (Nielsen & Sano, 1994) (Alhstrom & Allendoerfer, 2004), a quantitative method of category reconciliation is used to verify the results of a qualitative analysis. The data indicate that a number of variations in methodology exist within this category, an indication that potential exists for the identification of additional properties of the category. If this study were intended to categorize the characteristics of card sorting data analysis, this category would likely be considered as a property of the sorting analysis method. Due to the large amount of contextual and descriptive data that explain these processes, it was decided the data are best compared in narrative form, presented below. Reports of Qualitative Category Reconciliation A set of guidelines provided by Robertson (2002) recommends a group sort, with several different groups that are representative of different target audiences. The steps as outlined by Robertson (2002) are: 1) “when a pile is finalized, ask the group to nominate a label for the pile.” 2) “write down the groupings identified by the participants.” 3) “[create] a graphical presentation that displays a ‘mock-up’ of what the structure would look like.” Robertson (2002) contends, “it is often very revealing to compare the results of card sorting sessions with your different user groups. If a common structure appears across a wide range of users, you can be confident that this is the right way to go.” (Robertson, 2002). See Table 1, analysis 14. A case report by Hahsler and Simon (2001) describes their process as 1) “the users [participants] are asked to provide each set [grouping] with a unique name and a short description.” 2) “the project team compiles a preliminary navigation structure, a challenging task which requires a considerable degree of creativity.” 3) “evaluation of categories and the assignment of web objects is carried out by conducting a user survey” (Hahsler & Simon, 2001). See Table 1, analysis 13. A case report by Nielsen and Sano (1994) verifies the qualitative analysis of the card sorting data with a quantitative cluster analysis. The qualitative analysis process begins by 1) “[users] group the piles into larger groups…and [are] asked to invent a name for each group.” 2) “our design was based on ‘data eyeballing’ and not on formal statistics. For our manual clustering, we worked bottom-up and expanded these small groups into larger clusters by adding concepts that some users had sorted with most of the concepts in the group if the grouping made sense to us. This subjective interpretation of the data is dubious if the objective ‘truth’ is desired, but in our case we were after a coherent design” (Nielsen & Sano, 1994). See Table 1, analysis 12. A set of guidelines provided by Mauer and Warfel (2004) briefly mention both quantitative and qualitative approaches to reconciling categories that emerge from an open sort. The authors suggest that with a smaller number of cards, “you may be able to see patterns by simply laying the groups out on a table, or taping them on a whiteboard” (Mauer & Warfel, 2004). See Table 1, analysis 9. A set of guidelines provided by Fuccella (1997) 1) asks users to “provide a description for each group [not] a label or category name…[this activity] ideally should be performed with two separate sets of users, one for the sorting, and one for the description.” 2) “the designer can begin the iterative process of identifying the appropriate labels and clusters of information for the site”. See Table 1, analysis 7. A case report by Akerelrea and Zimmerman (2004) correlates the results of qualitative categorization of the card sort with a cluster analysis. For the qualitative analysis 1) “we asked the participants to choose their own group names.” 2) “it is straightforward to examine the group names used and look for patterns…these patterns are used to derive categories” (Akerelrea & Zimmerman, 2004). See Table 1, analysis 5. A set of guidelines presented by Fuccella and Pizzolato (1998) mirrors the description provided by Fuccella (1997). See Table 1, analysis 2. Akerelrea and Zimmerman (2002) provide the most distinct variation of the category identification process in their description of the reconciliation activity in step 4. After completing the individual open card sort, 1) “ask the participants to write a label … for each group. The label might be a single word, a phrase, or a sentence.” 2) “sort all the participants’ labeled groups into common piles.” 3) “write descriptive titles for the major groups based on the participants’ labels.” 4) “bring participants back into the room and read each card aloud individually and ask the group under which descriptive label or labels they would look to find the idea.” (Akerelrea & Zimmerman, 2002) See Table 1, analysis 1. A number of similarities (minimized differences) exist within these descriptions of qualitative category reconciliation. This suggests these activities are a generally accepted design practice. The participants are asked to provide a name or description for the card grouping. The practitioner conducting the card sort organizes the groupings. The practitioner suggests labels for the reconciled categories. Reports of Quantitative Category Reconciliation Many practitioners acknowledge the value of either qualitative or quantitative analysis of the card sorting data. For further insight into practitioner perspectives on quantitative and qualitative analysis of the data, see Category # 5: Sorting Analysis. The following excerpts summarize data concerning quantitative category reconciliation as revealed in the literature under review. In a case report by Kidwell and Martin (2001), the practitioners use the IBM USort® module of the IBM EZSort® tool to “convert each participant’s raw data to a set of distance scores for each possible card pair. Then, in the EZCalc® module, individual distance scores were averaged across participants to obtain a mean distance score for each card pair, and the mean scores were expressed in a distance matrix. The mean distance scores were analyzed using a complete linkage algorithm, a hierarchical agglomerative method of cluster analysis” (Kidwell & Martin, 2001). See Table 1, analysis 3. A case report by Martin (1999) reports the development of the IBM EZSort® and EZCalc® tools. Martin (1999) presents and analyzes the dendrograms generated from a cluster analysis performed by these tools (Martin, 1999). See Table 1, analysis 11. A case report by Dearholt et al. (1986) elaborates on a number of quantitative analyses performed on the card sorting results, including the creation of a co-concurrence matrix, a conditional probability matrix, and performing a cluster analysis (Dearholt et al. 1986). See Table 1, analysis 8. Quantitative Verification of Qualitative Analyses A case report by Nielsen and Sano (1994) reports on the use of a cluster analysis to verify the results of the qualitative “eyeballing” of the data. The authors concede that “[with only four participants] statistical methods are not very reliable…as it turned out, the statistical cluster analysis was very similar to that we had constructed manually” (Nielsen & Sano, 1994). See Table 1, analysis 12. A case report by Ahlstrom and Allendoerfer (2004) presents a comprehensive comparison of both the quantitative and qualitative analyses of the card sorting data. The authors begin by manually creating an “association matrix” which assigns values of 0-9 (reflecting the results of the nine participant card sorts) to each card sorted. A value of ‘9’ indicates that all participants placed the card in a similar pile; a value of ‘0’ indicates that no participants placed the card in a similar pile. These results were then subjected to a cluster analysis by the Statistica® software from Statsoft® and the results displayed in a dendrographic tree. Ahlstrom and Allendoerfer (2004) report “the large branches of the tree related well to the categories derived by hand, even when the precise members of the category differed somewhat” (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. The predominance of literature reviewed describes qualitative methods for analyses of the card sorting data. The references reviewed in this study that describe quantitative methods indicate the following similarities: Cluster analysis is an accepted form of quantitative analyses for card sorting data. The results of a cluster analysis are typically displayed by a dendrographic tree. In both cases where qualitative and quantitative analyses were performed on the same card sorting data, the identified relationships between the information items sorted were significantly similar. Discussion of Category # 5: Sorting Analysis The Sorting Analysis category is intended to reveal practitioner perceptions on the benefits and drawbacks of qualitative and quantitative analysis of the card sorting data. Perceptions and comments made by practitioners on the value of qualitative and quantitative sorting analysis are represented as properties of this category, and are “rated” as positive, neutral, or negative. Rather than iterating the entire scope of the data available in the references, the most salient points are provided in this analysis. Positive Perceptions of Qualitative Analysis “We prefer a qualitative approach due to the low number of participants” (five to ten participants are mentioned) (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. “This manual method has a number of benefits. First, it is straightforward to execute and does not require sophisticated analysis tools. Second, unlike many statistical techniques, small sample sizes do not restrict it. Third, results from this method are easy to present to audiences who are not experienced at interpreting multivariate statistics” (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. “Results, if not too extensive or complex, can be gathered by ‘eyeballing’ the card groupings” (Faiks & Hyland, 2000). See Table 1, analysis 6. “[Fuccella] prefers a more qualitative approach in which specific questions regarding the organization of the information have been identified prior to the card sorting tasks” (Fuccella, 1997). See Table 1, analysis 7. “When performing analysis on smaller numbers of cards, you may be able to see patterns by simply laying the groups out on a table” (Mauer & Warfel, 2004). See Table 1, analysis 9. “…we were after a coherent design, so we felt justified in applying our judgment in those cases where the user data was too sparse for a clear conclusion to be drawn on the basis of the numbers” (Nielsen & Sano, 1994). See Table 1, analysis 12. Negative Perceptions of Qualitative Analysis “This [qualitative] method also has several drawbacks. First, there is a level of subjectivity required to derive the categories. Second, the method becomes time consuming and extremely tedious when the number of items or participants is large. Third, the method examines the relationship of items to categories rather than items to other items” (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. “Manually searching through sorted card sets for patterns is inherently vulnerable to bias, as patterns that confirm the observer’s prior notions will be recognized more readily than those based on less familiar mental constructs” (Kidwell & Martin, 2001). See Table 1, analysis 3. “Some web site designers have ‘eyeballed’ card groupings created by a few test participants (e.g., Nielsen & Sano, 1994), and somehow divined a central tendency from the competing sorting structures. This method, if ever it were manageable, becomes unwieldy very quickly with the inclusion of more than a handful of topics or users” (Martin, 1999). See Table 1, analysis 11. “With a large data set, eyeballing a result is difficult” (Deaton, 2002). The general perception among practitioners is that a qualitative analysis of data has value. Only Martin (1999) and Kidwell and Martin (2001) appear to reject qualitative analysis methods. However, a larger number of participants or a large number of cards sorted may negatively affect the practitioner’s ability to use only qualitative analysis methods (Martin, 1999) (Ahlstrom & Allendoerfer, 2004) (Mauer & Warfel, 2004) (Faiks & Hyland, 2000) (Fuccella & Pizzolato, 1998) (Deaton, 2002). Positive Perceptions of Quantitative Analysis Deaton (2002) makes an interesting observation on the dendrographic representation of a quantitative data analysis: “cluster analysis is particularly apt for analyzing card sorting because it enables you to see how closely items are related across all your subjects. This is a form of qualitative analysis, where how you ‘see’ the result is more important than the numbers” (Deaton, 2002). Although a number of references (Martin, 1999) (Kidwell & Martin, 2001) (Dearholt, et al. 1986) (Nielsen & Sano, 1994) describe quantitative analysis of data, it was difficult to extract qualitative remarks pertaining to the value of cluster analysis. This may be the due to the nature of objective statistical analysis. Kidwell and Martin (2001) argue that “A more objective method of analyzing card sorting data is cluster analysis…cluster analysis can reveal an aggregate representations of users’ internal models of the relatedness of data items” (Kidwell & Martin, 2001). Faiks and Hyland (2000) suggest, "Running the statistical analysis is very helpful, not too complicated, and recommended, but it is not a necessary component" (Faids & Hyland, 2000). Other practitioners who qualitatively analyze card sorting data briefly acknowledge value in quantitative methods without significant elaboration. No negative perspectives on quantitative analysis were identified. Discussion of Category # 6: Number of Cards to Sort Where values for this category are identified, the number of cards sorted or recommended for sorting range from 30 to 219. Practitioner perspectives on the number of cards to include in the sort are provided below. A set of guidelines presented by Akerelrea and Zimmerman (2002) recommend “limiting the ideas to 75 to 100 cards with each idea to the card.” No explanation is provided for the recommendation (Akerelrea & Zimmerman, 2002). See Table 1, analysis 1. A case report by Kidwell and Martin (2001) sorted 66 cards, with no explanation as to why this number was chosen (Kidwell & Martin, 2001). See Table 1, analysis 3. A case report by Ahlstrom and Allendoerfer (2004) reports 95 cards in the sort. No explanation is provided (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. A case report by Faiks and Hyland (2000) reports using 50 cards in the sort. The 50 cards represent the 50 topics on the current online Help system under redesign (Faiks & Hyland, 2000). See Table 1, analysis 6. A case report by Dearholt et al. (1986) sorts 219 cards. The cards represent each of the UNIX command functions identified for inclusion in the online Help system (Dearholt et al., 1986). See Table 1, analysis 8. A set of guidelines provided by Mauer and Warfel (2004) recommend 30 to 100 cards. The authors contend, “…fewer than 30 cards typically does not allow for enough grouping to emerge and more than 100 cards can be time consuming and tiring for participants. However, we have performed successful card sorts with over 200 cards where the participants understood the content well” (Mauer and Warfel, 2004). See Table 1, analysis 9. A case report by Nielsen and Sano (1994) reports 51 cards in the sort. The development team “brainstormed about possible information services to be provided over the system” to arrive at this number of items (Nielsen & Sano, 1994). See Table 1, analysis 12. A case report by Hahsler and Simon (2001) had participants sort 120 cards that “[represent] the most important 120 web objects.” The authors report, “…some of the users lost patience. One user did not finish the card sorting exercise at all, while another did not provide category names. Accordingly, we suggest that the number of cards should not exceed 100” (Hahsler & Simon, 2001). See Table 1, analysis 13. The literature reveals that sorting 30 to 100 cards is a general practice, with the majority of studies or guidelines using or recommending between 50 and 100 cards. Hahsler and Simon (2001) provide a valid point to consider when the number of cards exceeds 100. They report that participants failed to complete the exercise or exhibited other signs of frustration. Conversely, an interesting contrary perspective is revealed in the comments made by Mauer and Warfel (2004), who contend, “We have performed successful card sorts with over 200 cards where the participants understood the content well” (Mauer & Warfel, 2004). The study provided by Dearholt et al. (1986) reports no difficulties with 219 cards. In this example, the information domain is a set of UNIX commands and the participants are experienced UNIX administrators (Dearholt et al, 1986). This suggests the participant knowledge of the information domain should be considered when using larger numbers of cards in the sort. Discussion of Category # 7: Number of Participants Within the references that state or recommend a number of participants for the card sorting exercise, the values range from 2 to 30. Where only 2 participants are used, there are other significant dissimilarities in the design and purpose of the experiment conducted by McGeorge and Rugg (2003). None of the references reviewed state any criteria used to determine the number of participants to involve. It should be noted that in the case of a group card sort, the number of participants in a group should be multiplied by the number of groups involved in the experiment. The following recommendations are made for group sorts: A recommendation is made for five groups of three participants (Mauer & Warfel, 2004). See Table 1, analysis 9. Four to eight participants per group, with a sufficient number of groups to represent your various target audiences (Robertson, 2002). See Table 1, analysis 14. In a study by Hahsler and Simon (2001), five to ten participants per group for each of three target audiences were tested, with a total of 20 participants (Hahsler & Simon, 2001). See Table 1, analysis 13. In an analysis of the work of Tullis and Wood (2004), Nielsen (2004) recommends 15 users for a card sorting exercise. Nielsen (2004) states that “for most usability studies I recommend testing five users, since that is enough data to teach you most of what you will ever learn in a test. For card sorting, however, there is only a .75 correlation between the results from five users and the ultimate results. That is not good enough … I think that correlations of .90 for fifteen users or maybe .93 for twenty users are good enough for most practical purposes” (Nielsen, 2004). Nielsen and Sano (1994) use four participants in their study, conceding that “given our discount usability engineering approach with only four users, the statistical methods are not that reliable” (Nielsen & Sano, 1994). See Table 1, analysis 12. In an extensive review of five studies that compare usability evaluation methods, Gray and Salzman (1998) contend, “Low statistical power and random heterogeneity of participants may be regarded as two sides of the same coin. Low statistical power may cause true differences not to be noticed; random heterogeneity of participants may cause noticed differences not to be true. Potential solutions to these problems are to increase the number of participants per group and to consider group differences in the context of individual differences” (Gray and Salzman, 1998). Discussion of Category # 8: Minutes for Card Sorting Exercise The values in this category may be related to the number of cards sorted and to the participant knowledge of the information domain. In general, practitioners expect the participants to finish in one hour or less. Further Expansion of Categories 9 – 11 Categories 9-11 provide an abundance of data that suggest considerable opportunity exists for the analysis of research methods that are complementary to the design of a card sorting exercise. The following categories represent the first activities that are conducted in the design of a card sorting exercise and present a potential basis for future studies that expand on this study of card sorting methods. These categories are critical considerations for successful design of a card sorting exercise. The analysis of the methods discovered within these categories is beyond the Limitations of this research and as such, no properties are assigned within these categories. Discussion of Category # 9: How to Define Target Audiences The definition of the target audience is generally considered essential to a successful design of a card sorting exercise. Fuccella and Pizzolato (1998), Fuccella (1997), Martin (1999), and Hahsler and Simon (2001) specifically list Audience Definition as the first step in their card sorting design process. Practitioner Perspectives on Target Audience Definition “An audience description should include all the qualities that pertain to their interest in the site” (Martin, 1999). See Table 1, analysis 11. “The ability to create usable and useful web site designs is highly dependent on the availability of a crisp audience definition” (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. Survey as a Method for Defining Target Audiences “The easiest and most cost-effective means for collecting audience definition data is to conduct a survey” (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. This concept is reinforced by Fuccella (1997). “As a first step, the target groups of the information system have to be defined …the project team might use existing customer information [however] only a precise knowledge of user needs enables the development of web sites with high user value. To obtain this kind of information, market research data can be extended by user surveys…” (Hahsler & Simon, 2001). See Table 1, analysis 13. Other authors imply that the audience definition already exists or is easily determined, such as in the redesign or design of an internal Intranet or audience specific portal. In these references, the selection of participants and content for the card sort exercise are directly tied to this audience definition (Faiks & Hyland, 2000) (Akerelrea & Zimmerman, 2004) (Kidwell & Martin, 2001) (Nielsen & Sano, 1994) (Robertson, 2002). Discussion of Category # 10: How to Select Participants The selection of participants shows substantial interrelation to the target audience definition and as such, evaluating Category # 9 – How to Define Target Audiences should precede this analysis. This analysis elucidates the interrelationships between the target audience definition and the selection of participants. Selection of Participants – Relationship to Targeted Audiences “The participants should reflect the breadth in abilities, jobs, and environments of the targeted user community" (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. "The most important aspect of selecting participants is that they come from and are representative of your user group. If you have multiple user groups, it is important to include a representative sample from each group" (Mauer and Warfel, 2004). See Table 1, analysis 9. “The attendees at the card sorting session must be the actual end-users of the system you are building” (Robertson, 2002). See Table 1, analysis 14. Other statements within the literature also suggest that random or casual selection of participants is not widely practiced and that the participant selection, although the selection may be randomized within target audiences, it is intended to reflect the target audience. This inference is suggested by the relatively precise definition of the participants selected in the following case report excerpts. “The target audience was identified as all employees … [those] employees who had assisted in the collection and characterization of the planned content were disqualified [as participants]… thirty [participants] were selected, including representatives of each of the company’s office divisions” (McGeorge & Rugg, 2003). See Table 1, analysis 4. “Because the [information resource] is intended for all users of the university population, the committee chose a random sample from the academic community ... both experienced and novice … users were welcome. The study populations consisted of [undergraduate and graduate students, faculty, and staff members]” (Faiks & Hyland, 2000). See Table 1, analysis 6. “Fourteen experienced UNIX users … participated in the study” (Dearholt et al., 1986). See Table 1, analysis 8. Deaton (2002) reports that in the literature reviewed, participants were selected randomly from a directory, recruiting co-workers, and recruiting from a corporate database (Deaton, 2002). See Table 1, analysis 10. Discussion of Category # 11: Content Selection Process A wide range of seemingly disparate information items may become potential candidates for inclusion in an information resource, in particular, a web site. The identification of current and potential content and the labels that are applied to the cards are important considerations in the design of the card sorting exercise. Below are listed a few content items that demonstrate the range of “objects” that have been included in card sorts. Content Object Definitions Fuccella and Pizzolato (1998) provide these examples of “content objects” to be included on software marketing website (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. White papers FAQ’s Downloadable code Call-in support numbers Success stories Robertson (2002) suggests looking to these sources for generating a list of content. (Robertson, 2002) See Table 1, analysis 14. Existing online content Descriptions of business groups and processes Planned applications and processes Potential future content Ahlstrom and Allendoerfer (2004) include these information items in their description of content considered in the design of an employee portal (Ahlstrom & Allendoerfer, 2004). See Table 1, analysis 5. Existing employee intranet Several human resource systems Employee directory Accounting and tracking systems Management information systems Email and collaboration systems Library card catalog Other practitioners have suggested various methods for identifying content to include in a card sorting exercise. Methods for Content Identification Hahsler and Simon (2001) suggest these methods for identifying content (Hahsler & Simon, 2001). See Table 1, analysis 13. Web server log file analysis. Advantages - “convenient and inexpensive” Disadvantages - “can only consider existing objects” and “may be misleading [for a number of technical reasons]” Analysis of search engine queries. Advantages - “helps identify the most frequently requested keywords” User survey. Advantages - “a reliable method for identifying the most important web objects” Disadvantages – “very expensive” Fuccella and Pizzolato (1998) outline a number of “Requirements and Task Gathering” processes for the identification of content for a web site (Fuccella & Pizzolato, 1998). See Table 1, analysis 2. Focus group. Advantages – “can collect large amounts of data in a short period of time” Disadvantages - “costly, usually requires a professional facilitator or moderator” Iterative survey – a first survey of open-ended questions reveals similarities in requirements. A second survey compiles the results and survey participants then rank them in importance. Advantages – remote participation (electronic surveys), large sample sizes do not significantly increase cost or data analysis. Disadvantages – time consuming, expensive Exploratory Surveys – “ask the users to list the specific content they would like to have on the site.” Advantages – inexpensive and simple and it is easy to “survey a large sample in a relatively short time” Disadvantages – data is difficult to analyze Scenario Building Exercises Advantages – inexpensive and simple, users can more easily identify tasks Disadvantages – one on one research is time consuming Competitive Review Advantages – inexpensive and simple Disadvantages – time consuming Nielson and Sano (1994) suggest the development team was responsible for identifying content (Nielsen & Sano, 1994). See Table 1, analysis 12. Brainstorming. The development group discussed and agreed on the content. Discussion of Category # 12: Term for Content Selected This category is not specifically related to the design of a card sorting exercise but is included for the semantic value of the terminology used to depict the content defined and the labeling of the cards sorted in the exercise. Practitioners consider a large array of information that may be included within the information domain. Specifying a name for the cards is in itself a daunting semantic task. Below is the list of conceptual terms that practitioners use to describe the content on the cards: Idea units Content objects Elements Information items Concepts Web site objects Commands Content labels Objects Content Web objects