In order to make effective decisions, it is important to be clear about the criteria you are using to decide between different options. However, identifying and applying decision criteria can be tough when each of your options involves a diverse set of characteristics and possible outcomes, and especially when multiple stakeholders with different opinions are involved.
In this article, we’ll define what decision criteria are, include examples of decision criteria, and provide tips and best practices for identifying and applying decision criteria in a way that leads to successful decision-making.
What are decision criteria?
Briefly put, your decision criteria are the things that matter to you when prioritizing or deciding between different alternatives of interest.
For example, when deciding on different ways to manage freshwater resources, what makes one management policy better than another? When choosing between different business projects to invest in, what indicators determine whether one project is a stronger strategic fit for the company than another?
Getting your decision criteria right is crucial for making sound decisions.
Decision criteria examples
Decision criteria will be different depending on the decision problem at hand. Here are some decision criteria examples for specific decision problems:
- Selecting vendor or supplier for procurement: price, reliability, quality, ethical sourcing, delivery speed
- Prioritizing public infrastructure projects: impact on citizens’ quality of life, environmental impacts, urgency, alignment with strategic urban planning goals
- Choosing new health technologies for hospital: clinical benefits, society-wide benefits, strength of evidence, extra training & supervision required, fit with organizational strategic direction
- Determining corporate social responsibility initiatives: social impact, impact on brand reputation, alignment with company values, feasibility
- Prioritizing areas for forest conservation: biodiversity richness, threat of deforestation, socioeconomic & cultural importance, feasibility
Qualitative vs quantitative criteria
Decision criteria may be quantitative, meaning they can be measured in numerical terms, or qualitative, meaning they cannot be directly measured. Some examples of quantitative criteria may include:
- Distance (in meters, kilometers, miles, etc)
- Time (in hours, days, months, years, etc)
- Population (in number of individuals)
- Revenue (in dollars, euros, etc)
On the other hand, qualitative criteria may include:
- Level of urgency
- Social impact
- Quality
- Difficulty
While it is relatively straightforward to measure performance on quantitative criteria, measuring qualitative criteria is trickier.
Where possible, it is better to be specific rather than general so that people have a common understanding of qualitative criteria – for example, what does “medium impact” mean? However, such imprecision is fine in some applications, as people have an approximate or “gut feel” understanding.
To quantify qualitative criteria, possible rating scales may include:
- very low, low, medium, high, very high
- poor, fair, good, very good, excellent
- none, a little, some, most, all – e.g. “of the time”
- no, minor, neutral, moderate, major – e.g. “effect”
- never, rarely, sometimes, often, always
Although five levels are common for rating scales, so are two, three, four, six and seven levels. Part of the “art” of specifying your criteria is figuring out how many levels to include and what words to use.
How to identify decision criteria
Arguably one of the trickiest but most vital aspects of successful decision-making is correctly identifying the relevant decision criteria. Oftentimes, decision-makers are inclined to believe that all attributes or characteristics of the alternatives of interest are relevant to their decision, and list them as decision criteria. In reality, many characteristics have little relevance to the decision being made – they are not important for choosing between the alternatives being considered.
Consider, for example, choosing a new car to purchase. Different cars have many different specifications, but something like the number of cupholders or top speed of a car is rarely relevant for most people. Instead, consumers are more likely to focus on the cost, performance, and safety of the vehicle when deciding which car to purchase.
Here we’ll discuss how to identify which criteria to use in the decision-making process, as well as things to watch out for when defining criteria.
Process for identifying decision criteria
The following process is a tried-and-tested approach to defining criteria for decision-making. It is suitable for both single-person and group decision-making.
1. Define the decision problem and key variables
The first step of the decision-making process involves defining the problem at hand and what the overarching goal is, and who should be included in the decision-making process – e.g. subject matter experts, stakeholders, key decision-makers.
2. List your options or possible solutions
Once you’ve defined the decision problem and key variables, list the alternatives you are prioritizing or choosing between. Choose around 10 that are representative of the whole to help you identify criteria at the next step.
If you do not have any specific alternatives defined – as may be the case, for example, when creating a patient prioritization tool for repeat use, or creating a tool to measure biodiversity of urban gardens – create fictional alternatives for the sake of this exercise.
Whether you are using real or imaginary alternatives, write a description with relevant information for each alternative.
3. Rank your alternatives intuitively
Rank your alternatives, or a subset of them, based on your intuition. As you do this, ask yourself why you are ranking one alternative higher or lower than other ones.
Discussing why the alternatives are ranked differently by different people forms the basis of generating your criteria. These discussions reveal which characteristics of the alternatives influence how you might choose between them; in other words, what you are using as decision criteria, as opposed to being mere attributes of little relevance to the decision being made.
If you are doing this exercise as a group, it may be useful for each group member to rank the alternatives independently on their own first, and later as a group. Doing so helps the group members warm up for the group ranking exercise.
Ranking alternatives independently first also reveals the variability present between people’s individual intuitive judgments, demonstrating the need for a more structured decision-making process. This process is also called a “noise audit”. 1000minds decision-making software offers ranking surveys to support these exercises.
Common mistakes to avoid
Having a sound decision-making process means selecting the right criteria. Here are common mistakes people make when drafting their criteria:
1. Including too many criteria
One of the most common mistakes is including too many criteria in a decision. In general, having 5-7 criteria in a decision is ideal. This allows for focused evaluation without being overwhelming. Having more criteria creates additional complexity and time spent in refining them and scoring alternatives, as well as difficulty in communicating results. This can easily become overwhelming and an inefficient use of time.
Furthermore, including too many criteria may adversely affect cognitive performance, as it becomes difficult to think about too many criteria at once. This increases the risk of decision fatigue and reduces decision quality.
If you have a large number of criteria and aren’t sure how to trim them back, here are some approaches that may help:
Rank your list of criteria and choose the top 5-7 to focus on in your decision.
With 1000minds, you can either use a ranking survey to rank your criteria intuitively, or you can create a decision (or survey) with the weighing method set to “pairwise sorting”. Pairwise sorting involves answering a series of simple pairwise comparison questions to produce a ranked list of criteria.
Whichever method you use to rank criteria, choose the top 5-7 criteria from your final list to use in your decision.
Combine related criteria.
For example, when choosing a new vehicle, you might consider the carbon footprint and energy consumption as part of your criteria. These two can be combined under “environmental sustainability”. Or, when prioritizing public infrastructure projects, “public pressure” and “risk of failure of existing systems” can be combined under “urgency”.
Combining related criteria helps you reduce complexity without losing important factors that influence the decision.
Run a 1000minds preferences survey with some criteria marked as optional.
This way, your decision-makers or stakeholders can select which criteria are most important to them and reveal their preferences only for these criteria. This allows you to include a larger number of criteria without overwhelming decision-makers with too many criteria to weigh.
You can also group related criteria together and include instructions for survey participants to select a particular number of criteria from each group, as in the example below.
2. Unclear wording of criteria
Making sure that your wording of the criteria is clear and unambiguous is important so that everyone has a common understanding of the meaning of the criteria. Try to be specific where you can.
Sometimes, you might already be in the process of weighing the criteria against each other, and only then realize that the wording is unclear. That’s okay – you can always go back and refine the wording, then come back to the weighing exercise.
To avoid such issues, it is also helpful to run a small test session to ensure that the wording of the criteria makes sense to the people involved.
3. Criteria that are not preferentially independent
Try to avoid overlap in your decision criteria, so that you do not risk double-counting. Including two highly dependent criteria can cause an alternative to receive a higher rating unfairly, distorting the final ranking. This can lead to a biased decision where some factors are overemphasized just because they appear multiple times in different forms.
For example, rating projects on both their strategic fit and business benefit may cause problems, as it is unlikely that a project with a high strategic fit would bring no business benefit. In the same vein, rating a car on both its style and its beauty would result in dubious results -- if you like the style of a car, you will find it more beautiful, so the two are not preferentially independent.
However, some criteria may be preferentially independent without being fully independent of each other. For example, a phone’s weight is related to its screen size, but your preference for screen size is unrelated to your preference for the phone’s weight. Such criteria are ok to include.
How to evaluate and prioritize decision criteria
Now that you have identified your decision criteria, you need to figure out their relative importance with respect to making the decision at hand.
For very simple applications, pairwise comparisons and direct rating can be used without the need for software. Direct rating involves assigning point values (perhaps on a scale of 1-10) for each criterion based on their perceived importance. For most applications, however, we don’t recommend direct rating methods because they are inherently inaccurate and subject to bias.
Criterion weighing methods that are suitable for more complex problems are generally based on weighted-sum models. Due to the methods’ technical sophistication, they are often implemented using specialized decision-making software such as 1000minds. You can find a list and description of commonly used weighing methods here.
We recommend weighing criteria using the PAPRIKA method due to its transparency, scientific validity and ease of use.
Using 1000minds’ PAPRIKA method to evaluate criteria
PAPRIKA was designed to weigh criteria in a way that is as cognitively simple and user-friendly as possible, as well as scientifically valid and reliable. The method involves the decision-maker answering a series of simple pairwise comparison questions prompting them to choose between two options defined on only two criteria at a time. Each time, the decision-maker needs to make a trade-off.
The advantage of such simple questions is that they are the easiest possible questions to think about. This means that decision-makers can answer them with more confidence, which creates more reliable results.
But PAPRIKA is more than just pairwise comparisons. As decision-makers answer each question, the algorithm adapts. Using the logic of transitivity, PAPRIKA rules out redundant questions to ask. This minimizes the number of questions decision-makers have to answer, without sacrificing accuracy.
PAPRIKA has been used for applications ranging from personal decisions like choosing a course of study at university, to high-level decisions like creating environmental management policies or prioritizing diseases for research globally.
How to incorporate criteria and sub-criteria in 1000minds
Sometimes, projects will involve a hierarchy of general criteria and more narrowly focused sub-criteria under each criterion. Similarly, another approach is to specify overarching objectives with specific indicators for measuring an alternative’s performance on each objective.
1000minds decision-making software is designed with a single layer or tier of criteria rather than a hierarchy so that the software is user-friendly.
In applications requiring a hierarchical approach, it can sometimes be a bit of a challenge to “flatten” the hierarchy of criteria and sub-criteria for use in 1000minds. Fortunately, three methods are available.
Method 1: Incorporate sub-criteria implicitly
The most common approach is to make your main criteria the subject of your work within 1000minds, and implicitly include any sub-criteria simply by acknowledging them as being part of the full definition of the criteria. This can be done via checklists or examples.
This prompts decision-makers to think about the sub-criteria as they assess how the alternative performs on each criterion. To rate alternatives on criteria, you can send out a 1000minds categorization survey to your group of decision-makers.
This approach generally works well for most applications.
Method 2: Build a separate model for each criterion, then combine
Another approach is to create a 1000minds decision model for each main criterion and enter the sub-criteria as regular criteria in the model. You will also need to create an “over-arching” model for the top-level criteria as well. This approach is more logistically challenging.
Then, establish weights for the sub-criteria in each model, as well as the for the criteria in the “over-arching” model, through 1000minds trade-offs as usual. Finally, apply the sub-criteria weights as well as the over-arching criteria weights, resulting in the alternatives being prioritized accordingly.
Method 3: Build a single model with all sub-criteria as top-level criteria
The third approach is to include all sub-criteria as top-level criteria in 1000minds. The drawback is that the more criteria (or, in this case, subcriteria) you have in one model, the more questions decision-makers will need to answer in order to establish weights, risking them feeling overwhelmed.
As discussed earlier, there are two solutions to this issue: trim down the number of sub-criteria, or run a 1000minds preferences survey in which decision-makers select the sub-criteria they care most about for inclusion in their survey. You can read more about these processes under “including too many criteria” in our list of common mistakes to avoid.
Conclusion
Identifying the correct decision criteria is crucial for sound decision-making, but it’s an area that many decision-makers struggle with.
Understanding how to identify criteria that are relevant to the decision at hand, and being aware of common mistakes, can improve decision processes and outcomes.
1000minds decision-making software provides tools to identify and weigh criteria in a transparent and scientifically valid manner, helping you not only identify the correct criteria, but also make decisions that you can be confident in.
Try a free 15-day-trial today, or book a free demo to see how 1000minds can help your organization make better decisions.
Share this post on:
