7. Chapter 7: Manage, Monitor and Measure
by Janaki Mythily Kumar and Mario Herger
"Gamification is not a project...it's a program that gets invested in for the long-term. Those that understand that see the most impactful and meaningful results."."
— Kris Duggan, Founder of gamification platform Badgeville
7.1 Creating a sustainable gamification strategy
We recommend thinking of gamification as a program in your organization, rather than a project. Sustainable gamification takes long-term commitment and iteration to be successful.
Playtesting is the type of usability testing done by game design companies, where players are observed while they play with the product. Plan to playtest your gamification designs throughout the design process at different levels of fidelity before eventual rollout.
To ensure the success of your gamification efforts, the three essential aspects to consider are managing the mission, monitoring player motivation and measuring the metrics to fine- tune the gamification strategy continuously.
7.1.1 Manage the mission
Managing expectations with regard to the mission is an important aspect of ensuring success for your gamification efforts. Chapter 4 described how to identify a mission, and encouraged you to strive for SMART mission statements — specific, measurable, achievable, realistic, and time bound. Share the mission with everyone in the team to create a common vision. Periodically evaluate the validity of the mission and adjust if needed.
We recommend planning for a few releases looking into the future and not just for the immediate one. The mission could stay the same over multiple releases or be modified for each release. For example, in a community site, the mission for the first release is to encourage participation, and in a subsequent release, the mission may be focused around increasing quality of content.
7.1.2 Monitor player motivation
After the implementation of gamification, conduct qualitative research through observation or interviews to study the impact on player interaction, delight and motivation.
It is important to plan for player fatigue. When gamification is introduced, players may be engaged and delighted. However, as time goes by and the novelty wears off, player engagement and delight may go down as well.
One strategy to counter player fatigue is to plan for a few releases ahead and introduce new features periodically to sustain novelty and interest. A similar strategy is to consider the players' journeys and their level of expertise with regard to the gamified system. Introduce a small set of features in the beginning as the player is onboarding, and unlock more functions as the player acquires more skills.
Traditional games do this by allowing players to graduate to more challenging levels to keep them engaged. Amy Jo Kim, describes the Player Life Cycle with respect to game design. She refers to the players as Rookie, Regular or Master based on their expertise, and designs the game appropriately around their needs to avoid dropout.
Courtesy of Janaki Kumar and Mario Herger. Copyright: CC-Att-ND (Creative Commons Attribution-NoDerivs 3.0 Unported).
Figure 7.1: Player's life cycle by Amy Jo Kim
This means the team has to plan a roadmap of gamification features a few releases ahead, and not just the immediate one. They also need to consider the players' skill level to sustain their engagement and delight.
7.1.3 Measure effectiveness of mechanics
Brainstorm with the product team to identify the metrics i.e., the key performance indicators of gamification success. Then find a way to get this data. For your reference, here are a few examples of common metrics.
- Average number of actions
- Median number of actions
- Number of users performing actions
- Number of times users return
- Progression of users through the experience
- Customer satisfaction results
- Decrease of response time
- Number of active users
- Productivity increase
- Cost reductions
- Sales increase
- Article ratings — in case of community site
- Fields filled out — in case of sales automation tool where the mission is to encourage sales reps to enter data completely for accurate reporting
Be selective in identifying the metric since it requires data collection. Note that the more data you collect the more storage and processing requirements you generate. Big data is a topic that is relevant to gamification. Plan for big data, and consider the performance impact on the players.
7.2 Tips for success in a corporate setting
There are some special aspects to gamification in a corporate setting and some tips to help you.
7.2.1 Get a gamification sponsor
Identify an executive in the company to sponsor your gamification efforts. This has many benefits. First, it gives you a budget to work with. We will discuss why this is necessary below. Secondly, there is still a stigma attached to playing at work. Employees not familiar with gamification may be skeptical, and you might end up spending much time countering this skepticism. Getting an endorsement from a high level executive can help your case, since you can use these executives as enlightened role models.
Of course, executive sponsorship is always a two-way street. Once you get a sponsor, you will need to keep your sponsor regularly updated on your activities in addition to sharing your success stories and lessons learnt. This has the additional benefit of indirectly recruiting an executive to be your public relations spokesperson in meetings in which he or she is present and you are not.
Here are a few line items you will need to consider as part of your gamification budget:
- Creating the gamification experience
- Game designers
- Graphic designers
- Maintaining the game
- Updating the game
- New Missions/Challenges
- Adding Challenge Levels
- Gamification Platform license fees
7.2.3 Prepare for rollout
There are two strategies for rollout.
220.127.116.11 Big-bang rollout
This involves a standard software maker rolling out gamification to all customers, or the entire user base for internal teams. To do this successfully, the team will need to invest in publicity and marketing of gamification to gain support. There is a risk and cost associated with this approach. We do not recommend this unless you and the product team have a high level of confidence in the gamification strategy.
18.104.22.168 Start with a pilot, monitor results and roll out slowly
For first releases of gamification, the recommended approach is to start with a pilot, gather feedback from players, and roll out in phases. This is a less risky strategy.
7.2.4 Opt in or mandatory?
If participation in gamification is mandatory, it will take the fun out of it. Players should be able to able to stop playing at any time. In the chapter on Legal and Ethical considerations, we will elaborate on labor laws that protect employees in some countries.
7.2.5 "Hello help desk? My points are missing!"
Employees and customers have different set of expectations when they are interacting with consumer software or enterprise software. If a person uses a consumer website that offers some gamification points and badges, and if some of the points are lost due to a technical glitch, the consumer may be annoyed briefly, but will move on. If an employee has earned points for technical expertise and if those points are lost due to a technical error, the employee may go beyond mere annoyance and call HR to complain. Plan for such scenarios with data backup and rollback.
During early releases of the product, when the platform is still maturing, such backup and restore services may not yet be available. In such cases, make players aware that the company cannot provide any guarantees for their points earned in the current release, and allow them to opt out at any time.
Once you have developed a gamification strategy and implemented it, you are not done. You have just begun. In this chapter, we point out ways to create a holistic and sustainable gamification strategy for the enterprise. We also provide some practical advice pertinent to a corporate setting.
7.4 Insights from SAP Community Network
To make SCN successful, administrators have established some rules and metrics, and monitor the activities of the community.
- All players are required to treat each other with respect and with courtesy.
- New users were required to learn by following the forums, browsing the blogs, and using the search tool before asking a question.
- More experienced users can blog and access advanced features of SCN.
Metrics that were monitored changed over time. The most common were:
- Number of registered users
- Number of active users<
- Number of top contributors (above a certain point threshold per period)
- Average response times to forum questions
- Other metrics like the Net Promoter Score, which is a measure of the satisfaction of members in the community.
Monitoring was not restricted to quantitative metrics. Administrators and moderators (community members who had earned the status of moderator) proactively monitored the community and focused on the quality of the discussion. They enforced the agreed upon SCN rules by punishing rule-breakers and cheaters.
Whenever there are rules, and benefits to be gained, cheating will happen. It was no surprise that some members of the SCN also tried to cheat the system to earn more points. One of the most common forms of cheating is when a member creates multiple accounts, asks a question with one name, and responds with another name. Both the questions and the answers were often very simple and a repetition of information that was already available on the SCN. Regular SCN members felt that this was a dilution of content on the community that made searching for helpful information more difficult. This led to regular members voluntarily taking on the role of governance. They developed a sharp eye to identify such behavior, and they reported such incidents to the SCN administrators who either warned the offending member or suspended them.
This shows that rules need to be enforced to keep the community valuable for compliant members who form the majority of any community.
SCN administrators proactively rewarded good behavior and sought ways to enrich the SCN experience. For example, players who reached a point threshold received a t-shirt with their SCN points printed on them. Members proudly wore it at SAP events and locally organized SCN events as status symbols. Eventually this practice was discontinued because the community proposed SAP could donate the money to charity on their behalf.
The community members' employers started to monitor their employees' activities on SCN. Due to the openness of the system, they could easily keep track of the number of contributions, as well as their quality.
Consulting companies would publicize their employees' SCN ranking to promote their services. This led some companies to incentivize employees to contribute by adding this as an objective to their performance evaluation. Companies that needed experts on their projects, staffed them by searching for them on SCN. Hiring managers indicated that they preferred candidates who listed their SCN point score on their resumes. Through SCN, they could examine the quality and rating of the contributions, since the track record is publicly available. This raises some legal and ethical questions, which we will examine in the next chapter.
To be continued at the end of next chapter...