The use of simulations in grading student performance: challenges and opportunities

Introduction

It is well known that simulations, particularly those employed in business schools, have been highly effective in benchmarking student performance. The logic went like this for the most part: Student A has performed better in the simulation than Student B, but it is not necessarily the case that Student A should be awarded the higher grade. We'll do our best to explain.

We would like to present a few reasons why assigning grades based simply on the results obtained in an educational simulation could potentially lead to a lowering of educational standards. We will also suggest some potential ways in which you, as a teacher, could seek to mitigate the risks associated with a simplistic approach when it comes to grading with the help of simulations. 

At Simbound we have been lucky enough to notice firsthand how simulations are a valuable learning activity for many classes. Incorporating memorable exercises that engage learners, instill confidence, and prepare them for the world of work are a valuable addition to courses. We observed that grading is not a mandatory part of the assessment of learning with simulations. In some cases, it might even be advisable to avoid it altogether.

Ultimately, we believe there are arguably a series of highly relevant phenomena surrounding the use of simulations in education some which are more in need of improvement than those of grading. Read on to find out more.

Thought-Provoking Exercise

Here is a thought-provoking exercise. Imagine that we are teaching a class of several teams of three students each, all making decisions in an online simulation of an e-business.

1. Team Blue has enjoyed considerable success, with earnings of 100,000. 
2. Team Green has also performed well, with earnings of 50,000. 
3. Team Purple has experienced a more modest level of success, with earnings of 15,000. 
4. Team Red, however, has encountered some challenges, with earnings of -10,000.

It would be understandable to assume that Team Blue has had the best run in the simulation and therefore should be awarded the highest grade. This is all to easy to conclude just by looking at the results chart. To continue on this thought-provoking exercise, we will add more information:

1. It become apparent that Team Blue has discovered a way to exploit a hidden malfunction in the system, which may have contributed to their decision to explore only a limited number of options and learning materials. Given that their leading position was never in jeopardy, it's understandable that they might have focused their efforts in this way. 
2. We uncover that Team Green has had just one of the three students contributing to the decision-making process on a constant basis, with the other two being mostly absent. 
3. Team Purple has invested a great deal in marketing and R&D, which will likely yield positive results in the next stages of the simulation.
4. Team Red may have made a minor mistake when inputting their decisions, inadvertently adding one extra trailing figure to their budget that was not intended. This led to them incurring a loss. 

In light of this new information, how should one go about grading this class? 

We hope you will understand that we deliberately avoid mentioning the individuals' prior experience with the topics being simulated or each member's readiness to use technology. Similarly, we do not indicate how collaboration took place in each team or if there was a decision-making system in place at every team and how performant this was. With its prevalence in Education the use of AI should also be considered. We believe that in order to maintain a high quality approach including all of this information in the grading process would be necessary. 

This basic exercise was designed to show you how grading based solely on results obtained in the simulations alone should be approached with caution. Given the many limitations that exist we still believe however that using a sound methodology in place, grading can become more fair and effective. 

Proposed approaches:

1. A better understanding of simulations

It would add value to this article to provide some additional context. It seems that the majority of experiential learning systems in use today are becoming a somewhat passive tool for teachers, which they can set up and then leave to run independently. The most modern simulation platforms and the learning experience they provide may not require frequent human intervention from tutors, as they embrace the ‘set and forget’ principle by dispatching all of the essential components: ready-made guidebooks, videos, AI-assisted support and guidance, various analysis reports and more. This could create a disconnect when it comes to teachers understanding the conditions created by the simulation and the actual learning experience of students. 

Perhaps the solution here would be for each teacher to familiarize themselves with the models, algorithms and instructions provided by the simulation in order to gain a deeper understanding of how the results obtained by the participants were created, which would in turn facilitate more accurate grading. However, this can sometimes present a challenge, as simulations often comprise complex systems that undergo numerous changes and updates throughout the year. It is therefore possible that the grading methodology employed last academic year may no longer be suitable this year. 

2. Spending more time with students

Rather than focusing exclusively on numbers, we could consider the role of the teacher to be that of a supportive coach, who encourages learners to explore, take risks and interpret the concepts being taught and simulated in class in different ways. Perhaps, instead of awarding grades simply based on numerical results, one teacher could consider assessing the level of knowledge and motivation of students at the beginning of the simulation experience. Future research that will be carried out by Simbound together with academic partners should shed more light on this. It might also be helpful to add new indicators such as new concepts learned and whether reasoning or the approach manifested in tackling complex tasks has evolved post-simulation.

We would like to suggest that teachers consider moving away from the front of the class (or screen) and instead take a more side-by-side collaborative position next to their students. This could lead to new insights and ways of learning together. Fortunately, there are already analysis and evaluation tools in place that make it relatively straightforward to identify areas for improvement for every participant to a simulation game. 

3. Develop grading based on other experiences

It would be preferred that students are encouraged to explore entrepreneurial thinking by being introduced to additional what-if scenarios, and examining performance and the underlying factors that contribute to it. It would be worthwhile to consider skills that are relevant to companies and that could potentially give candidates an advantage over their peers in a competitive market for young professionals. It may be interesting to consider pairing up simulations with real-life assignments and case studies. This could provide more data and help design a more robust framework for grading and learning.

Conclusion

It would be worthwhile for us all in the Education sector to consider new ways in which we could simplify the process of grading with the help of simulations. By doing so, we may be able to make this an easier and more standardized procedure in the future. We will need to bring together all our collective expertise and resources to make this a reality. The effort will undoubtedly be rewarded with a superior approach that will greatly outperform what we have in place today. 

Some recent statistics:

1.     Less than 40% of courses in which Simbound is employed use grades from the simulation.

2.     In most cases, grades are combined with that of other assessment, like written exams or reports.

3.     The most common name for a course where 100% of the grade is awarded based on the simulation is "Business Games" or "Simulation of Business functions” These classes help students focus their studies on how  business simulation works and how various modeling and game-based techniques can be used in different situations. 

Louis Havriliuc

Founder & GM @Simbound

We encourage educators the world over to contribute to the latest developments in education with simulations and in grading 👌