Please read the attached document - Improving Decision Making Skills through Business Simulation Gaming and Expert Systems, Alexander Fuchsberger, University of Nebraska, Omaha, 2016. Please answer

Players develop a holistic view of the business, they learn that sometimes alternatives have to be considered and that losses in an early stage might lead to higher profit in a later stage. Business games and simulations became popular over the last 20 years; and they differ in co mplexity, focus, settings or intentions. They are web or application-based and can include random elements. This research aims to design such a business simulation, which allows multiple players to train their management skills in a competitive environment. No perfect utilization can be reached only by the player’s ac tions; other players are influencing the pa rticipant’s outcome as well. Another core element of this research was to provide a setting where an expert system can take a substantial and useful part in su ch a simulation game. The idea was to develop a virtual “mentor”, which acts as an advisor and biases the human player in his or her decisions. Therefore, the primary research objectives are: How can a business simulation game be constructed, in which… ...human players can impr ove their strategic management skills through decision-making in a competitive environment. ...an intelligent agen t (IA) acts as an advisor to improve the learning effectiveness of the players’ skill improvement.

A goal was to prove that participants can improve decision-making th rough interacting with a business simulation with other human participants (competitors). In order to achieve these objectives, a scope and balance for the business model had to be found. The simulation must feature enough complexity to completely cover all major elements of a manufacturing company bu t can’t be too complex, otherwise the learning effect can no longer be measured effectively. A manufacturing company was chosen as business environment because the basic value proposition is more straightfo rward than for example, the business process of a servi ce provider. In its very basic form, the value creation process of a manufacturing company is to purchase resources, produce goods and then sell them. It was also essential to study the effects of integrating an in telligent agent that would act as an advisor to the participants. 2016 49th Hawaii International Conference on System Sciences 1530-1605/16 $31.00 © 2016 IEEEDOI 10.1109/HICSS.2016.107 827 To what extend can an artificial advisor bias the decision making of the participants? As a part of the Design Science methodology, an artifact is created throu gh which knowledge is generated based on the Information Systems Research Framework described by Henver et.al . Identifying the environment of this artifact is necessary; otherwise, it might lead to an inappropriate design and undesirable si de-effects [3]. 2. Literature Review Simulation gaming has been used as a tool by researchers and industry for more than 50 years. Many simulations for educational purposes are without doubt accepted as useful learning tools; however, qualitative simulation games are hardly developed with a specific scientific purpose. Design Science provides an ideal supplement to align such simulations to the scientific community. 2.1. Design Science Methodology To study and analyze behavioral decision making in a business environment Design Science has proven the most appropriate approach for this research. It can be described as a problem-solving paradigms which seeks to create an innovative construct in order to generate knowledge about a phenomenon [3]. While research is an activ ity that contributes to the understanding of a phenomenon, in Design Science the phenomenon can be created artificially, and need not occur naturally [4]. In Design Science, the reality is replaced with an artificial construct that contributes to answer the research problem. This is especially useful, when th e reality is not suitable or too costly to be studied directly. Using this approach allows for a second significant advantage: part of this research is to learn about human perception of an intelligent agent. This agent is already an artificial construct and can therefore be directly integrated into the artifact. To ensure Design Science is science and not only design, it is necessary to prioritize the production of useful knowledge ahead of the production of the artifact. Based on the Information Systems Research Framework by Henver et.al . the business environment is served by IS research through developed instruments. In order to add value to the knowledge base of the scientific community the designed artifact(s) has to be justified and evaluated [3]. In this case, the organi zational strategies are the area of interest in the business environment and they are highly influenced by external factors. Market, suppliers, legal regulation s and competence availability all affect a busi ness and are constantly interacting across the borders of the business. The question becomes which business environment components are effectively re levant for improving the learning process when con sidering a holistic view of such a manufacturing business model, and how such a learning process can become effective through a scientific artifact like a business simulation. Creating this artifact also has an impact on Information Technology . Modern web tools fortunately provide an ideal framework to create an artifact like that required for this research. Graff states that computerized hypertext provides an explicit structure to the material being learned, which is advantageous to learners and encourages them to engage and move around [5]. Using web technologies also allows use of extensive, open-source frameworks, bringing functionality like dynamic charts, sortable tables, animations, form elements and more. To ensure a con temporary instantiation, the simulation has to fully support the majority of modern smartphones. Researchers in Design Science have to be careful to support their research goals through an unbiased design of the artifact. Lainem a described this problem as a challenge of providing acceptable scientific research to the academic community rather than just presenting own opinions [2]. It can be difficult for a simulation designer to distinguish between failure and success due to the fact that no one designs a simulation to fail [6]. 2.2. Approaches to Simulation Gaming Thavikulwat defined a simulation as an exercise of real activities in an artificial environment and a game as an experience, featuring competition and rules [6]. The most discussed topics in business simulation education and learning in recent years are [7]:

Experience accumulation Strategy aspects Decision-making experience accumulation Learning outcomes Teamwork experience Simulations follow di fferent patterns, and they can be classified according to their characteristics. Thavikulwat classified si mulations using computers 828 based on who takes control and how interaction takes place [6]. Basically, two different types of simulations can be distinguished: continuo us and discrete simulations [8]. The gaming industry ha s a more straightforward terminology: real-tim e and round-based games. Specifically, concerning the learning aspect, much research has been perform ed. Simulation-based training is inherently more engaging than other training methods. Knowledge can be gathered more quickly and efficiently, and simulations are simpler to operate but provide a more complex, realistic and manageable learning environment [9]. Besides all positive advantages, simulation research is often criticized for measuring the affective and not the cognitive learn ing, resulting in challenged legitimacy and a lack of firm conclusions [10]. Business games are often accidently considered the same as management games. It is assumed that this equation is only true when the game actually features the management of e.g. a firm, organization, portfolio, teamwork or ot her factors in the business area [11]. Therefore, business games are games with a business environment lead ing to the training of players in business skills or the evaluation of players’ performances. The first known use of games for educational purposes dates back to about 3000 BC in China [2, 7]. The acknowledged beginn ing of business games in the modern sense started in Europe with Mary Birshstein in 1932, who had the idea of adapting the concept of war games to the modern business environment [7]. The artifact in this study builds on a competitive, interactive, deterministic total enterprise mechanism, in which manufacturing acts as the theme. It is played by individuals against each other, supported through the computer. The simulate d time frame is split in turns or ‘periods’. Business games have established a reputation as serious alternative methods for teaching managerial skills. Global organizations are the industry drivers and act as meeting points for researchers and practitioners [10]. Despite proof that simulations are valid teaching instruments, researchers agree that the full potential of business simulation games is not yet revealed [9, 12, 13].

2.3. Expert Systems and Artificial Intelligence in Simulation Gaming Artificial Intelligence (AI) can be described as a broad interdisciplinary field, which may cover elements from computing disciplines to mathematics, linguistics, economics, neuro science and many others [14]. It is sometimes difficult to decide what really belongs under the dom ain. While AI is far more capable in information processing than a human being (as long as the information can be transformed into digital language), it lacks the ability to ‘think’ creatively or critically. An expert system is co mputer logic, designed to gather knowledge and make decisions comparable to a human expert. Mostly artificial intelligence is used for processing data into useful knowledge. AI adaptations in computer games are as old as computer games themselves. “Nimatron,” a machine capable of playing the game “Nim,” was one of the first computerized games, de veloped in 1940 [15]. Until the 70s, AI could only be found in computerized two-player games like checkers or chess [16]. Later in the 70s, arcade games popped up, featuring single-player gami ng against computer- controlled enemies [17]. In the 90s AI was assigned with complex tasks like dealing with incomplete information, path-findin g or real-time decision making and economic planning [18]. Intelligent agents (IA) mark the class of AIs that are often used in simulation gaming and are therefore of special interest for this research. Tecuci developed a comprehensive definitio n of an intelligent agent [14]: “an agent is a knowledge-based system that perceives its environment […] and acts upon that environment to realize a set of goals or tasks for which it has been designed.” Summers adds, that an IA is defined by its ability to determine its own be havior [21]. Artificial agents can be complex, combining multiple different decision-making models. The model of a knowledge- based agent shows how the agent acts with the environment, as can be seen in Figure 1:

Figure 1. Main modules of a knowledge-based agent / expert system [14] Modern intelligent agen ts are capable of learning from the players’ behavior and switch their tactics accordingly. Also phenomena from reality are transferred into computer games. StarCraft 2 oriented their unit movement after the flow of water in a stream; if there is an obstacle in the path, the water 829 flows with reduced speed around the obstacle, considering the available space through the environment and other water particles. Simulations have been the te sting environment for many AI applications like neural networks or crowd simulation [19, 20]. AI in educational business simulations has mostly taken coaching roles so far, and there is a trend towards the integration of AI as decision-support and expert systems [21]. Based on a set of input variables, a knowledge base and a processing algorithm(s) output is generated. Knowledge-based agents may additionally support complex modules like a learning or problem solving engine. Artificial intelligence con tinues to be an emerging domain in computer scienc e. Given the capabilities of AI and intelligent agen ts, much progress in simulation-based gaming an d learning is expected. Contemporarily, AI research benefits from advancements coming from the highly profitable gaming industry [7].

3. The Artifact – Architecture and Design According to the guid elines for design science in IS research [3], the artifact has to be innovative and purposeful. The web-based business simulation suggested in this research has allow students to improve their decision-maki ng skills in an intuitive way. To demonstrate the utility, quality and efficacy of the simulation, methods ha ve to be implemented to allow for a serious evaluation. This is done by measuring the effective improvements of participants (feedback through simulation) and their perceived improvements (feedback through participants). The Business Simulati on for this research is designed for desktop and mobile browsers and provides a simple interface to give participants control over various aspects of their business (Figure 2). A simulation game consists of four participants competing in a closed mark et. The ultimate goal is to finish with the highest accumulated revenue over the period of ten timed rounds in which participants can decide on eight different corporate strategies. Figure 2. The Business Simulation in Action The simulation provides the participants with a lot of information on their and the competitors’ businesses, mostly derived fr om charts and tables. Besides that, the artificial intellig ence tries to provide feedback on advantages and disadvantages of recognized strategies. In each round (period), the same decisions are available, but an ideal strategy is impossible to predict or achieve since the users have to deal with incomplete information, and the competitors’ actions for the current turn can’t be predicted. After the simulation, an extensive analysis is provided, sh owing all the decisions made along with additional charts and tables. 3.1. Decision-Making Decision-making in the Busin ess Simulation is designed with the goal to increase usability to a maximum through a simple, intuitive interface. Learning should happen by understanding the impacts of decisions on the business, not through dealing with numbers and complex mechanisms. Typical organizational strategies bu ild the core of the simulation and they usually feature two extremes which are mutually exclusive. For example managers can focus entirely on product A or B or they have to split available resources among them. Through a slider, users can declare if they fully support strategy A, strategy B or if they are indifferent. Figure 3 shows the mechanism of such a slider: Figure 3. Decision-making Through Sliders Every slider has five stages (increments), and there are a total of seven such strategic decisions that have to be made. The simulation is designed to allow advantages and disadvantages for each position on the spectrum. It is possible that a specific decision is favorable at an early stage in the game, but the opposite strategy is beneficial at a later stage. Additionally, players can toggle on/off a competitor analysis as the eighth decision. The artifact was designed to provid e a holistic set of realistic decisions, and every decision features advantages and disadvantages. A major challenge in designing the artifact was balancing out the strategies, to give each strategy validity at some point, and to give them 830 serious consideration in sp ecific situations. Once a decision is altered, it is immediately updated in the database. Since the simulati on is purely discrete and there are no random elements, only the decisions have to be stored in the database. Calculations on outcomes like revenue can be done on demand, which simplifies the d ata structure and increases the performance of the app. 3.2. Timing and Tutorials The business simulation is time controlled. Users who are done with their deci sions before the time has run out have the oppo rtunity to end their turn prematurely. If all four participants finish early, the simulation immediately proc esses to the next round. Each of the ten rounds is limited to two minutes, after which the game processes to the next round automatically. The first round makes an exception, for which the time limit is five minutes to allow new players more time to get familiar with the simulation elements. This feature has been aligned with the tutorial, which is only displayed in the first round. Instead of the charts, which wouldn’t be of much use in the first round anyway, the players experience tutorial-like text information. This information aims to explain what can be done in each section and what is important. 3.3. Artificial Intelligence (“Eddie”) The AI is implemented as an expert system using deductive rules to observe the market and decisions of the player as well as critical variables resulting from decisions in previous rounds. It is integrated as an ‘advisor’ and it has a face and a name (“Eddie”). Figure 4. Charts, Tables and Decision Feedback As knowledge base all variables associated to the player decisions from the current and last round are taken in consideration. This includes: Player Decisions directly calculated outcomes (e.g. costs of employees based on amount, salary and amount of extra hours) Indirectly calculated / imp lied outcomes (e.g. a resulting market share cumulated over all previous rounds) The most appropriate feedback is determined when the engine is loading all the game variables at the beginning of a new round. An inference engine then generates the outcome in form of subjective feedback to the player. The feedback is a brief verbal statement, either a warning, suggestion or information on current issues the player might look into. The engine compares and evaluates the relevance of a total of 30 different options. During several pre-experiments these 30 feedbacks were identified as most fitting and helpful based on strategic mistakes the players made. The inference engine works by determining and assigning a priority value for the relevant feedback statements independently from other st atements. High priority results are then compared by including the relevant conditions and variables from the other high-priority statements with a reduced impact and a new priority value is generated. The statement which has the highest priority by the e nd of this second step wins, and is displayed to the player. This feedback is selected by best-fit to the current situation, and it is ensured that Eddie never gives the same advice twice in the simulation, should this mathematically happen, Eddie suggests the next best (not already displayed) feedback. He appears to have a personality and may be described as provocative, bold, funny or pushing. His behavior and capability to bias participants is one of the primary research areas in this study. Feedback Examples (11 out of 30):

Pushing up your marketing expenses at the end of the product life cy cle is lost money! Rapid expansion in the beginning might result in huge personal costs and overproduction! Focusing on one product is dangerous. Only do it, when you see potential in this market! Your moral is dangerously low. The productivity in your firm is suffering! You are producing more than you are selling. Try do reduce production to save costs! If you don't start expending soon, you will get behind! 831 We are losing grip on the market. We need more sales! Only risk extra hours, if you need them and you can pay for them! We can't compete with the prices of your competitors. Why not trying out some product development? Your quality is already amazing. Maybe it is time to increase the production and lower costs! You have a good moral! It might be save to ask for a little more of your employees. 3.4. Screens / Pages The simulation is split into five tabs, each focusing on one department of the business. A lobby (before the game starts), a final score page (when the game ends), and a brief description of the implementation of the survey are also included to give a holistic view.

3.4.1. Lobby . The lobby serves as a starting point to direct the participants into the simulation. On this first page instructions and a chat are accessible. Header and Footer provide compact information: Language Switch (German and English) Timer Actual Round / Total Rounds Navigation Tabs 3.4.2. Progress / AI. This initial screen displays a selection of charts that shows, the actual progress in terms of revenue, turnover and cost. A simplified balance sheet is available, and it is also the place where Eddie, the AI, can be found. 3.4.3. Production . On this screen, the player can choose a production focus an d the amount of product development. The company produces two different types of goods, products A and B. They have different resource demands, production times and selling prices. High product development increases the quality of the product and, therefore, the price for which it can be sold, but comes at the cost of a reduced production capacity. 3.4.4. Marketing . In this section, the user is able to influence the whole market. All players start with 10 shares (25% market share each). They can increase their amount of shares between 0 and 4 per round, depending on how much they invested into advertisement and marketing. Additionally, to the variable individual increase, the market also has a natural lifecycle. The market demand increases with the total amount of shares and the market share defines how many products of this global demand customers are ready to purchase from the players. Should the player produce less than he or she can sell on the market, the remaining demand can be exploited by his or her competitors. The second decision the player can make here is to undertake a competitor analysis. This en abled additional information on competitors on various charts for the price of some variable costs. 3.4.5. Personnel. On this screen, players can handle two strategies concerning human resources. They can set a salary level and order extra hours to increase productivity. Both decisions influence the morale (effectiveness) of the workers. A high salary has the disadvantage of causing add itional personnel costs, but is necessary if the morale drops low in the company. The morale is a general indicator for the productivity of each worker. Extra hours increased the production capacity but decreased the morale and caused additional costs.

3.4.6. Strategy. On this screen, players have the opportunity to make strategic corporate decisions.

The first choice is between a rapid expansion over improving the business. Ex pansion leads to more employees, and therefore, to a higher production capacity. Too rapid expansion results in unaffordable personnel costs. Improving the business, on the other hand, simulates improvem ents in internal processes and reduces the time needed to produce products. Users can also decide on a price or a quality focus. A price strategy (cost leadership) leads to discounts, making the pur chase of resources more affordable, while a quality focus impr oves the quality of the products resulting in an alternative method to increase selling prices. 4. Evaluation and Verification Referring back to the research problem, the first four issues were dealt with by developing an artifact that is capable of training human players in decision- making in a competitive environment. The artifact was accessible from any device with internet access, and German and English were implemented as languages. An intelligent agent increased the learning potential further by advisi ng the player on important decisions based on developments during the simulation. These four issues were addressed by 832 collecting feedback from pa rticipants in controlled experiments performed in two stages. In the first stage, two business intelligence university courses with a total of 26 students, held at the Management Center Innsbruck in Austria, served as an environment for the experiments. The simulation was first explained to the students who then performed one game. The students were then led to a survey that include d qualitative and quantitative feedback related to the simulation and AI. After the first simulation and the surv ey, the students were then randomly mixed and prepared for a second game.

The purpose was to see if the overall performance in the second game increased through learning and adaptation of the game mechanics. Also, potentially interesting behavioral pa tterns were identified. 4.1 Survey The survey served two main purposes: first, players were asked about good strategies for the early, middle and late game. The survey tested the artifact against the issues identified in the research problem. The responses were compared to the observed behavioral changes between the first and second round of experiment s. This was done to verify if the players had understood and adapted the concepts of the business simulation. Table 1 shows the number of participants w ho evaluated the chosen strategy as strong during the early, mid or late game. For example only one person felt that using a strong quality-focus (5) ove r price-leadership was a beneficial strategy in the early game. The market research strategy is excluded from the Table since it only had two states (on and off). Table 1. Efficient* Game Strategies Decision Strategy / # participants Early Game (Round 1-3) 1 2 3 4 5 Product Development 1 3 3 11 8 Marketing 1 5 2 7 11 Salaries 7 11 5 2 1 Extra hours 10 6 5 5 0 S1: Improve / Expand 9 5 4 5 2 S2: Price / Quality 5 7 2 10 1 Midgame (Round 4-7) 1 2 3 4 5 Product Development 0 4 9 5 7 Marketing 0 5 11 8 2 Salaries 0 5 14 7 0 Extra hours 3 9 7 6 1 S1: Improve / Expand 2 8 4 8 2 S2: Price / Quality 2 6 6 9 2 Endgame (Round 8-10) 1 2 3 4 5 Product Development 10 5 3 5 3 Marketing 10 6 3 3 4 Salaries 5 2 10 8 1 Extra hours 7 7 5 4 2 S1: Improve / Expand 5 9 4 4 3 S2: Price / Quality 6 5 1 8 5 n=26; (*) participants evaluated with 4 or 5 Second, data about previous experiences and knowledge of business management, along with qualitative feedback related to aspects of the simulation was collected. The last screen included two questions about artificial intelligence to find out about its place in business intelligence and management. The survey asked the students for one positive and three negative observations they made during the experiments. The results were grouped and counted based on similarity. Table 2 shows the amount of individual feedback for the derived group: Table 2. Qualitative Feedback on Simulation Positive Feedback Count Artificial Intelligence 3 Charts 3 Competition 6 Design & Structure 7 General / Idea 4 Learning effects 4 Total (26 expected) 27 Negative Feedback Count Artificial Intelligence 1 Bugs in the Simulation 5 Calculations / Mechanics 4 Design & Structure 15 Explanations 13 Limited Time 7 Total (78 expected) 45 n=26; categories based on similarities in feedback 4.2 Simulation Decision Results The survey and the simulation were both designed to supplement each other. Since the survey asked the students for the perceived best strategies in early, mid and late game, the same aspect was analyzed from the actual performance of the participants. Decisions from the first and second round of experiments were separated to find the behavioral changes influenced by adaptive learning and the impulse through the su rvey. Evaluating all the decisions turned out to be challenging. The limited 833 number of options for each decision helped to reduce complexity in the pr ocess. Clear changes in decision behavior were identified, both between the different stages in the simulation, as well as between the two rounds of experiments. Table 3 shows these behavioral changes in term s of absolute participants, who have chosen the specific strategy: Table 3. Simulation Data : Decision Strategies Decision Experiments (First Round) Experiments (Second Round) Early Game (Round 1-3) 1 2 3 4 5 1 2 3 4 5 Competitor Analysis* 5 64 6 60 Production Focus 10 22 15 21 1 13 12 28 5 8 Product Development 10 21 19 17 2 13 14 11 13 15 Marketing 4 17 22 17 9 10 6 8 19 23 Salaries 0 35 32 2 0 9 29 16 7 5 Extra hours 21 38 10 0 0 27 25 12 2 0 S1: Improve / Expand 10 24 19 13 3 15 17 15 13 6 S2: Price / Quality 4 13 16 25 11 6 22 11 13 14 Midgame (Round 4-7) 1 2 3 4 5 1 2 3 4 5 Competitor Analysis* 22 70 6 82 Production Focus 11 21 31 20 9 14 15 40 13 6 Product Development 21 25 22 13 11 16 18 17 21 16 Marketing 16 19 17 17 23 17 17 13 25 16 Salaries 1 34 38 15 4 8 33 34 11 2 Extra hours 32 30 27 3 0 29 32 18 6 3 S1: Improve / Expand 11 26 26 21 8 17 18 29 18 6 S2: Price / Quality 5 27 23 28 9 10 29 30 8 11 Endgame (Round 8-10) 1 2 3 4 5 1 2 3 4 5 Competitor Analysis* 21 48 11 55 Production Focus 12 18 25 10 4 24 10 19 7 6 Product Development 19 10 20 14 6 18 13 7 10 18 Marketing 25 23 7 2 12 19 16 12 15 4 Salaries 4 24 31 7 3 18 12 24 7 5 Extra hours 30 19 11 9 0 18 20 17 7 4 S1: Improve / Expand 14 18 16 13 8 19 11 10 9 17 S2: Price / Quality 12 19 16 18 4 22 19 11 6 8 This table represents how players decided (1...5) in each stage of the game. The values are representing the absolute amount of occurrences for each strategy chosen.

(*) Competitor analysis has only has two values, true (2) and false (1) The competitor analysis was a feature used consistently by most pl ayers throughout all simulations. The decisions for production focus were more diversified. Producing Product A rather than B was noticeable at all stages. A clear learning effect can be observed in the product development strat egy. Players realized correctly that this strategy im proved the total revenue, especially in the early and midgame, and changed their behavior in the second simulation. The marketing strategy showed an even superior learning effect. While players focused on an average marketing investment strategy during the first round of experiments, they reduced their efforts in the late game. An analysis of the salary showed that most decisions concerning the salary were set statically. Using Extra Hours is a feature designed to generally have most effect in later stages of the game. This was recognized by the players. The final two strategies were more challenging to analyze. There was no ‘ideal’ way to handle these strategies. Players had strong and very different opinions on the usefulness of all strategies; the only thing they agreed on was that taking no preference is a disadvantage.

Improving the business was considered a better alternative in the early game rather than expanding it. This changed in the midga me; in the endgame, the results were equal to those in the early game. A similar result was identi fied in the second business strategy (Price / Quality). According to the survey, players believed either in the success of a price focus or a quality focu s in the early game. In 834 the midgame, people were more indifferen t, resulting in a ‘slight’ or no-preference strategy. Players were more experimental again in the endgame, featuring either a price strategy or a quality strategy. Table 4 shows the relative change in % for a specific strategy from the first to the second round of experiments. For example, only 79 % of the original players who had chosen a light quality strategy (4) in the midgame had chosen it again in the second round. However, these players are not necessarily the same.

Table 4. Relative Behavioral Changes in Decision- making between the tw o Rounds of Simulations Early Game 1 2 3 4 5 Competitor Analysis 2 -2 Production Focus 5 -14 21 -23 11 Product Development 5 -9 -11 -5 20 Marketing 9 -16 -20 4 22 Salaries 14 -7 -22 8 8 Extra hours 10 -17 4 3 0 Improve / Expand 8 -9 -5 1 5 Price / Quality 3 14 -7 -17 5 Midgame 1 2 3 4 5 Competitor Analysis -17 17 Production Focus 4 -6 12 -7 -3 Product Development -5 -7 -5 10 6 Marketing 2 -1 -4 10 -7 Salaries 8 1 -3 -4 -2 Extra hours -2 4 -9 4 3 Improve / Expand 7 -8 5 -2 -2 Price / Quality 6 4 9 -21 3 Endgame 1 2 3 4 5 Competitor Analysis -14 14 Production Focus 19 -11 -7 -4 3 Product Development 0 5 -18 -5 19 Marketing -7 -9 8 20 -11 Salaries 21 -17 -9 0 3 Extra hours -16 3 10 -2 6 Improve / Expand 8 -9 -8 -5 14 Price / Quality 16 1 -7 -17 6 The values represent the relative change in number of occurrences per strategy in % Evaluating the artifact as an instrument was done by collecting and analyzing qualitative feedback of the participants in the expe riments. Eddie proved to be valuable as an advisor an d gained mostly positive feedback as well. Four players mentioned Eddie or the artificial intelligence as a positive aspect of the simulation. Two players appr oved the intelligent and useful help of Eddie after each round; one player wished the advices to be more specific. 5. Conclusion A major challenge in this research was to avoid creating an outcome that automatically favors and validates the research method. W ith that in mind, this research tried to judge su ccess and validity through two different data sources. The success should be measured on how the players experienced the artifact based on their own judgment (survey) as well as how they experienced it based on the decisions they actually made. The research clearly shows that students were adapting strategies as intended by the simulation design. In cases where those general statements were valid (Marketing, Product Development, CA, Salary and Extra Hours) the participants showed a clear learning effect from the first to the second round of experiments (Marketing, Pr oduct Development and CA) or a partial learning effect (Salary and Extra Hours). In strategies that were intended to be situation-dependent, peopl e acted very differently and also claimed different strategies as “optional” in the survey. Considering these clear results and the aforementioned academic problems, the research suggests that all poin ts defined in the research problem could be successfully dealt with, but the final validation has to come from an outside observer. It was also important to find out if players trusted the expert system during the simulation and allowed it to bias their decisions. The results clearly exhibited a user bias, most decisions changed in the second round of experiments according to the advice of Eddie. Also the positive feedback on Eddie indicates the acceptance of the intelligent agent among the players. The participants stated their trust in a potential artificial intelligence in various areas of business management. The students prioritized Market Analysis, Logistic Optimization and Process Optimization as areas with the highest potential to effectively use Artificial Intelligence to suppo rt the outcome. 6. Limitations of the Study A major challenge was balancing the game variables such that each slider became valid, useful and realistic. Intelligent Agent Eddie, although accepted positively and successful in biasing the players, had very limited capabilities behind the scenes. Eddie was not able to learn from the decisions humans made during the simulations, and couldn’t advise based on consi dering more than just the latest round. Since feed back was only shown once 835 per simulation, the most relevant feedback might have been missed out on future rounds. This research, and Design Science in general, may lag behind theory-based research in gathering specific useful findings; however, the findings showed a clear learning ef fect among the participants of the experiment and a positive acceptance of the artifact and the artificial intelligence. It may also be criticized for not studying the aspect of artificial intelligence in enough deep. The expert system had its valid place in the artifact, and the research tried to identify its acceptance and biasing capabilities as an advisor during the simu lation. The focus in this research, however, was not artificial intelligence, but the design of the artifact itself.

7. Future Research The researcher believes th at this research is the start of a far more comprehensive study involving real AI and a remodeling of the artifact. The purpose of the experiments was, be sides providing data for this research, to reveal the weaknesses and problems of the artifact. A lot of feedback has been provided by the participants, making a remodeling of the artifact tangible and viable. In such an upgrade, the following aspects could be addressed:

Improved artificial intelligence Fixing of gaming m echanics and variables Broader and more diversified group of participants Modifications and additions to the artifact Linking the feedback to related decisions for more academic insights Stepping away from this specific research and back into the area of business simulation research, academics anticipate a lot of new development over the next years [21]. Academic research involving both artificial intelligence and simulation gaming is still rare. This study was created to delve in to this nich e and provide a basis on which further research can be continued. References [1] Avramenko, A., "Enhancing students' employability through business simulation", Education + Training, 54(5), pp. 355-367, Emerald Group Publishing Limited 2012. [2] Lainema, T., “Enhancing organizational business process perception: Experiences from constructing and applying a dynamic business simulation game”, Turku School of Economics and Business Administration, 2003. [3] Hevner, A. R., S. T. March, J. Park, and S. Ram, “Design science in information systems research”, MIS quarterly, 28(1), pp. 75-105, Springer, 2004. [4] Vaishnavi, V. K., and W. Kuechler. “Design science research methods and patterns: innovating information and communication technology”. CRC Press, 2015. [5] Graff, M., E. Sadler-Smith, and C. Evans, "Constructing and maintaining an effective hypertext-based learning environment: Web-based learning and cognitive style", Education+ Training, 48(2/3), pp. 143-155, 2006. [6] Thavikulwat, P., "The architecture of computerized business gaming simulations", Simulation & Gaming, 35(2), pp. 242-269, 2004. [7] Faria, A. J., D. Hutchinson, W. J. Wellington, and S. Gold, "Developments in Business Gaming A Review of the Past 40 Years", Simulation & gaming, 40(4), pp. 464-487, 2009. [8] B. Jovanoski, R. Nove Minovski, G. Liechtenegger, and S. Voessner, "Managing strategy and production through hybrid simulation", Industrial Management & Data Systems 113(8) pp. 1110-1132, 2013. [9] Salas E., J. L. Wildman, and R. F. Piccolo. "Using simulation-based training to enhance management education", Academy of Management Learning & Education, 8(4), pp. 559-573, 2009. [10] Clarke T., and E. Clarke, "Learning outcomes from business simulation exercises: Challenges for the implementation of learning technologies" Education+ Training, 51(5/6), pp. 448-459, 2009. [11] Greco, M., N. Baldissin, and F. Nonino, "An exploratory taxonomy of business games", Simulation & Gaming, 44(5), pp. 645-682, 2013. [12] Bell, B. S., A. M. Kanar, and S. W. J. Kozlowski, "Current issues and future directions in simulation-based training in North America", The International Journal of Human Resource Management, 19(8), pp. 1416-1434, 2008. [13] Vos, L, and R. Brennan, "Marketing simulation games: student and lecturer perspectives", Marketing Intelligence & Planning, 28(7), pp. 882-897, 2010. [14] Tecuci, G. “Artificial intelligence”, WIREs Comp Stat, 4, pp. 168–180, 2012. [15] Grant, E. F., and R. Lardner, “The Talk of Town, ‘It’", The New Yorker, pp. 18–19, 1952. [16] Wallace, S. A., R. McCartney, and I. Russell, “Games and machine learning: a powerful combination in an 836 artificial intelligence course”. Computer Science Education, 20(1), pp. 17–36, 2010. [17] Lowood, H., "Video Games in Computer Space: The complex history of Pong–2005", Videoludica Vintage, 15(73), 1984. [18] Schwab, B., “AI game engine programming”. Cengage Learning, 2009. [19] Lam, H.K., and H. T. Nguyen, “Computational intelligence and its applications: evolutionary computation, fuzzy logic, neural network and support vector machine techniques”, World Scientific, 2012. [20] Rao, Y., C. Leiting, L. Quihe, L. Weiyao, Z. Jun, "Real-time control of individual agents for crowd simulation", Multimedia Tools and Applications, 54(2), pp. 397-414, 2011.

[21] Summers, G. J. "Today’s business simulation industry.", Simulation & Gaming, 35(2), pp. 208-241, 2004. 837