fyp-buglens/VideoGameReviews-ActionWhenNotAllowed-BERT

Action When Not Allowed Detection Model

This model is trained to detect Action When Not Allowed issues in video games. These issues occur when players are able to perform actions that should not be allowed under specific conditions, such as when the game is paused or during a cutscene. These bugs can significantly affect gameplay balance and fairness, creating unintended gameplay experiences.

For example:

• In Global Agenda’s 1.3.2 patch, a bug in the auction house allowed players to create money out of nothing, breaking the game’s economy.
• In League of Legends: Season 11, there was a bug that allowed players to one-shot enemies using weapons that should not have permitted this action, disrupting the intended game mechanics.

Model Details

• Model Type: Classification (Binary)
• Training Data: The dataset for this model was created from scratch by scraping reviews from the Steam platform. These reviews were labeled into different categories, including Action When Not Allowed, to train the model. The model is based on concepts from the paper "What went wrong; the taxonomy of video game bugs"
• Task: Bug detection in video games related to actions being taken under invalid conditions, such as while the game is paused or during a cutscene.
• Intended Use: The model is designed for game developers and QA teams to detect issues where players can perform disallowed actions, disrupting the game’s intended behavior.

Training Metrics

The model was trained for 4 epochs with the following performance metrics:

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score	Precision	Recall
0	0.6062	0.6198	66.33%	0.6017	0.7764	0.4912
1	0.4141	0.4691	77.72%	0.8046	0.7369	0.8861
2	0.4946	0.5182	74.97%	0.7140	0.8746	0.6031
3	0.3118	0.4989	79.45%	0.7972	0.8152	0.7800

Key Metrics:

• F1 Score: Balances precision and recall, with a final value of 0.7972 after 3 epochs.
• Precision: Measures the accuracy of positive predictions, reflecting how many of the predicted disallowed actions were correct.
• Recall: Measures how well the model identifies actual disallowed actions, capturing the true positives.

Intended Audience

• Game Developers: To detect issues where players are allowed to perform disallowed actions during specific in-game conditions, such as when the game is paused or during cutscenes.
• Quality Assurance (QA) Teams: To automate the detection of action-related bugs in video games.
• Researchers: Interested in exploring and extending bug detection models for action-related issues in video games.

Limitations

• This model is trained specifically for detecting Action When Not Allowed issues, focusing on detecting cases where actions are performed under invalid conditions (e.g., paused games, cutscenes).
• It does not account for other types of bugs, such as those related to performance or graphical issues.
• Performance may vary depending on the game’s context and system configuration. Fine-tuning may be required for different games or environments.

How to Use

You can use the model for binary classification to predict whether a given game state exhibits an Action When Not Allowed issue. Here's an example using the Hugging Face transformers library:

from transformers import pipeline

# Load the model from Hugging Face
action_detection = pipeline('text-classification', model='fyp-buglens/VideoGameReviews-ActionWhenNotAllowed-BERT')

# Example usage
result = action_detection("The player is able to one-shot enemies while the game is paused")
print(result)  # Output: label indicating if it's an action when not allowed issue or not