fyp-buglens/VideoGameReviews-ActionNotPossible-BERT

Action Not Possible Model

This model is trained to detect Action Not Possible issues in video games. These issues occur when a player is unable to take an in-game action in a specific game state that should be permissible. Examples include situations where:

• In Sanitarium, the character Max gets stuck around corners and claims "Can’t go that way," even when a path is available.
• In Crysis, the final boss becomes randomly ‘untargetable’ (and thus invulnerable), making it impossible to perform any actions.
• In Space Station Silicon Valley (Nintendo 64), the game becomes impossible to win with a hundred percent completion because an action required to pick up a critical item is blocked.
• In Sphynx and the Cursed Mummy, if a player saves the game at a midpoint save point, a gate needed to progress is permanently closed, and the action to open the door is no longer possible.

The model aims to detect such issues related to in-game actions being blocked, unresponsive, or otherwise impossible, which impacts the gameplay experience.

Model Details

• Model Type: Classification (Binary)
• Training Data: The dataset for this model was created by scraping reviews from the Steam platform. These reviews were labeled into different categories, including Action Not Possible, to train the model. The model is based on concepts from the paper "What went wrong; the taxonomy of video game bugs".
• Task: Bug detection in video games related to unresponsive or blocked actions that should be permissible in certain game states.
• Intended Use: The model is designed for game developers and QA teams to detect issues where players are unable to take actions that are necessary to progress within the game.

Training Metrics

The model was trained for 4 epochs with the following performance metrics:

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score	Precision	Recall
0	0.6367	0.5908	70.60%	0.6975	0.7228	0.6739
1	0.3807	0.4763	79.71%	0.7889	0.8276	0.7536
2	0.3593	0.3761	84.45%	0.8505	0.8235	0.8792
3	0.3399	0.4699	83.60%	0.8428	0.8135	0.8744

Key Metrics:

• F1 Score: Balances precision and recall, with a final value of 0.8428 after 3 epochs.
• Precision: Measures the accuracy of positive predictions, reflecting how many of the predicted "Action Not Possible" issues were correct.
• Recall: Measures how well the model identifies actual action-related issues, capturing the true positives.

Intended Audience

• Game Developers: To detect in-game action issues where players cannot perform necessary actions to progress in the game.
• Quality Assurance (QA) Teams: To automate the detection of action-blocking bugs during game testing.
• Researchers: Interested in exploring and extending bug detection models for in-game action issues.

Limitations

• This model is specifically trained for detecting Action Not Possible issues related to blocked or unresponsive actions in video games.
• It does not account for other types of bugs or failures, such as graphical or performance-related issues.
• Performance may vary depending on the game's context and system configuration. Fine-tuning may be required for different games or environments.

How to Use

You can use the model for binary classification to predict whether a given in-game action is possible or not possible. Here's an example using the Hugging Face transformers library:

from transformers import pipeline

# Load the model from Hugging Face
action_detection = pipeline('text-classification', model='fyp-buglens/VideoGameReviews-ActionNotPossible-BERT')

# Example usage
result = action_detection("In Crysis, the final boss becomes untargetable and invulnerable")
print(result)  # Output: label indicating if the action issue is detected or not