fyp-buglens/VideoGameReviews-InvalidContextStateOvertime-BERT

Invalid Context State Over Time Detection Model

This model is trained to detect Invalid Context State Over Time issues in video games. These issues occur when the observable state of a game element happens too early, too late, or persists for a duration longer or shorter than expected. This category includes faults where the visual state or behavior of a game element doesn't align with the intended timeline, potentially affecting the gameplay experience.

For example, in Super Mario, when Mario catches a star, its image starts flickering, and Mario enters an invulnerable state for 20 seconds. If the invincible state lasts less or more than 20 seconds, it would fall into this category of Invalid Context State Over Time.

Model Details

• Model Type: Classification (Binary)
• Training Data: The dataset for this model was created from scratch by scraping reviews from the Steam platform. These reviews were labeled into different categories, including Invalid Context State Over Time, to train the model. The model is based on concepts from the paper"What went wrong; the taxonomy of video game bugs".
• Task: Bug detection in video games related to the timing and duration of in-game element states, such as incorrect invincible state durations or faulty event timing.
• Intended Use: The model is designed for game developers and QA teams to detect state-related issues in video games caused by incorrect durations or timings of game element states.

Training Metrics

The model was trained for 4 epochs with the following performance metrics:

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score	Precision	Recall
0	0.6047	0.5894	70.56%	0.6961	0.7353	0.6608
1	0.4839	0.4822	78.20%	0.7770	0.8125	0.7445
2	0.4571	0.3812	83.82%	0.8349	0.8708	0.8018
3	0.3039	0.3866	83.82%	0.8302	0.8934	0.7753

Key Metrics:

• F1 Score: Balances precision and recall, with a final value of 0.8302 after 3 epochs.
• Precision: Measures the accuracy of positive predictions, reflecting how many of the predicted context state issues were correct.
• Recall: Measures how well the model identifies actual context state issues, capturing the true positives.

Intended Audience

• Game Developers: To detect state-related issues in video games, such as incorrect durations or timings of in-game element states.
• Quality Assurance (QA) Teams: To automate the detection of context state issues during game testing.
• Researchers: Interested in exploring and extending bug detection models for state-related issues in video games.

Limitations

• This model is trained specifically for detecting Invalid Context State Over Time issues, focusing on timing and duration problems with in-game element states.
• It does not account for issues caused by other types of bugs or failures, such as performance-related or graphical issues.
• Performance may vary depending on the game's context and system configuration. Fine-tuning may be required for different games or environments.

How to Use

You can use the model for binary classification to predict whether a given game state exhibits an Invalid Context State Over Time issue. Here's an example using the Hugging Face transformers library:

from transformers import pipeline

# Load the model from Hugging Face
context_state_detection = pipeline('text-classification', model='fyp-buglens/VideoGameReviews-InvalidContextStateOverTime-BERT')

# Example usage
result = context_state_detection("Mario's invincibility state lasted less than 20 seconds")
print(result)  # Output: label indicating if it's a context state issue or not