fyp-buglens/VideoGameReviews-InvalidEventOccuranceOvertime-BERT

Invalid Event Occurrence Over Time Detection Model

This model is trained to detect Invalid Event Occurrence Over Time issues in video games. These issues occur when discrete in-game events happen too often, too seldom, or at incorrect times. This can include incidents like a Hell Event in Lords Mobile starting at the wrong time or terminating before its expected duration of 55 minutes.

For example, if the Hell Event in Lords Mobile, which is supposed to last 55 minutes, begins at the wrong time or ends prematurely, it would fall into this category of Invalid Event Occurrence Over Time. The model aims to detect such issues related to the timing and occurrence of in-game events, which could impact gameplay experience.

Model Details

• Model Type: Classification (Binary)
• Training Data: The dataset for this model was created from scratch by scraping reviews from the Steam platform. These reviews were then labeled into different categories, including Invalid Event Occurrence Over Time, to train the model. The model is based on concepts from the paper "What went wrong; the taxonomy of video game bugs".
• Task: Bug detection in video games related to the timing and occurrence of in-game events, such as events starting or ending at incorrect times or with incorrect durations.
• Intended Use: The model is designed for game developers and QA teams to detect event-related issues in video games caused by incorrect event timings or durations.

Training Metrics

The model was trained for 4 epochs with the following performance metrics:

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score	Precision	Recall
0	0.6545	0.6502	60.27%	0.6856	0.5657	0.8699
1	0.6073	0.5941	69.65%	0.6926	0.6986	0.6866
2	0.5311	0.5671	70.84%	0.6816	0.7469	0.6267
3	0.4689	0.5266	74.25%	0.7428	0.7390	0.7466

Key Metrics:

• F1 Score: Balances precision and recall, with a final value of 0.7428 after 3 epochs.
• Precision: Measures the accuracy of positive predictions, reflecting how many of the predicted event-related issues were correct.
• Recall: Measures how well the model identifies actual event-related issues, capturing the true positives.

Intended Audience

• Game Developers: To detect event-related issues in video games, such as in-game events starting or ending at incorrect times.
• Quality Assurance (QA) Teams: To automate the detection of event timing issues during game testing.
• Researchers: Interested in exploring and extending bug detection models for in-game event issues.

Limitations

• This model is trained specifically for detecting Invalid Event Occurrence Over Time issues, focusing on event timing and occurrence problems.
• It does not account for issues caused by other types of bugs or failures, such as graphical or performance-related issues.
• Performance may vary depending on the game's context and system configuration. Fine-tuning may be required for different games or environments.

How to Use

You can use the model for binary classification to predict whether a given in-game event exhibits an Invalid Event Occurrence Over Time issue. Here's an example using the Hugging Face transformers library:

from transformers import pipeline

# Load the model from Hugging Face
event_detection = pipeline('text-classification', model='fyp-buglens/VideoGameReviews-InvalidEventOccurrenceOvertime-BERT')

# Example usage
result = event_detection("The Hell Event in Lords Mobile started at the wrong time")
print(result)  # Output: label indicating if it's an event timing issue or not