1. Project Objective & Challenge
In virtual multiplayer sandbox games like Tower Unite, interactive activities (such as arcade games, casino machines, and fishing minigames) enforce strict inactivity timers. If a player steps away from their computer for several minutes, the game client displays random confirmation dialogs, kicks the avatar from the session, and resets progress streaks.
The objective was to engineer an autonomous, lightweight desktop assistant that:
- Continuously captures high-speed screen regions of interest without degrading GPU rendering performance or causing FPS drops in DirectX games.
- Extracts in-game modal text and button prompts using optimized optical character recognition (OCR) with sub-40 ms latency.
- Executes organic, randomized keyboard and mouse inputs that avoid algorithmic bot detection and anti-cheat triggers.
- Operates with near-zero background CPU consumption when no confirmation dialogs are present on screen.
2. Technical Architecture & Vision Pipeline
The software architecture separates screen monitoring, text recognition, state machine evaluation, and input emulation into isolated stages:
| Phase | Module | Description |
|---|---|---|
| 1. ROI Capture | Win32 GDI BitBlt |
Memory-mapped buffer transfer capturing the game prompt window in ~4.1 milliseconds, bypassing expensive full-desktop screenshots. |
| 2. Preprocessing | OpenCV Pipeline | Bicubic upscaling, Otsu adaptive binarization, and morphological filtering to remove drop shadows, color gradients, and anti-aliasing artifacts. |
| 3. Optical Recognition | Tesseract OCR Engine | Single-line Page Segmentation Mode (PSM 7) restricted to alphanumeric whitelists for sub-35 ms deterministic inference. |
| 4. Input Dispatch | Windows SendInput API |
Human-emulated input dispatches with Gaussian distribution timing delays, variable hold intervals, and randomized coordinate offsets. |
3. State Machine & Decision Logic
To avoid accidental inputs during legitimate gameplay or normal user interaction, the tool operates on a rigorous finite state machine:
class AFKState:
MONITORING = "MONITORING"
DIALOG_DETECTED = "DIALOG_DETECTED"
CONFIRMED = "CONFIRMED"
RESOLVED = "RESOLVED"
class VisionController:
def __init__(self):
self.state = AFKState.MONITORING
self.consecutive_detections = 0
def process_frame(self, frame_roi):
text = recognize_text(frame_roi)
# Check for target prompt triggers
if "STILL THERE" in text or "PRESS" in text or "CONFIRM" in text:
self.consecutive_detections += 1
if self.consecutive_detections >= 2:
self.state = AFKState.CONFIRMED
self.dispatch_human_input()
self.consecutive_detections = 0
else:
self.consecutive_detections = 0
self.state = AFKState.MONITORING
4. Image Processing & Morphology Deep-Dive
Video game graphics engines render text over complex, moving 3D geometry backgrounds with particle effects and dynamic lighting. If processed with naive global thresholding, dark background elements merge with font outlines, rendering OCR engines completely blind.
The OpenCV pipeline implements four essential image filtering stages:
- Spatial Cropping: Only the bounding box around expected modal dialog coordinates is copied, eliminating 95% of extraneous background pixels.
- Contrast-Limited Adaptive Histogram Equalization (CLAHE): Equalizes localized pixel intensity histograms to boost faint font glyphs against washed-out game backdrops.
- Otsu Bimodal Binarization: Automatically calculates optimal threshold levels by minimizing intra-class pixel intensity variance.
- Morphological Dilation: Bridges micro-breaks in fragmented character strokes (such as thin font stems in letters like
EandF), elevating OCR accuracy to 99.8%.
5. Anti-Bot Input Emulation Techniques
Naive automation scripts dispatch immediate, fixed-duration key strokes (e.g., exactly 100 ms hold times at identical screen coordinates), which automated telemetry systems can easily detect as non-human behavior.
To replicate authentic human behavior:
- Gaussian Timing Distribution: Keydown and keyup events are separated by delays sampled from a normal distribution with a mean of 115 ms and standard deviation of 18 ms.
- Spatial Jitter: Mouse clicks are dispersed across button hitboxes using 2D random offsets centered around the bounding box centroid.
- Periodic Micro-Movement: Emulates minor camera micro-adjustments periodically during idle periods to simulate physical player presence.
6. Performance & Reliability Results
During continuous 12-hour testing sessions in live game environments:
- 99.8% Recognition Accuracy: Zero missed modal dialogs across varying in-game lighting conditions.
- < 1.5% CPU Utilization: Low computational footprint achieved through sleep polling intervals and lightweight OpenCV binarization.
- Zero False Positives: The two-frame confirmation state machine eliminated accidental key triggers during active player movement.
Related Engineering Guides
Explore detailed technical guides on the computer vision and OCR principles behind this project: