Okey Helper: Real-Time Computer Vision Game Assistant

1. Executive Summary & Project Goal

Okey is a popular traditional tile-based game played with 106 distinct wooden or plastic tiles, distributed across four color sets (Red, Blue, Black, Yellow) numbered 1 through 13, plus two unnumbered "false jokers". Players must organize their hands into valid sets (runs and groups) while tracking discarded tiles to deduce opponents' hands and calculate probabilities of finishing moves.

The challenge was to build an automated visual companion—Okey Helper—that could run directly within a standard web browser on a smartphone or laptop, access the user's camera feed, detect game tiles dynamically under varying lighting conditions, and compute real-time strategic recommendations without transmitting raw video frames to a remote cloud server.

2. Architectural Overview & Vision Pipeline

The image processing pipeline operates sequentially on incoming video frames retrieved through the HTML5 MediaDevices.getUserMedia() API. Below is the technical pipeline flow:

Camera Frame (RGB)
      ↓
Downsampling & Grayscale Conversion
      ↓
Bilateral Filtering (Noise Reduction while Preserving Edges)
      ↓
Adaptive Gaussian Thresholding + Morphological Operations
      ↓
Contour Extraction & Polygon Approximation (Tile ROI Detection)
      ↓
Perspective Transform (Warp to Normalized 128x192 Tile Matrix)
      ↓
Color Space Segmentation (HSV Masking for Red, Blue, Black, Yellow)
      ↓
Digit / Glyph OCR & Template Matching
      ↓
Game State Heuristic Engine (Set Formations & Win Probabilities)

3. Deep Dive into Computer Vision Techniques

A. Region of Interest (ROI) & Perspective Rectification

When a user points their camera at a game board, the tiles appear at perspective angles. To normalize these tiles for feature extraction:

1. Contour Detection: Using OpenCV contour retrieval (cv.findContours with RETR_EXTERNAL and CHAIN_APPROX_SIMPLE), candidate rectangular blobs are identified.
2. Bounding Box Filtering: Contours are filtered based on aspect ratio ($1.4 \le \text{height}/\text{width} \le 1.8$) and minimum bounding area to reject background clutter.
3. Four-Point Transform: For every qualified tile contour, a homography transformation matrix is computed to warp the quadrilateral into a standard rectangular planar patch of $128 \times 192$ pixels.

function warpTilePerspective(srcMat, corners) {
    const dstCorners = [
        0, 0,
        128, 0,
        128, 192,
        0, 192
    ];
    
    const srcCoords = cv.matFromArray(4, 1, cv.CV_32FC2, corners);
    const dstCoords = cv.matFromArray(4, 1, cv.CV_32FC2, dstCorners);
    
    const M = cv.getPerspectiveTransform(srcCoords, dstCoords);
    const warped = new cv.Mat();
    cv.warpPerspective(srcMat, warped, M, new cv.Size(128, 192), cv.INTER_LINEAR, cv.BORDER_CONSTANT, new cv.Scalar());
    
    srcCoords.delete();
    dstCoords.delete();
    M.delete();
    return warped;
}

B. Dual-Channel Color Space Analysis (HSV Masking)

Standard RGB color representation is vulnerable to brightness and shadow fluctuations. To robustly identify whether a tile belongs to the Red, Blue, Black, or Yellow set, the warped tile's glyph region is converted to the Hue-Saturation-Value (HSV) color space:

• Red Detection: Requires dual-interval hue checks ($\text{Hue} \in [0, 10] \cup [170, 180]$) with $\text{Saturation} \ge 120$.
• Blue Detection: Filtered via $\text{Hue} \in [100, 130]$, $\text{Saturation} \ge 90$.
• Yellow Detection: Filtered via $\text{Hue} \in [20, 35]$, $\text{Saturation} \ge 110$.
• Black Detection: Characterized by low $\text{Value}$ ($V \le 55$) regardless of hue angle.

4. Game State & Probability Calculation Engine

Once tile identities are resolved, they are fed into a deterministic state engine implemented in TypeScript/JavaScript. The engine evaluates:

• Group Formations: 3 or 4 tiles of the same number with distinct colors (e.g., Red-7, Blue-7, Black-7).
• Run Formations: Consecutive numbers of identical color (e.g., Blue-4, Blue-5, Blue-6). Note that in Okey rules, 13-1-2 is invalid, but 12-13-1 is legal.
• Missing Tile Odds: By subtracting all visible tiles (player hand + discards) from the known 106-tile deck, the exact remaining distribution is determined to compute probability scores for drawing required tiles.

5. Performance Benchmarks

Below are real-world performance benchmarks measured across various devices running the production WebAssembly/JS computer vision pipeline:

6. Conclusion & Key Takeaways

Okey Helper demonstrates that complex, real-time computer vision tasks do not necessarily require expensive backend GPU servers or high-bandwidth video streaming. By leveraging client-side WebAssembly, efficient algorithmic heuristics, and targeted color-space transformations, responsive web applications can deliver zero-latency experiences while safeguarding user privacy.