The AI That
Sees Your Screen
A multi-platform teaching assistant that overlays GPT-5 powered insights directly onto any desktop application.
Identify the Function
This is a polynomial function. We apply the power rule: ∫x^n dx = (x^(n+1))/(n+1).
The "Hard Tech" Problem
We needed a way to demonstrate complex AI capabilities beyond simple chatbots. We wanted to build a tool that could "see" and "reason" about educational content in real-time.
Multi-Platform Integration
Connecting a native Windows desktop app (WPF) with a modern React web dashboard and a serverless backend.
Complex Orchestration
Managing asynchronous flows: Screenshot capture → Cloud Upload → AI Reasoning → JSON Parsing → UI Rendering.
Spatial Intelligence
Prompting an LLM to understand X/Y coordinates and screen layout to place annotations without obscuring content.
# SYSTEM PROMPT (v5.0)
// Spatial Context Definition
YOU ARE A VISUAL TEACHING ASSISTANT.
INPUT: Image + 25-Point Grid Overlay
OUTPUT: JSON Array of Annotations
RULES:
1. Analyze background luminance at coords.
2. Select text_color from ["#fff", "#000"].
3. AVOID_OVERLAP = TRUE
4. Break complex math into <3 steps.
5,000 Tokens of Reasoning
The magic isn't just the AI; it's how we talk to it. We developed a sophisticated 25-point spatial reference system that allows GPT-5 to "see" the screen layout.
The system analyzes background contrast to determine font colors, calculates optimal tooltip placement, and breaks down complex visual data into pedagogical steps.
Orchestrating Intelligence
Three complex n8n workflows handle the heavy lifting. They manage session state in Supabase, process screenshot uploads, and bridge the gap between the Windows desktop C# code and the OpenAI API.
From Concept to Capabilities
This internal project served as the ultimate proving ground. We now have a battle-tested architecture for:
Need Complex AI Automation?
We don't just build chatbots. We build systems that see, reason, and act across your entire software stack.