Team Ai
Apppublic

Agents-MCP-Hackathon/computer-use-agent-mcp

sourceHugging Faceupdated 1y agoView on Hugging Face
0likes
App README

Computer Use Agent MCP

This project provides a MCP server that allows AI agents to interact with the Windows desktop environment just like a human user would. It provides a suite of tools for:

  • —Launching applications
  • —Executing PowerShell commands
  • —Monitoring desktop state
  • —Mouse control (clicking, dragging, scrolling)
  • —Keyboard input
  • —Clipboard operations
  • —UI element interaction

Demo with claude

🎥 [Watch it here] (https://drive.google.com/file/d/1XIBzrLeIivDn36MG4Ve4CXRCKHb_Gg66/view?usp=sharing)

<video width="100%" controls> <source src="demo.mp4"> </video>

Features

  • —Desktop State Monitoring: Get real-time information about:
  • —Active applications
  • —Interactive UI elements
  • —Informative text elements
  • —Window states and positions
  • —Input Control:
  • —Mouse movement and clicking
  • —Keyboard input and shortcuts
  • —Drag and drop operations
  • —Scrolling
  • —System Integration:
  • —Application launching
  • —PowerShell command execution
  • —Clipboard management
  • —Window manipulation

Technical Details

  • —Built with Python 3.13+
  • —Uses Gradio for the web interface
  • —Key dependencies:
  • —gradio: Web interface and MCP server
  • —pyautogui: Mouse and keyboard control
  • —uiautomation: Windows UI automation
  • —pyperclip: Clipboard operations

Installation

  1. 1.Make sure you have Python 3.13 or higher installed
  2. 2.Install the required dependencies:
bash
pip install -r requirements.txt

Usage

Run the server:

bash
python app.py

The server will start with both a web interface for manual testing and an MCP endpoint for agent integration.

MCP Tools

The following tools are available via the MCP interface:

  • —launch_tool: Launch Windows applications
  • —powershell_tool: Execute PowerShell commands
  • —state_tool: Get desktop state information
  • —click_tool: Perform mouse clicks
  • —type_tool: Input text
  • —scroll_tool: Scroll windows
  • —drag_tool: Perform drag operations
  • —move_tool: Move mouse cursor
  • —shortcut_tool: Execute keyboard shortcuts
  • —key_tool: Press individual keys
  • —wait_tool: Add delays
  • —clipboard_tool: Manage clipboard content

Acknowledgments

Created for the Hugging Face Agents-MCP-Hackathon 2025.