Back to all updates

6 months ago

XR:AI Some Prototyping Tools

Hello participants!

The hackathon will start soon! Therefore, we wanted to share a  list of low-code and no-code tools you can use to rapidly prototype AI-driven XR experiences.

OpenAI Agent Builder (AgentKit)

• Link: https://openai.com/index/introducing-agentkit/
• Free / Cost: Pay-per-use (you only pay for OpenAI model token and tool usage; no separate license required).
• What it does: A visual, drag-and-drop environment for building intelligent agents powered by OpenAI’s new Agents SDK and Responses API. You can define workflows, tool calls, and reasoning logic without needing to write full backend code.
• Unity/Unreal integration:
• Use AgentKit to build the agent logic (e.g., how your XR character responds, plans, or reasons).
• Call the agent from Unity or Unreal via REST or WebSocket (HTTP POST request with user input).
• Parse JSON or text responses in your game to drive in-world actions, speech, or visuals.
• Why it’s great for the hackathon: Offloads complex AI reasoning to a cloud agent, so students can focus on XR interaction and UX.

 n8n

• Link: https://n8n.io/
• Free / Cost: Completely free if self-hosted; 14-day free trial for the cloud-version
• What it does: A powerful low-code workflow automation tool. Think of it like “Zapier + AI nodes.” Students can chain APIs, LLMs, and logic blocks visually.
• Unity/Unreal integration:
• Unity or Unreal sends webhooks to n8n (e.g., when a user interacts or speaks).
• n8n handles the logic — calls an LLM, fetches data, or performs computation.
• Returns a result to the XR app for display or action.
• Why it’s great: Perfect for teams with little backend experience; easy to connect XR input → AI response.

 Flowise

• Link: https://flowiseai.com/
• Free / Cost: Free and open-source (can self-host or use hosted version).
• What it does: A visual AI agent builder similar to LangChain, where you connect blocks to build conversational or reasoning agents.
• Unity/Unreal integration:
• Create an agent flow and expose it as a REST endpoint.
• Your XR app sends a message (text, JSON) to that endpoint.
• The Flowise agent replies — e.g., dialogue for an NPC or behavior decision for an AR assistant.
• Why it’s great: Fully open-source, hackathon-friendly, and easy to customize or run locally.

MindStudio

• Link: https://www.mindstudio.ai/
• Free / Cost: Free to start; paid tiers unlock more agent templates and analytics.
• What it does: No-code visual builder for creating AI agents with conditional logic, memory, and contextual prompts.
• Unity/Unreal integration:
• Create a MindStudio agent (e.g., “VR Mentor Bot”).
• Use MindStudio’s REST API or webhook output to integrate with your XR scene.
• Display the response as subtitles, 3D speech bubble, or voice audio.
• Why it’s great: Simple web interface — great for non-technical students to create logic-driven agents.

Voiceflow

• Link: https://www.voiceflow.com/
• Free / Cost: Free plan available; paid for collaboration and advanced analytics.
• What it does: A no-code platform for designing voice and chat experiences. Originally for Alexa/Google Assistant, now supports custom LLMs and APIs.
• Unity/Unreal integration:
• Unity/Unreal captures voice or text → sends to Voiceflow API.
• Voiceflow returns natural conversation text → XR displays it or plays TTS.
• Perfect for in-world NPC conversations or voice-activated assistants.
• Why it’s great: Easy conversational flow design, integrates seamlessly with XR via REST API.

How to Combine These Tools

You can mix and match tools depending on your hackathon project idea:

• Unity or Unreal Engine → handles 3D world, interaction, voice capture.
• Agent Builder / Flowise / MindStudio / Voiceflow → defines the agent’s brain (logic, conversation, decision-making).
• n8n → coordinates backend automation (API calls, data processing, LLMs).
• Use REST APIs or Webhooks to connect your XR app to the agent workflows.

Tips for Hackathon Teams

• Start from a simple Unity/Unreal template that can send/receive HTTP requests. Example integration with n8n: Unlocking Automation in Unity with n8n and Webhooks: A Developer’s Perspective | by Xavier Torrent Boix | Medium
• Use Flowise or n8n (also 14 days free cloud trial) for free if you want to host everything locally.
• If using AgentKit, set a shared OpenAI API key and track usage.
• Use Voiceflow for speech-driven prototypes — it’s very intuitive.
• Focus on creativity: combine XR gestures, spatial triggers, and AI dialogue for a unique experience.

In addition, there are some participants who are still looking for a team! So feel free to use the discord or devpost channels to connect !

Have a great start into the week!

The best,

Your XR:AI Hackathon Team