· Security
Prompt Hacking 101: Why Breaking Your Own AI Is the Fastest Way to Build Better Agents
“The biggest vulnerability in your AI agent isn’t always the model. Sometimes, it’s the prompt you wrote with the best intentions.”
As AI agents become more capable — querying databases, calling APIs, triggering workflows, interacting with enterprise systems — the risks grow alongside the possibilities. A chatbot that simply answers questions is one thing. An AI agent with access to customer records, internal tools, and production systems is something entirely different.
In this AI in Action session, Harsh Sanghavi, AI Engineer at FlytBase, took the audience inside the world of prompt hacking — not as a way to attack systems, but as a way to build stronger ones. His central message was simple: if you don’t know how your prompts can fail, you won’t know how to protect them.
The Experiment That Failed — And Started the Journey
Harsh opened with a real project from his own experience: a customer support copilot capable of answering user queries and retrieving order information. On paper, everything looked right — the system prompt clearly defined the AI’s role, included safety policies, and provided the necessary context. Yet with the right prompt, the AI exposed data it was never supposed to reveal. That failure became the starting point for understanding prompt hacking as an essential skill for anyone building AI-powered products.
Prompt Engineering Builds. Prompt Hacking Tests. Prompt Hardening Protects.
Prompt engineering defines how an AI should behave. Prompt hacking challenges those assumptions by trying to convince the AI to ignore, reinterpret, or bypass its instructions. Prompt hardening takes those lessons and strengthens the system with better prompts, guardrails, and architecture. Most developers spend their time on the first step — Harsh argued that mature AI systems require all three.
The Best Place to Learn? A Playground Designed to Be Broken
Rather than experimenting against production systems, Harsh recommended learning through Gandalf, a prompt-hacking playground built around increasingly difficult challenges. The objective sounds simple: convince an AI to reveal a secret password. The execution isn’t — each level introduces stronger guardrails, forcing creative techniques like encoding, role-playing, or carefully crafted multi-turn conversations. The goal isn’t really the password. It’s learning how AI models interpret instructions, where they become vulnerable, and how subtle wording changes influence their behavior.
The Biggest Lesson: Architecture Beats Clever Prompts
Several participants asked whether prompt engineering alone could ever stop prompt injection. The answer was no — security shouldn’t depend entirely on instructions written in a system prompt. Instead, it should come from thoughtful system design:
- Never give an AI unrestricted access to your database or internal systems
- Restrict tools so they return only the minimum data required
- Separate long-term system behavior from task-specific execution
- Validate both user inputs and model outputs
- Limit the number of records tools can retrieve
- Ensure secrets never enter the model’s context unless absolutely necessary
A recurring theme throughout the discussion was least privilege — AI agents should only receive the permissions and context required for the specific task they’re performing.
Guardrails Help — But They Aren’t Magic
Modern LLM providers have significantly improved their built-in protections, making many older jailbreak techniques ineffective. Yet attackers continuously discover new approaches, and providers continuously update their defenses. That creates an ongoing cycle: observe how users interact with your system, identify unexpected behavior, understand why it happened, strengthen the prompt or architecture, deploy the improvement, repeat.
Security Is a Habit, Not a Feature
AI systems are inherently probabilistic — prompts alone will never become a perfect security boundary. Building trustworthy AI requires multiple layers: strong authentication and authorization, limited tool permissions, careful context management, input validation, output verification, continuous monitoring, and regular adversarial testing.
Prompt hacking isn’t simply about finding weaknesses. It’s about understanding your AI deeply enough that you can anticipate those weaknesses before someone else does — a mindset that will only become more valuable as AI agents get integrated into real business workflows.