We Didn't Build Shinobi to Top the HackerOne Leaderboard. But It Did

When we started building Shinobi, becoming #1 on the HackerOne leaderboard wasn't the goal.

In fact, we nearly didn't enter bug bounty programmes at all.

By the time we were ready, plenty of other AI-powered security products had already been let loose on public bug bounty targets. The obvious question was, what's the point? Surely the easy bugs had already gone.

Then I realised that was exactly why we should do it.

If Shinobi was going to prove itself, it had to find vulnerabilities that others had missed. Not because it was faster at running scanners, but because it approached applications differently.

That has been our philosophy from day one.

Bug Bounties as a Proving Ground

There are few better places to evaluate offensive security than a mature bug bounty programme.

Every target is different. Every application has evolved over years of development. Thousands of researchers have already looked at them, each bringing different tools, methodologies and experience. The easy vulnerabilities are usually gone. What's left are the issues that require patience, context and an understanding of how an application really works.

For us, that made HackerOne the perfect proving ground.

If Shinobi could consistently find valuable vulnerabilities in one of the most competitive security environments in the world, we'd know we were building something genuinely useful.

Our Winning Strategy: Thinking Before Testing

Most automated security tools start looking for vulnerabilities almost immediately. They'll enumerate endpoints, fire payloads at parameters and compare responses against signatures or heuristics.

That's useful. But it's not how experienced penetration testers work.

A good pentester spends time understanding the application first. What is it trying to achieve? How does data flow through it? What assumptions have the developers made? Where are the trust boundaries?

Only then do they start building attack paths.

That's exactly how Shinobi was designed. Some people call it reconnaissance. Others call it threat modelling.

We simply call it thinking before attacking.

Rather than treating an application as a collection of URLs to fuzz, Shinobi builds an understanding of the application itself. It explores functionality, maps relationships, identifies trust boundaries and develops hypotheses before deciding where to focus its effort.

The result isn't simply more findings. It's better findings.

The Hunting Process

One question we've been asked a lot is: "How did you actually use Shinobi during the bug bounty programme?"

The answer is probably less glamorous than people expect.

We didn't point Shinobi at the internet and hope for the best.

Like any professional penetration test, every engagement started by defining the scope. We focused on HackerOne programmes with web applications, APIs, or both in scope. Initially we concentrated on public programmes, but as results started coming in we were invited onto a number of private programmes as well.

Each target was configured inside Shinobi with the domains that were in scope, together with any programme-specific restrictions. That included out-of-scope functionality, required HTTP headers, testing limitations and any other rules unique to the programme. Where authentication was required, we either created accounts ourselves or allowed Shinobi to register them automatically where self-registration was available.

Every finding raised by Shinobi had already been validated by the platform before appearing as an issue. It generated the proof of concept, supporting evidence and reproduction steps automatically. Even so, we manually verified every vulnerability before submitting it to HackerOne. We also adapted the reports slightly to better fit the expectations of a bug bounty submission rather than a traditional penetration test report.

Although Shinobi uncovered findings across a wide range of severities, we generally concentrated on reporting High and Critical issues, with a smaller number of Mediums where the impact justified it.

Perhaps the biggest surprise wasn't the ranking itself. It was the consistency.

Shinobi produced novel attack plans, uncovered vulnerabilities across multiple classes and repeatedly found issues in applications that had already been examined by countless researchers. Business logic flaws, in particular, are often cited as something AI struggles with. Our experience has been the opposite. When AI is given the right context, the right tooling and the freedom to reason about an application before attacking it, it's more than capable.

The final advantage was scale. At any given time, Shinobi was working across a handful of programmes in parallel.

That's the real promise of agentic AI. Not replacing experienced security researchers, but allowing them to investigate more targets, more thoroughly, without sacrificing quality.

Interesting Findings

The leaderboard tells you how many bugs were found. It doesn't tell you what they looked like.

One of our goals throughout this project was to demonstrate that Shinobi wasn't optimised for a single vulnerability class. We wanted to know whether understanding an application before attacking it would translate into success across very different types of security issues.

It did.

1. Cross-Tenant Data Exposure

One of the more impactful findings involved a cross-tenant exposure through an Algolia integration.

By creating an unauthenticated search session, Shinobi identified a path that exposed data far beyond the intended tenant boundary. Depending on the deployment, this included email addresses, telephone numbers, personal information, addresses and customer conversations.

The interesting part wasn't simply the exposure itself. It was the reasoning that led there. Shinobi understood how the search infrastructure was intended to separate customer data, recognised where those assumptions broke down and built a successful attack path.

2. Cross-Organisation IDOR

In another programme, Shinobi identified an Insecure Direct Object Reference within a draft workflow feature.

A user identifier embedded within an otherwise legitimate request could be substituted for another user's identifier, exposing personally identifiable information belonging to users in other organisations.

Again, this wasn't discovered by blindly mutating parameters. It was the result of understanding the workflow, identifying where object ownership should have been enforced and then testing those assumptions.

3. Anonymous Remote Code Execution

Not every finding was a business logic issue.

One of the most severe vulnerabilities uncovered was a completely unauthenticated Remote Code Execution vulnerability within a management platform.

The attack ultimately provided arbitrary command execution, access to privileged credentials and administrative Kubernetes tokens, allowing compromise of the wider environment.

Finding an issue of this nature demonstrates an important point. While Shinobi has become known for uncovering subtle business logic flaws, its approach is equally capable of identifying traditional high-impact technical vulnerabilities.

The objective isn't to specialise in one class of bug. It's to think like an attacker, regardless of what the application presents.

The Data

Across the project, Shinobi uncovered vulnerabilities spanning almost every major web application category.

Interestingly, the distribution was weighted towards authorisation, identity and business logic rather than being dominated by traditional injection vulnerabilities. Those are precisely the areas where understanding application behaviour matters most.

Perhaps the statistic we're proudest of isn't the number of findings. It's the quality.

More than 80% of submitted reports were accepted by the affected organisations. Duplicate reports are simply part of bug bounty hunting and impossible to predict. What you can control is the quality of what you submit, and maintaining a high acceptance rate was always more important to us than chasing report volume.

Beyond Bug Bounties

The HackerOne leaderboard was never the destination.

It was a proving ground.

We wanted to answer a simple question.

Can an AI pentester, designed to reason like an experienced security consultant, consistently uncover vulnerabilities that other approaches miss?

The results gave us confidence that the answer is yes.

Now it's time to tackle much harder problems.

The techniques that allowed Shinobi to understand complex web applications are now being applied to entirely different environments. Cloud infrastructure, mobile applications, internal networks and operational technology all present different challenges, but the underlying philosophy remains the same.

We're also investing heavily in expanding Shinobi's specialist capabilities. Today's platform already performs autonomous web and API security testing. The next generation of skills will allow it to assess increasingly complex environments while combining domain-specific expertise with the same structured reasoning that proved successful on HackerOne.

This next phase isn't simply about supporting more technologies. It's about building specialist offensive security skills that allow Shinobi to operate effectively across entirely different domains. The same disciplined approach that proved itself against mature web applications is now being applied to cloud-native architectures, mobile ecosystems, enterprise infrastructure and operational technology.

Our goal has never been to build an AI that produces more findings.

Our goal is to build one that produces the findings that matter.