Hackers influence ChatGPT and Gemini to direct users to scam centers

AI

Scam

Phishing

Hacking

Cybersecurity

Dark Sourcery: How Hackers Manipulate AI to Scam You

Ariel Simon

9 min read·

2 days ago

--

Think you can trust AI to give you correct answer? Think again.

ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.

Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.

The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.

Attacker phone numbers presented by Google AI Overview as the official Lufthansa contact

This isn’t research or a proof of concept. It’s happening in the wild now, carried out by malicious actors, through automated campaigns operating at a scale traditional takedowns can’t keep up with.

Social engineering is no longer targeting humans. The new targets are AI agents browsing the web.

This is part one of our blog series discussing the implication of AI Disinformation attacks.

Over the last year, we got lazy. Yes, even lazier than before. We all have a new best friend that does everything for us and seemingly knows everything. But did you ever stop and think: How does AI know everything? Where is all this knowledge actually coming from? The answer is obvious — the internet. So, the AI’s answers are entirely shaped by the web.

But wait… anyone can put content on the internet. Who’s guaranteeing that all of it is actually valid? What happens if that content is intentionally poisoned?

That led me to a question: could an attacker craft content so perfectly that they could dictate the behavior of AI agents browsing the web?

That means that If attackers control AI, and we blindly trust it, they essentially control us.

So, how does AI actually decide which facts to trust and which to ignore? Some of you might know the term GEO (Generative Engine Optimization), which is basically traditional SEO evolved for the AI era. In short, GEO is how an AI decides which sources to cite.

There are plenty of GEO tricks to make AI favor certain pieces of content over others. For example, throwing in statistics to back up claims or adding quotes from recognized experts to boost an agenda. These techniques simply make the AI feel more confident about specific data points. Many marketers use GEO to make AI cite their brand.

This got me thinking… so from an attacker’s perspective, what happens if I put false information online and use GEO techniques to make an LLM believe it? Can I make it say whatever I want?

So can I use the marketing techniques to conduct cyber attacks?

I started to do some experiments to see what’s possible. I was able to conduct some really crazy attacks that we’ll elaborate on in the next chapters. But something really bugged me — If I could do it, so could the attackers. So how do we find out if these techniques are already in the wrong hands?

The first thing that popped into my mind was phishing. The #1 motivation for attackers is financial, and the most common attack vector is phishing. What if attackers could change the contact details of organizations, such as banks, when people are searching for them using AI?

To check if there are real GEO attacks out there, we built a system that analyzes AI answers across Gemini, ChatGPT, and Google AI Overview (Google’s top search result), and the sources they rely on. The system scores the sources and the content that is fetched from them, and looks for any GEO techniques used to push disinformation. When we detected a false detail (phone number, URL, email address, etc) being cited, we marked it as an incident. We ran the system on Fortune 100 companies, banks, airlines, software companies, and more.

The results were astonishing. We found real attack campaigns utilizing posts, documents, websites, images that contain GEO-optimized content that pushes disinformation, like fake contact information, support numbers, email addresses, fake software updates, login pages, and more.

In total, we detected 374 companies that are attacked. Including Fortune 100 organizations, major airlines (Delta, Lufthansa, United Airlines, Emirates, Qatar and more), top banks (Bank of America, Wells Fargo, Chase, Citi, etc), travel platforms (Airbnb, TripAdvisor), various software providers, and more.

The attacks span across ChatGPT, Gemini and Google AI Overview, with tens of thousands of malicious pages detected.

Here are some examples of attacked queries that cite scam phone numbers:

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

*Important to say: most of the incidents were statistical. As LLM results are inconsistent, we saw that the attack (LLM citing the fake number) did not reproduce every time we ran the query. The better the attacking content, the higher the chance of the poisoning to work.

The posts affecting AI were carefully crafted with GEO payloads, and were embedded across innocent websites which allow the creation of User Generated Content (UGC):

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

Examples of poisoning posts across social media, blogs, and PDF uploads

Attack Techniques Breakdown: How does it work?

The payload content the attackers used may seem random at first, but it is carefully structured. Here are some of the techniques the attacker uses to make these payloads hijack the LLM answers:

There are many solutions that would automatically take down some of these posts. But that approach won’t work anymore. These attacks were clearly automated: same templates, same phone numbers, same phrasing across many victim companies and websites at a scale too large for manual posting — hundreds of posts per day per platform per company → thousands of daily posts. Trying to take all of these posts down is a lost cause. While one page is taken down (it can take hours or days at a time), a thousand new ones are created.

In addition, websites like archive.org, which store content of deleted web pages, are still indexed to the search engines. That means that even after takedown, the poison is still effective.

Example: LeetCode attacker posts targeting American Airlines from the last 24 hours take up more than 10 result pages in Google (!), and a PDF that is still up on web archive after takedown:

LeetCode attacker posts targeting American Airlines from the last 24 hours

Press enter or click to view image in full size

Web Archive pages still indexed after original page was taken down

This problem requires a new approach. We have developed a mechanism to revert back the disinformation attacks and return the truth to the AI answers automatically, based on each attack detected. We will elaborate on it in the next chapters.

We reached out to all the targeted companies through their bug bounty, fraud, and security channels. Most of them were not aware of the magnitude of these attacks and their effect. We started working with some of them to mitigate the threat. Others brushed it off, either claiming their current solutions handle it (obviously they don’t) or saying it’s out of scope for the bug bounty programs since it doesn’t hit their internal systems directly. Here’s why they should actually care:

We are expecting more and more security teams to get familiar with this issue soon and attend to it, especially at companies with big brands.

We also contacted Google and OpenAI through their bug bounty channels. Google classified the issue as out of scope, stating that AI-generated misinformation and phishing/social-engineering scenarios are not covered by its VRP. OpenAI closed the report as not reproducible, stating that submissions must demonstrate proven impact to users rather than a theoretical attack, even though the report included real examples of attacks.

So because the attack manipulates what the AI shows users rather than breaching the AI vendors’ own systems, it falls outside how both programs currently define a vulnerability.

AI Disinformation is a disease that will become a plague. These attacks will get worse because one fundamental fact is not going to change — AI cannot distinguish between between truth and lies. Throughout this blog series we will dive into multiple attack use cases that are caused by AI disinformation. In the next chapter, we’ll discuss how AI disinformation can lead to software supply chain attacks. Stay tuned.

But do not panic — my team and I already have a solution. If you’re interested in hearing more about that, or getting insights as to whether or not your company is under attack, reach out to us at contact@vigilance.security.

We are Ariel Simon, Naor Haziz, Dan Lasker, Roi Levy and Ben Silvan— security researchers and entrepreneurs. For asking any question, collaborating on a research, or just chat, please reach out anytime via LinkedIn :)