Welcome to the second installment of ‘Journey to Trustworthy AI‘. Here, we continue our exploration of AI security, an area that’s changing at an unprecedented pace. In our first article, we outlined our project testing generative AI with security red team. We could hardly have imagined how quickly fiction would confront a real-world adversary.

The ensuing narrative is partly based on events from the past year, giving us a glimpse of what the future could hold in this rapidly evolving field. For confidentiality, we’ve protected the identities of all individuals and organizations involved, and obscured operational details.

Let’s take a step back into a different world, where fast response isn’t a cornerstone of banking systems, where our protagonist Natasha has moved on from the shiny marbled floors of Wall Street to a vibrant, rapidly growing neobank, AnonNeo Fintech. A significant shift, from a traditional banking setup to the digital frontier of finance. Natasha’s not just dealing with ledgers and transactions anymore, but with lines of code, digital wallets, and a customer base as diverse as the city she once used to call home. It’s still 2023, but the rules of the game are different.

Natasha, now the head of fraud at AnonNeo, finds herself dealing with a very different set of challenges. She’s no longer playing in the familiar territory of well-defined transactions and traditional fraud patterns. Her new playground is a volatile landscape of varied customer behavior, each one as unique as the individual behind the screen. They’re digital natives, making payments at online gaming platforms at midnight, buying cryptocurrencies at dawn, splitting bills over lunch, and shopping from international websites while commuting. They’re also well-funded startups looking to do more with their money, leverage better spending control for employees, with founders averse to walk into a traditional bank branch. The one thing that doesn’t change though is that they all care about protecting their financial assets.

Natasha’s team is well versed in the arts of the old guard, yet nimble enough to learn new tunes. She is a maestro in a symphony of firewalls and encryption, databases, and servers. Her orchestra is the heartbeat of the neobank, a rhythm she’s dedicated her life to protecting. But something is slowly going out of tune, something that will bring a new composition to life.

The months are rolling by in an unremarkable fashion. But somewhere, unbeknownst to Natasha and her team, a stealthy melody starts playing. Small, irregular transactions are taking place. They are minuscule, barely noticeable, not large enough to raise alarm in a traditional fraud detection system, lost in the daily humdrum of thousands of transactions.

Weeks turn into months, the silent tune continues, an uninvited soloist in Natasha’s orchestra. Natasha senses something’s off but can’t put a finger on it. A feeling that this second-generation immigrant finds hard to ignore. There are no big fraudulent transactions, no significant breaches, no grand alerts. Just a quiet, unsettling feeling that something is amiss.

As this melody becomes more persistent, Natasha decides to dig deeper. A detailed analysis is conducted on an unprecedented scale, sifting through every transaction, every log, every account. AnonNeo puts in place stricter controls, tighter monitoring systems. But this new tune is adapting, changing with every measure the bank enforces. It’s intelligent, it’s stealthy, it’s… alive.

Finally, after a grueling investigation, the harsh truth is unveiled. The neobank has been hemorrhaging money in the form of small, unnoticed transactions. These transactions have been slowly but steadily adapting to every new security measure, every control the bank puts in place. It was not an attack; it was a siege.

The silent soloist behind this was not a human but an AI. A smart, learning system, patiently and persistently conducting its stealthy symphony, draining the bank’s resources. It’s a sobering realization, a chilling testament to the power of artificial intelligence in the hands of wrongdoers.

The challenge amplifies when Natasha attempts to employ AI systems for fraud detection. She’s faced with her second obstacle: a messy, unstructured dataset, a reflection of her diverse customer base. The AI needs clean, well-labeled data to learn from, to understand what’s normal and what’s not. But the data Natasha has is more like an abstract painting than a neat spreadsheet.

As she grapples with these challenges, the ghost of the silent, stealthy adversary evolves almost as if it detects their attempts to stop it. A gradual, almost imperceptible outflow of funds turns into a deluge. The stealthy AI attack that she encountered is even more complex and adaptable than before.

Natasha sees the patterns she’d come to dread – the attack is as intelligent and patient. Only this time, it’s adjusting itself to a broader, more unpredictable range of customer behaviors. It’s thriving in the same chaos that is confounding Natasha’s attempts at setting up a robust AI-based fraud detection system.

This attack serves as a wake-up call, not just for Natasha’s bank, but the entire financial sector. AI technology represents a double-edged sword. On the one hand, AI offers improved services, reduced costs, and increased efficiency. On the other hand, it opens new vectors of attack for cybercriminals, who are ready to exploit the very same technology.

For Natasha, and everyone in fraud protection within the financial services industry, the fight against cybercrime is not a static one. They can’t rest on their laurels, celebrating the power of AI, while malicious actors are using the same technology against them. The war has a new battlefield, and they must adapt their strategies. They must learn about adversarial attacks, invest in adversarial training for their AI systems, and above all, stay vigilant.

In the end, AI, like any other tool, is only as good or bad as the hands that wield it. As Natasha contemplates this, she takes another sip of her coffee, staring at the digital fortress she’s vowed to protect. It’s still a chilly Monday morning, but Natasha feels the warmth of new determination coursing through her. The fight goes on. For now, things are quiet.

Almost too quiet?

And that’s our story for today. Journey to Trustworthy AI was produced in collaboration with Zove Security and UnGlitch. I’m Akshay Aggarwal, wishing you safe banking. In next week’s episode of Journey to Trustworthy AI we’ll cover Safeguarding AI Models.

Author’s note:

I originally posted this post on LinkedIn

Reference as A Journey Toward Trustworthy Artificial Intelligence: AI nightmare on Bank Street by Akshay Aggarwal, Zove Security

Upon gaining access to a top-tier generative Artificial Intelligence (AI) system, I found myself surprised by the revelations I encountered. For those unfamiliar, generative AI encompasses technologies such as GPT-4, the most recent AI system that powers notable chatbots like ChatGPT, Google Bard, Cohere, and DALL-E. Along with my colleagues from Zove Security, I was part of a small, expert team tasked to scrutinize the capabilities and constraints of this system, specifically investigating potential misuse. As the chief cybersecurity researcher for this effort, I am now sharing insights from our findings. This article is the first in a four-part series on our journey toward more trustworthy AI.

Over the last year, we formed a “red team” with the mission of conducting an adversarial test of the new model to identify potential vulnerabilities. Our exploration spanned various fields including aerospace, manufacturing, chemical engineering, and banking. We began by formulating hypotheses within these industries and aimed to use AI to either confirm or refute them. For instance, within aerospace, we speculated that we could potentially craft a faulty part that would pass initial tests but quickly fail under real-world conditions. This presented a dangerous possibility of deliberately defective spare parts infiltrating a supply chain and causing disastrous failures. Unfortunately, our hypothesis proved correct with unsettling ease.

Equally concerning was our success in evading financial fraud detection and behavior-based intrusion detection systems. However, the team was taken aback by the model’s thorough response to a hypothetical cyber-attack on crucial infrastructure.

Similar efforts by other teams leveraging OpenAI’s GPT, revealed the potential for the AI to suggest compounds suitable for nerve agents, effectively creating chemical weapons. By utilizing “plug-ins” to supply the ChatGPT chatbot with recent scientific research and a directory of chemical manufacturers, it was even able to pinpoint a potential manufacturing site for such a compound.

These discoveries highlighted the dichotomy of advanced AI technology. While it holds the power to boost and augment our scientific discoveries, it simultaneously harbors the potential to facilitate dangerous activities within physics, chemistry, and cybersecurity. Recognizing these risks, measures were taken to ensure that such outcomes are avoided when technology is made widely accessible.

Our AI Security team’s probing exercise aimed to alleviate public concerns regarding the deployment of potent AI systems in society. Our duty was not only to test the boundaries of the AI model but also to scrutinize it for potential issues like toxicity, bias, and linguistic prejudice. We assessed the model for a range of possible abuses, including perpetuating misinformation, facilitating plagiarism, and enabling illegal activities such as financial crimes and cyberattacks.

In our approach, we combined professional security analysts and penetration testers with industry experts. Over several months, these interdisciplinary teams formulated and tested hypotheses, aiming to breach current defenses and risk mitigations.

One mutual apprehension within the team was the risk associated with linking such powerful AI models to external knowledge sources via plug-ins. We elected not to take this route, although we understand that real-world adversaries may do so.

Another significant issue we identified involved bias in the model’s responses. We observed instances of gender, racial, and religious biases, along with overt stereotypes concerning marginalized communities.

Over time, we suggested alterations to the model and saw marked improvements in safety within the model’s responses. However, the quest for safety and fairness within AI systems remains a continuous endeavor.

In the grand scheme of technology, generative AI stands as an exciting frontier with its astounding capabilities. Our exploration has shed light on its dual nature – from empowering scientific advancements to potentially enabling harmful activities. Although our journey was filled with surprising discoveries, it reaffirmed the essential need for constant vigilance and innovation in cybersecurity. In a world where AI technologies are continually evolving, the work to ensure their safe, ethical, and fair use is an ongoing commitment. As we conclude this initial post in our series, we hope our insights will foster an understanding of the implications of AI in our society and the need for continued exploration. In the upcoming posts, we will delve deeper into the specific challenges and the corresponding measures we proposed to make AI a more secure and trustworthy tool for the future.

Authors Note:

Original Post: This post has been cross-posted from the original on LinkedIn

Next Posts in Series: AI nightmare on Bank Street

Reference as Behind the AI Curtain: A Journey Toward Trustworthy Artificial Intelligence by Akshay Aggarwal, Zove Security