Welcome to the second installment of ‘Journey to Trustworthy AI‘. Here, we continue our exploration of AI security, an area that’s changing at an unprecedented pace. In our first article, we outlined our project testing generative AI with security red team. We could hardly have imagined how quickly fiction would confront a real-world adversary.
The ensuing narrative is partly based on events from the past year, giving us a glimpse of what the future could hold in this rapidly evolving field. For confidentiality, we’ve protected the identities of all individuals and organizations involved, and obscured operational details.
Let’s take a step back into a different world, where fast response isn’t a cornerstone of banking systems, where our protagonist Natasha has moved on from the shiny marbled floors of Wall Street to a vibrant, rapidly growing neobank, AnonNeo Fintech. A significant shift, from a traditional banking setup to the digital frontier of finance. Natasha’s not just dealing with ledgers and transactions anymore, but with lines of code, digital wallets, and a customer base as diverse as the city she once used to call home. It’s still 2023, but the rules of the game are different.
Natasha, now the head of fraud at AnonNeo, finds herself dealing with a very different set of challenges. She’s no longer playing in the familiar territory of well-defined transactions and traditional fraud patterns. Her new playground is a volatile landscape of varied customer behavior, each one as unique as the individual behind the screen. They’re digital natives, making payments at online gaming platforms at midnight, buying cryptocurrencies at dawn, splitting bills over lunch, and shopping from international websites while commuting. They’re also well-funded startups looking to do more with their money, leverage better spending control for employees, with founders averse to walk into a traditional bank branch. The one thing that doesn’t change though is that they all care about protecting their financial assets.
Natasha’s team is well versed in the arts of the old guard, yet nimble enough to learn new tunes. She is a maestro in a symphony of firewalls and encryption, databases, and servers. Her orchestra is the heartbeat of the neobank, a rhythm she’s dedicated her life to protecting. But something is slowly going out of tune, something that will bring a new composition to life.
The months are rolling by in an unremarkable fashion. But somewhere, unbeknownst to Natasha and her team, a stealthy melody starts playing. Small, irregular transactions are taking place. They are minuscule, barely noticeable, not large enough to raise alarm in a traditional fraud detection system, lost in the daily humdrum of thousands of transactions.
Weeks turn into months, the silent tune continues, an uninvited soloist in Natasha’s orchestra. Natasha senses something’s off but can’t put a finger on it. A feeling that this second-generation immigrant finds hard to ignore. There are no big fraudulent transactions, no significant breaches, no grand alerts. Just a quiet, unsettling feeling that something is amiss.
As this melody becomes more persistent, Natasha decides to dig deeper. A detailed analysis is conducted on an unprecedented scale, sifting through every transaction, every log, every account. AnonNeo puts in place stricter controls, tighter monitoring systems. But this new tune is adapting, changing with every measure the bank enforces. It’s intelligent, it’s stealthy, it’s… alive.
Finally, after a grueling investigation, the harsh truth is unveiled. The neobank has been hemorrhaging money in the form of small, unnoticed transactions. These transactions have been slowly but steadily adapting to every new security measure, every control the bank puts in place. It was not an attack; it was a siege.
The silent soloist behind this was not a human but an AI. A smart, learning system, patiently and persistently conducting its stealthy symphony, draining the bank’s resources. It’s a sobering realization, a chilling testament to the power of artificial intelligence in the hands of wrongdoers.
The challenge amplifies when Natasha attempts to employ AI systems for fraud detection. She’s faced with her second obstacle: a messy, unstructured dataset, a reflection of her diverse customer base. The AI needs clean, well-labeled data to learn from, to understand what’s normal and what’s not. But the data Natasha has is more like an abstract painting than a neat spreadsheet.
As she grapples with these challenges, the ghost of the silent, stealthy adversary evolves almost as if it detects their attempts to stop it. A gradual, almost imperceptible outflow of funds turns into a deluge. The stealthy AI attack that she encountered is even more complex and adaptable than before.
Natasha sees the patterns she’d come to dread – the attack is as intelligent and patient. Only this time, it’s adjusting itself to a broader, more unpredictable range of customer behaviors. It’s thriving in the same chaos that is confounding Natasha’s attempts at setting up a robust AI-based fraud detection system.
This attack serves as a wake-up call, not just for Natasha’s bank, but the entire financial sector. AI technology represents a double-edged sword. On the one hand, AI offers improved services, reduced costs, and increased efficiency. On the other hand, it opens new vectors of attack for cybercriminals, who are ready to exploit the very same technology.
For Natasha, and everyone in fraud protection within the financial services industry, the fight against cybercrime is not a static one. They can’t rest on their laurels, celebrating the power of AI, while malicious actors are using the same technology against them. The war has a new battlefield, and they must adapt their strategies. They must learn about adversarial attacks, invest in adversarial training for their AI systems, and above all, stay vigilant.
In the end, AI, like any other tool, is only as good or bad as the hands that wield it. As Natasha contemplates this, she takes another sip of her coffee, staring at the digital fortress she’s vowed to protect. It’s still a chilly Monday morning, but Natasha feels the warmth of new determination coursing through her. The fight goes on. For now, things are quiet.
Almost too quiet?
And that’s our story for today. Journey to Trustworthy AI was produced in collaboration with Zove Security and UnGlitch. I’m Akshay Aggarwal, wishing you safe banking. In next week’s episode of Journey to Trustworthy AI we’ll cover Safeguarding AI Models.
Author’s note:
I originally posted this post on LinkedIn
Reference as A Journey Toward Trustworthy Artificial Intelligence: AI nightmare on Bank Street by Akshay Aggarwal, Zove Security
