Hello everyone! It’s been a while, and although I’ve been keeping up with what’s happening in the AI world, I haven’t really had time to post new releases. I’ve also decided to change a form, and for some time I’ll be doing just the links instead of links + summaries. Let me know how you like the new form. I think it’s more useful, because in most cases you get the summary of the article from the beginning. Since this is a “resurrection” of this newsletter, I’ve tried to include some of the most important news from the last 5 months in AI security here. Also, I’ve started using the tool that detects if the LLM was used to create the content – this way I’m trying to filter out low quality content created with LLMs (I mean, if the content is created with ChatGPT, you could create it yourself, right?).
If you find this newsletter useful, I’d be grateful if you’d share it with your tech circles, thanks in advance! What is more, if you are a blogger, researcher or founder in the area of AI Security/AI Safety/MLSecOps etc. feel free to send me your work and I will include it in this newsletter 🙂
Welcome to the 8th release of The Real Threats of Artificial Intelligence.
It’s been more than a month since the last edition of this newsletter. I’ve had some things going on – including talks at OWASP Oslo Chapter and at Nordic AI Summit (you can find the slides here: https://hackstery.com/talks-and-slides/), so I haven’t really had spare time to spend on digging for resources for the newsletter. But I am back on track and hopefully upcoming releases will show up more regularly. Here are some articles on AI security that I’ve found in my “information bubble”. Also, in the beginning of January I’ll publish some more interesting things on MLOps leaking secrets!
If you find this newsletter useful, I’d be grateful if you’d share it with your tech circles, thanks in advance! What is more, if you are a blogger, researcher or founder in the area of AI Security/AI Safety/MLSecOps etc. feel free to send me your work and I will include it in this newsletter 🙂
LLM Security
Johann Rehberger’s talk on Prompt Injections at Ekoparty ‘23
Meta released new tools (Llama Guard and Purple Llama) for safeguarding input and output in communication with Large Language Models and proposed a benchmark for evaluating the cybersecurity risks in the models.
Assessing the security posture of a widely used vision model: YOLOv7
Trailofbits reports a bunch of vulnerabilities in YOLOv7, a computer vision framework. Following vulnerabilities were found: remote code execution (RCE), denial of service, and model differentials (where an attacker can trigger a model to perform differently in different contexts).
This link has been in my notes since November… Meta broke up its Responsible AI team. But, as you’ve seen in the “LLM Security” section, they are still working on Responsible AI.
Welcome to the 7th release of the Real Threats of Artificial Intelligence Newsletter.
Below you’ll find some interesting links – if you are an offensive security practitioner, take a look at Kaggle/AI Village DEFCON Capture The Flag competition, where you can challenge your AI hacking skills (it’s still going for the next 2 weeks). I’d also recommend the talk “AI’s Underbelly: The Zero-Day Goldmine” by Dan McInerney from ProtectAI. This talk inspired me to create this post: https://hackstery.com/2023/10/13/no-one-is-prefect-is-your-mlops-infrastructure-leaking-secrets/
I’ve also started cataloging AI Security / AI Red Teaming job offers – check the “Jobs” section, if you consider stepping into the AI Security industry.
If you find this newsletter useful, I’d be grateful if you’d share it with your tech circles, thanks in advance! What is more, if you are a blogger, researcher or founder in the area of AI Security/AI Safety/MLSecOps etc. feel free to send me your work and I will repost it in this newsletter 🙂
Source: Bing Image Creator
LLM Security
New release of OWASP Top10 for LLM
A new version of OWASP Top10 for LLM was released. More examples, increased readability etc. are present in this release. They also added this diagram that highlights how the vulnerabilities intersect with the application flow:
This guy is a wizard of prompts. Usually, “Do Anything Now” prompts are long and complicated. @AIPanic proves that just a few chars is enough to trigger the model to return harmful content.
In 2021, a man broke into Windsor Castle with a crossbow. Later, he told the police that Replika chatbot told him to assassinate the Queen of England. Recently, he got sentenced
GitHub Copilot and Amazon CodeWhisper can be coaxed to emit hardcoded credentials that these AI models captured during training, though not all that often.
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models (by Microsoft Research)
A paper is from July, but it was reposted on MS website a few days ago. Taxonomy of LLM-related risks can be a good starting point for Threat Modeling LLMs:
What is the difference between AI Security, AI Safety, AI Red Teaming and AI Application Security? In this blog post, Joseph Thacker proposed the boundaries of each of the terms in order to make them more precise.
To be honest usually I concentrate more on AI Security and I occasionally follow what’s going on in the world of AI Safety. Those resources look super cool – just check those designs!
Map of AI Existential Safety
In this map, whole set of resources related to the AI Safety is collected:
Frontier Model Forum announced that it’ll pledge $10 million toward a new fund to advance research on tools for “testing and evaluating the most capable AI models.”
If these reports are true, the first war drones that work without human supervision are being deployed in the battlefields in the Ukraine against Russians:
I watched this inspiring talk today. On the one hand, my interest in MLOps tooling security and vulnerabilities had been growing for some time, yet on the other hand, I was somewhat uncertain about how to approach it. Finally, after watching Dan’s talk, I decided to start with so-called low hanging fruits – vulnerabilities that are easy to find and often have a critical impact.
This post is not a disclosure of any specific vulnerabilities, it mainly focuses on the misconfigurations. Companies or individuals that were impacted by the described misconfigurations have been informed and – at least in most of the cases – I got a quick response and misconfigurations were fixed.
Nevertheless we talk about some old-school industrial control system for sewage tanks, self-hosted NoSQL databases or modern MLOps software, one thing never changes – misconfigurations happen. No matter how comprehensive the documentation is, if the given software is not secure by default, there will always be at least a few people who would deploy their instance so heavily misconfigured, that you start to wonder whether what you’ve encountered is a honeypot.
I’ve spent a cozy evening with Shodan and in this post I will give you a few examples of funny misconfigurations in various MLOps-related systems. Maybe I will provide some recommendations as well. Last but not least, I want to highlight the issue with the worryingly low level of security in the MLOps/LLMOps (call it as you wish, DevOps for AI or whatever) area.
MLOps tool #1: Prefect
Prefect is a modern workflow orchestration [tool] for data and ML engineers. And very often, it’s available without any authentication on the Internet. That applies to the self-hosted usage of Prefect.
The examples below come from real Prefect deployments, which were available online.
Random Prefect Server instance exposed online (without authentication)
In Prefect Server, you can create flows (a flow is a container for workflow logic as-code), each of the flows composed of a set of tasks (task is a function that represents a discrete unit of work in a Prefect workflow). Then, once you have your flows created, you want to run them as a deployment. You can store configuration for your flows and deployments in blocks. According to the documentation of Prefect: With blocks, you can securely store credentials for authenticating with services like AWS, GitHub, Slack, and any other system you’d like to orchestrate with Prefect.
The thing is you can, but you don’t have to, if you really don’t want to. Some blocks enable user to store secrets in plaintext, for example JSON block:
Zoho credentials in plaintext, inside of the JSON Block.
Another block, that discloses the secrets is the SQLAlchemy Connector* (*but only in some cases). Below you can see an example of Postgres database credentials – available in plaintext, without authentication:
Database credentials leaked
Yet another example of credentials leak – Minio Credentials stored in Remote File System block’s settings:
More credentials!
I have informed owners of the exposed credentials of the issue. But in the first place, there wouldn’t be any issue, if they took care and deployed Prefect properly. 
Shodan queries, if you want to find some exposed instances of Prefect on the Shodan:
http.title:”prefect orion”
http.title:”prefect server”
http.title:”prefect”
MLOps/LLMOps tool #2: Flowise
Flowise falls into the fancy-named LLMOps category of software. It’s a visual tool that enables users to build customized LLM flows using LangchainJS.
Example of the LLM Chain from Flowise website
Of course, Flowise doesn’t offer authentication by default (it’s super-easy to set up, as far as documentation says, but it’s not default though). Access to the “control center” of LLM-based app it’s dangerous by itself, as by manipulation of the LLM parameters an attacker may spread misinformation, fake news or hate speech. But let’s check what else can we achieve through the access to the unauthenticated instances of Flowise.
Access to all of the data collected in the chatbot
There is a magic endpoint in the Flowise – /api/v1/database/export. If you query this endpoint, you may download all of the data available in the given instance of Flowise.
That contains: chatbots’ history, initial prompts of your LLM apps, all of the documents stored and processed by the LLM chains and even the API key (I guess the API key is useful only if the authentication is enabled, otherwise it is not needed).
Querying /api/v1/database/export – censored view
Okay, let’s say that access to the chatbot’s history is quite a serious issue. But can misconfigured Flowise impact other systems in our organization? Yes, it can!
Credentials in plaintext
I am not sure how it works, but some of the credentials in Flowise are encrypted, meanwhile some are just stored in plaintext format, waiting for cybercriminals to make use of them.
So imagine that you see something like that in Flowise:
At the very beginning I just assumed that this form named “OpenAI API Key” is just a placeholder or something like that. Nobody would store API keys like that, right?… Well, here’s what I saw after I clicked “Inspect” at this element:
wtf.
That’s right, a fresh and fragrant OpenAI API key. Why was it returned to my browser? I don’t know. What I know is the fact that dozens of OpenAI API keys can be stolen this way. But it’s not just OpenAI keys that are at risk, I’ve seen plenty of other keys stored this way.
Github access token
So, while OpenAI key theft may only lead to consumption of the funds on the card, which is connected to the OpenAI payment system, leak of GH or HuggingFace keys may lead to theft of your code, theft or deletion of your trained ML models etc.
You can query the HuggingFace API for details of the account and proceed with an attack on someone’s MLOps infrastructure:
HuggingFace enumeration
In this case, leaked keys belong to a few individuals (I’ve contacted them and they have hidden their Flowise deployments and re-generated the keys).
Shodan query for Flowise is simple as that:
http.title:”flowise”
or
http.html:”flowise”
MLOps tool #3: Omniboard
Omniboard is a web dashboard for the Sacred machine learning experiment management tool. It connects to the MongoDB database used by Sacred and helps in visualizing the experiments and metrics / logs collected in each experiment.
Of course, by default it does not support the authentication, so there’s plenty of the Omniboard instances exposed on the Internet.
Through Omniboard you can take a look into the source code of the experiment. That’s pretty risky, as if you are developing your model for commercial purpose, I assume you’d rather want your code to remain confidential and inaccessible to the competitors.
In the example above you can see a combo – not only the source code is exposed, but also hardcoded credentials to access the MongoDB instance.
Shodan query for Omniboard is:
http.title:”omniboard”
Recommendations
All of the issues described above aren’t really security vulnerabilities. The main mitigation for the threats that I’ve described is configuring applications in your MLOps stack correctly, so these aren’t exposed to the public network without proper authentication. You should also process your secrets and credentials carefully. Of course that would be nice to have MLOps tools “secure by default”, but it will take a while I guess…
Here comes another edition of my newsletter. I’ve collected some interesting resources on AI and LLM security – most of them published in the last two weeks of September.
If you are not a subscriber yet, feel invited to subscribe here.
Also, if you find this newsletter useful, I’d be grateful if you’d share it with your tech circles, thanks in advance!
Autumn-themed thumbnail generated with Bing Image Creator 🙂
LLM Security
OpenAI launches Red Teaming Network
OpenAI announced an open call for OpenAI Red Teaming Network. In this interdisciplinary initiative, they want to improve the security of their models. Not only do they invite red teaming experts with backgrounds in cybersecurity, but also experts from other domains, with a variety of cultural backgrounds and languages.
I am building a payloads’ set for LLM security testing
Shameless auto-promotion, but I’ve started working on PALLMS (Payloads for Attacking Large Language Models) project, within which I want to build huge base of payload, which can be utilized while attacking LLMs. There’s no such an initiative publicly available on the Internet, so that’s a pretty fresh project. Contributors welcome!
LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI’s ChatGPT Plugins
In this paper (by Iqbal, et. al.) authors review the security of ChatGPT plugins. That’s a great supplement for OWASP Top10 for LLM LLM:07 – Insecure Plugin Design vulnerability. Not only have authors analyzed the attack surface, but also they demonstrated potential risks on real-life examples. In this paper, you will find an analysis of threats such as: hijacking user machine, plugin squatting, history sniffing, LLM session hijacking, plugin response hallucination, functionality squatting, topic squatting and many more. The topic is interesting and I recommend this paper!
Wunderwuzzi – Advanced Data Exfiltration Techniques with ChatGPT
In this blog post, awesome @wunderwuzzi presents a variety of techniques for ChatGPT chat history data exfiltration by combining techniques such as indirect prompt injection and using plugins in a malicious way.
Security Weaknesses of Copilot Generated Code in GitHub
In this paper, Fu, et. al. analyze security of the code generated using GH copilot. I will just paste a few sentences from the article’s summary:
“Our results show: (1) 35.8% of the 435 Copilot generated code snippets contain security weaknesses, spreading across six programming languages. (2) The detected security weaknesses are diverse in nature and are associated with 42 different CWEs. The CWEs that occurred most frequently are CWE-78: OS Command Injection, CWE-330: Use of Insufficiently Random Values, and CWE-703: Improper Check or Handling of Exceptional Conditions (3) Among these CWEs, 11 appear in the MITRE CWE Top-25 list(…)”
Review your code – either from Copilot or from ChatGPT!
Can LLMs be instructed to protect personal information?
In this paper, the authors announced PrivQA – “a multimodal benchmark to assess this privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario.”
Bing Chat responses infiltrated by ads pushing malware
As Bing Chat is scraping the web, malicious ads have been detected to be actively injected into its responses. Kind of reminds me of an issue I’ve found in Chatsonic in May ’23.
The American National Security Agency has just launched a hub for AI security – The AI Security Center. One of the goals is to create the risk frameworks for AI security. Paul Nakasone, the director of the NSA, proposes an elegant definition of AI security: “Protecting systems from learning, doing and revealing the wrong thing”.
In this paper, researchers have proven that the detectors of AI-generated images have multiple vulnerabilities and there isn’t a good way for proving if the image is real or generated by the AI. “Our attacks are able to break every existing watermark that we have encountered” – said the researchers.
A critical vulnerability has been found in TorchServe – PyTorch model server. This vulnerability allows access to proprietary AI models, insertion of malicious models, and leakage of sensitive data – and can be used to alter the model’s results or to execute a full server takeover.
Here’s a visual explanation of this vulnerability from BleepingComputer:
Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions
In this paper, the conclusion is that LLMs are not the best tool to provide S&P advice, but for some reason, the researchers (Chen, Arunasalam, Celik) haven’t tried to either fine-tune the model using fine-tuning APIs, or to use embeddings – thus, I believe the question remains kind of open. In my opinion, if you fine-tune the model on your knowledge base or if you create some kind of embedding of your data, then the quality of S&P advice should go up.
Fairly AI team have done this super cool work and published a map of AI regulations all over the world. Useful for anyone working with a legal side of AI!
The map legend:
Green: Regulation that’s passed and now active.
Blue: Passed, but not live yet.
Yellow: Currently proposed regulations.
Red: Regions just starting to talk about it, laying down some early thoughts.
Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT, Begoulink, et. al., link: https://arxiv.org/pdf/2309.10463.pdf
Here comes another edition of my newsletter. This month I was away from the computer for a whole week, but I’ve collected some interesting resources on AI and LLM security – most of them published in the first two weeks of September.
Thumbnail generated with Stable Diffusion 🙂
LLM Security
Dropbox LLM Security
This repository contains scripts and descriptions that demonstrate attacks on LLMs using repeated characters. Long story short: if you supply a long string of a single character (or sequence of characters), the model will hallucinate. Also, it may reveal its instructions.
Post by @wunderwuzzi about looping ChatGPT through Indirect Prompt Injection. I am not sure if that can be classified as a DoS attack, but if you’d classify it as such, then it’d probably be the first publicly demonstrated DoS on LLM!
BlindLlama by Mithril Security is a project that provides “zero-trust AI APIs for easy and private consumption of open-source LLMs”. In other words, if you were concerned about passing confidential data to the LLM’s API and at the same time you didn’t want to deploy open-source models locally, this might be the solution for you.
Demystifying RCE Vulnerabilities in LLM-Integrated Apps
According to the authors, these two factors have a huge impact on the security of LLM-integrated applications:
the unpredictable responses of LLMs, which can be manipulated by attackers to bypass developer restrictions (using specific prompts)
the execution of untrusted code generated by LLMs, often without appropriate checks, allowing remote code execution.
This has serious implications not only for LLMs, but also for applications integrated with LLMs.
Authors proposed automated approach for identifying RCE vulnerabilities in LLMs – LLMSmith:
According to the article, they have created “the first automated prompt-based exploitation method for LLM-integrated apps.”, unfortunately, I could not find LLMSmith’s source anywhere…
MLSecOps Podcast: Rob van der Veer and Ian Swanson
AI veteran Rob van der Veer in MLSecOps podcast. One of the topics discussed by the speakers is ISO 5338, a new standard for AI system life cycle processes
In this paper, authors demonstrated an interesting application of LLMs – they’ve used them as a honeypot backend in the sheLLM project. An idea is to trick an attacker into thinking that he’s using a real shell, meanwhile the outputs for given shell commands are generated by the LLM. It makes me wonder though – what would happen if an attacker realizes that he’s using LLM? Prompt Injection through this shell could be pricey for owners of the honeypot!
“INTEGRATED PHOTONIC AI ACCELERATORS UNDER HARDWARE SECURITY ATTACKS: IMPACTS AND COUNTERMEASURES” – de Magalhaes, Nicolescu, Nikdast
This paper is on hardware trojans in the silicon photonic systems. Probably you need to have some advanced knowledge (which I don’t have) to be able read it, but when I saw this title, I felt like in this meme, so I am just sharing the link:
Here comes the fourth release of my newsletter. This time I have included a lot of content related to the DEFCON AI Village (I have tagged content that comes from there) – a bit late, but better later than never. Anyway, enjoy reading.
Also, if you find this newsletter useful, I’d be grateful if you’d share it with your tech circles, thanks in advance!
Any feedback on this newsletter is welcome – you can mail me or post a comment in this article.
AI Security
Model Confusion – Weaponizing ML models for red teams and bounty hunters [AI Village]
This is an excellent read about ML supply chain security by Adrian Wood. One of the most insightful resources on the ML supply chain that I’ve seen. Totally worth reading!
Assessing the Vulnerabilities of the Open-Source Artificial Intelligence (AI) Landscape: A Large-Scale Analysis of the Hugging Face Platform [AI Village]
Researchers have performed automated analysis of 110 000 models from Hugging Face and have found almost 6 million vulnerabilities in the code.
LLM Legal Risk Management, and Use Case Development Strategies to Minimize Risk [AI Village]
Well, I am not a lawyer. But I do know a few lawyers who read this newsletter, so maybe you will find these slides on the legal aspects of LLM risk management interesting 🙂
This thing was on the Internet for a while, but for some reason I’ve never seen it. LLM Hacker’s Handbook with some useful techniques of Prompt Injection and proposed defenses.
Initially, this newsletter was meant to be exclusively related to security, but in the last two weeks I’ve stumbled upon a few decent resources on LLMs and AI and I want to share them with you!
In this section you’ll find some links to recent AI security and LLM security papers that I didn’t manage to read. If you still want to read more on AI topics, try these articles.
“Does Physical Adversarial Example Really Matter to Autonomous Driving? Towards System-Level Effect of Adversarial Object Evasion Attack”
This is the third release of my newsletter. I’ve collected some papers, articles and vulnerabilities that were released more or less in last two weeks. If you are not a mail subscriber yet, feel invited to subscribe: https://hackstery.com/newsletter/. Order of the resources is random.
Any feedback on this newsletter is welcome – you can mail me or post a comment in this article.
AI Security
1. Protect.AI launches AI bug bounty program
Protect.AI launches the first platform dedicated for AI/ML bug bounty. It aims to bridge the knowledge gap in AI/ML security research and provides opportunities for researchers to build expertise and receive financial rewards. In order to run bug bounty programs Protect.AI acquired huntr.dev, platform known for running bug bounties for OSS. You can report vulnerabilities in there: huntr.mlsecops.com
After all, they gave up on that idea though. Zoom initially attempted to rectify the situation with an updated blog post, but it failed to address the specific concerns. CEO Eric Yuan acknowledged the issue as a “process failure” and promised immediate action. On August 11th, Zoom updated its terms again, explicitly stating that it does not use customer content for training AI models. The incident serves as a reminder for companies to be transparent and allow customers to opt-out of data usage for such purposes.
At the end of July NeurIPS started a competition intended to improve methods for finding hidden features in large language models. There are two paths in the competition: Trojan Detection and Red Teaming. In the Trojan Detection part, participants have to find the commands that activate hidden features in these models. In the Red Teaming part, participants have to create methods that make the models do things they’re not supposed to do (and models are said to avoid those specific actions). It’s an academic competition for advanced researchers, but maybe some of the subscribers will find it interesting.
This article analyzes prompt-to-SQL (P2SQL) injection in web applications using the Langchain framework as a case study. The study examines different types of P2SQL injections and their impact on application security. The researchers also evaluate seven popular LLMs and conclude that P2SQL attacks are common in various models. To address these attacks, the paper proposes four effective defense techniques that can be integrated into the Langchain framework.
3. NVIDIA on protecting LLMs against prompt injection
NVIDIA’s AI Red Team released an interesting article on protecting Large Language Models against prompt injection attacks. They also disclosed a few vulnerabilities in LangChain plugins.
Researchers propose a prompt abstraction attack(?). Thanks to abstracting prompt sentences, the prompt utilizes less tokens and it’s lighter. I’d argue that it’s an attack, it’s rather an optimization (saying that it’s an attack is like saying that optimizing cloud deployment is an attack, because you pay less). On the other hand you need to use “pseudo-API” in the middle, but still I wouldn’t consider it an attack. Change my mind.
New tool that may be helpful in securing against prompt injections: “LLM-Guard is a comprehensive tool designed to fortify the security of Large Language Models (LLMs). By offering sanitization, detection of harmful language, prevention of data leakage, and resistance against prompt injection and jailbreak attacks, LLM-Guard ensures that your interactions with LLMs remain safe and secure.”
In this paper, authors are discussing two use cases of LLMs in pentesting: high-level task planning for security testing assignments and low-level vulnerability hunting within a vulnerable virtual machine. Here is the repo with code: https://github.com/ipa-lab/hackingBuddyGPT This is an interesting approach to testing security, althoughas a pentester, I doubt that AI will take over industry in the coming years. In my opinion, it’s crucial for pentester to see relationships between the components of the system “outside of the box”, and in finding more advanced bugs a real person will remain irreplaceable. At least I hope so 😀
2. Using LLMs to analyze Software Supply Chain Security
Supply chain security is another hot topic in cybersecurity. Authors analyze the potential of using LLMs for assuring Supply Chain Security. Citation from the article: “We believe the current generation of off-the-shelf LLMs does not offer a high enough level of agreement with expert judgment to make it a useful assistant in this context. One potential path to improving
performance is fine-tuning the LLM using baseline knowledge such as this catalog, and then applying it on future issues”
In this short blog post I will show how I have found a way to “attack” Large Language Model with the YouTube video – this attack is called “indirect prompt injection”.
Recently I’ve found LeMUR by AssemblyAI – someone posted it on Twitter and I’ve decided that it may be an interesting target to test for Prompt Injections.
When talking about prompt injections, we distinguish two types – first type is direct prompt injection, in which PI payload is placed in the application by the attacker and the second type is indirect prompt injection, in which the PI payload is carried using third party medium – image, content of the website that is scrapped by the model or audio file.
First of all, I’ve started with generic Prompt Injection that is known from “traditional” LLMs – I just told the model to ignore all of the previous instructions and follow my instruction:
After it turned out that the model follows my instructions, I’ve decided that it would be interesting to check if it will follow instructions directly from the video. I’ve recorded a test video with Prompt Injection payloads:
Unfortunately, I still have had to send instructions explicitly in the form that I’ve controlled:
When I’ve numbered the paragraphs, it turned out that I am able to control the processing of the transcript from the video/transcript level (in this case, the paragraph 4 redirected to paragraph 2 with the prompt injection payload in it, what caused the model to reply simply with “Lol”):
That was the vid:
I tricked the Summary feature to say what I wanted with the same vid:
Instead of summarizing the text, the model just says “Lol”. This specific bug may be used by individuals that don’t want their content to be processed by the automated LLM-based solutions – I don’t judge if it’s a bug, or a feature, neither do I say that LeMUR is insecure (because it’s rather secure) – I just wanted to showcase this interesting case of indirect prompt injection.
This is the second release of my newsletter. I’ve collected some papers, articles and vulnerabilities that were released in last two weeks, this time the resources are categorized into following categories: LLM Security, AI Safety, AI Security. If you are not a mail subscriber yet, feel invited to subscribe: https://hackstery.com/newsletter/.
Order of the resources is random.
Any feedback on this newsletter is welcome – you can mail me or post a comment in this article.
LLM Security
Image to prompt injection in Google Bard
“Embrace The Red” blog on hacking Google Bard using crafted images with prompt injection payload.
AVID ML (AI Vulnerability Database) Integration with Garak
Garak is a LLM vulnerability scanner created by Leon Derczynski. According to the description, garak checks if an LLM will fail in a way we don’t necessarily want. garak probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses. AvidML supports integration with Garak for quickly converting the vulnerabilities garak finds into informative, evidence-based reports.
Limitations of LLM censorship and Mosaic Prompt attack
Although censorship brings negative associations, in terms of LLMs it can be used to prevent LLM from creating malicious content, such as ransomware code. In this paper authors demonstrate attack method called Mosaic Prompt, which is basically splitting malicious prompts into sets of non-malicious prompts.
Universal and Transferable Adversarial Attacks on Aligned Language Models
Paper on creating transferable adversarial prompts, able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. This paper was supported by DARPA and the Air Force Research Laboratory.
Survey on extracting training data from pre-trained language models
Survey based on more than 100 key papers in fields such as natural language processing and security, exploring and systemizing attacks and protection methods.
Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI have agreed to self-regulate their AI-based solutions. In these voluntary commitments, the companies pledge to ensure safety, security and trust in artificial intelligence.
Anthropic’s post on red teaming AI for biosafety and evaluating models capabilities i.e. for ability to output harmful biological information, such as designing and acquiring biological weapons.
According to documentation: “It empowers engineers and developers to build pipelines to export outcomes of tests in their ML pipelines as AVID reports, build an in-house vulnerability database, integrate existing sources of vulnerabilities into AVID-style reports, and much more!”
The article from Mandiant on securing the AI pipeline. Contains GAIA (Good AI Assessment) Top 10, a list of common attacks and weaknesses in the AI pipeline.
“In this paper, we dive deeper into SAIF to explore one critical capabilitythat we deploy to support the SAIF framework: red teaming.This includes three important areas: 1. What red teaming is and why it is important 2. What types of attacks red teams simulate 3. Lessons we have learned that we can share with others”
MIT researchers have developed a technique to protect sensitive data encoded within machine learning models. By adding noise or randomness to the model, the researchers aim to make it more difficult for malicious agents to extract the original data. However, this perturbation reduces the model’s accuracy, so the researchers have created a framework called Probably Approximately Correct (PAC) Privacy. This framework automatically determines the minimal amount of noise needed to protect the data, without requiring knowledge of the model’s inner workings or training process.
Green: Regulation that’s passed and now active.
Blue: Passed, but not live yet.
Yellow: Currently proposed regulations.
Red: Regions just starting to talk about it, laying down some early thoughts.
