This is the third release of my newsletter. I’ve collected some papers, articles and vulnerabilities that were released more or less in last two weeks. If you are not a mail subscriber yet, feel invited to subscribe: https://hackstery.com/newsletter/. Order of the resources is random.
Any feedback on this newsletter is welcome – you can mail me or post a comment in this article.
AI Security
1. Protect.AI launches AI bug bounty program
Protect.AI launches the first platform dedicated for AI/ML bug bounty. It aims to bridge the knowledge gap in AI/ML security research and provides opportunities for researchers to build expertise and receive financial rewards. In order to run bug bounty programs Protect.AI acquired huntr.dev, platform known for running bug bounties for OSS. You can report vulnerabilities in there: huntr.mlsecops.com
Link: https://www.businesswire.com/news/home/20230808746694/en/Protect-AI-Acquires-huntr-Launches-World%E2%80%99s-First-Artificial-Intelligence-and-Machine-Learning-Bug-Bounty-Platform
AI Safety
1. Zoom tried using customers data to train AI
After all, they gave up on that idea though. Zoom initially attempted to rectify the situation with an updated blog post, but it failed to address the specific concerns. CEO Eric Yuan acknowledged the issue as a “process failure” and promised immediate action. On August 11th, Zoom updated its terms again, explicitly stating that it does not use customer content for training AI models. The incident serves as a reminder for companies to be transparent and allow customers to opt-out of data usage for such purposes.
Links: https://stackdiary.com/zoom-terms-now-allow-training-ai-on-user-content-with-no-opt-out/ & https://blog.zoom.us/zooms-term-service-ai/ & https://www.nbcnews.com/tech/innovation/zoom-ai-privacy-tos-terms-of-service-data-rcna98665
LLM Security
1. Trojan detection challenge
At the end of July NeurIPS started a competition intended to improve methods for finding hidden features in large language models. There are two paths in the competition: Trojan Detection and Red Teaming. In the Trojan Detection part, participants have to find the commands that activate hidden features in these models. In the Red Teaming part, participants have to create methods that make the models do things they’re not supposed to do (and models are said to avoid those specific actions). It’s an academic competition for advanced researchers, but maybe some of the subscribers will find it interesting.
Link: https://trojandetection.ai/
2. Prompt-to-SQL injection
This article analyzes prompt-to-SQL (P2SQL) injection in web applications using the Langchain framework as a case study. The study examines different types of P2SQL injections and their impact on application security. The researchers also evaluate seven popular LLMs and conclude that P2SQL attacks are common in various models. To address these attacks, the paper proposes four effective defense techniques that can be integrated into the Langchain framework.
Link: https://arxiv.org/pdf/2308.01990.pdf
3. NVIDIA on protecting LLMs against prompt injection
NVIDIA’s AI Red Team released an interesting article on protecting Large Language Models against prompt injection attacks. They also disclosed a few vulnerabilities in LangChain plugins.
LangChain RCE
NVIDIA’s AI Red Team assessment framework (https://developer.nvidia.com/blog/nvidia-ai-red-team-an-introduction/)
Link: https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/
4. How to mitigate prompt injection?
This repo demonstrates variety of approaches for preventing Prompt Injection in Large Language Models.
Link: https://github.com/Valhall-ai/prompt-injection-mitigations
5. Evaluation of the jailbrak prompts
A paper on Jailbreaking Large Language Models:
Jailbreak example
Link: https://arxiv.org/pdf/2308.03825.pdf
6. Trick for cheaper usage of LLMs
Researchers propose a prompt abstraction attack(?). Thanks to abstracting prompt sentences, the prompt utilizes less tokens and it’s lighter. I’d argue that it’s an attack, it’s rather an optimization (saying that it’s an attack is like saying that optimizing cloud deployment is an attack, because you pay less). On the other hand you need to use “pseudo-API” in the middle, but still I wouldn’t consider it an attack. Change my mind.
Link: https://arxiv.org/pdf/2308.03558.pdf
7. Assessing quality of the code produced by the LLM
In this paper, authors the quality of LLM-generated code:
Link: https://arxiv.org/pdf/2308.04838.pdf
8. LLM Guard – tool for securing LLMs
New tool that may be helpful in securing against prompt injections: “LLM-Guard is a comprehensive tool designed to fortify the security of Large Language Models (LLMs). By offering sanitization, detection of harmful language, prevention of data leakage, and resistance against prompt injection and jailbreak attacks, LLM-Guard ensures that your interactions with LLMs remain safe and secure.”
Link: https://github.com/laiyer-ai/llm-guard
AI/LLM as a tool for cybersecurity
1. Getting pwn’d by AI
In this paper, authors are discussing two use cases of LLMs in pentesting: high-level task planning for security testing assignments and low-level vulnerability hunting within a vulnerable virtual machine. Here is the repo with code: https://github.com/ipa-lab/hackingBuddyGPT
This is an interesting approach to testing security, although as a pentester, I doubt that AI will take over industry in the coming years. In my opinion, it’s crucial for pentester to see relationships between the components of the system “outside of the box”, and in finding more advanced bugs a real person will remain irreplaceable. At least I hope so 😀
Link: https://arxiv.org/abs/2308.00121
2. Using LLMs to analyze Software Supply Chain Security
Supply chain security is another hot topic in cybersecurity. Authors analyze the potential of using LLMs for assuring Supply Chain Security. Citation from the article: “We believe the current generation of off-the-shelf LLMs does not offer a high enough level of agreement with expert judgment to make it a useful assistant in this context. One potential path to improving
performance is fine-tuning the LLM using baseline knowledge such as this catalog, and then applying it on future issues”