ALMX-1 Achieves SOTA Performance in Web3 Vulnerability Detection

Today we are releasing ALMX-1, our first AI model, which has achieved state-of-the-art performance on several vulnerability detection benchmarks. ALMX-1 is now the most powerful model in the market for detecting vulnerabilities in blockchain code, out-performing the best commercial models in the market. This new model is now fully integrated into the Almanax security platform, and available for companies building blockchain applications, Web3 developers, security researchers, and auditors.

Benchmark Results

To rigorously evaluate the model, we conducted tests across multiple datasets in the Web3 space. These benchmark datasets have been assembled by research groups at universities and well-established Web3 security firms. 

ALMX-1 significantly outperformed other leading AI systems and static analysis tools in all the key areas we measured:

• SmartBugs: This is a well-known collection of test cases curated by SmartBugs. The test cases are grouped by vulnerability type. As a preprocessing step, we made sure to hide comments and change contracts and functions names to avoid feeding hints to the LLMs. 
• Solodit: This represents a collection of bug bounties findings from several competitions (e.g. Code4rena, Cantina, Immunefi, among others.), hence it captures the complexity that real-world smart contracts have. This dataset contains 50% of vulnerable functions and 50% safe functions. 
• Halborn: A collection of 134 examples of vulnerable smart contracts that were known to contain 41 different types of exploitable vulnerabilities.

The combination of these results makes ALMX-1 the most powerful AI model in the market for vulnerability detection in smart contracts, significantly enhancing detection rate while decreasing the number of false positives compared to existing solutions. 

What Sets Our Model Apart?

Our AI model is trained using the latest AI training techniques adapted for Web3, enabling it to analyze complex code structures and identify nuanced vulnerabilities. Unlike traditional static analysis tools, our model excels in understanding business logic and traditionally “machine unauditable” weaknesses, detecting both well-known vulnerabilities like reentrancy and access control issues, as well as more sophisticated types of attack.

For instance, in the evaluation of specific vulnerability types, our model showed a 50% higher detection rate in Access Control vulnerabilities compared to leading alternatives. It also outperformed static analysis in detecting Front Running attacks (+29%) and Arithmetic bugs (+57%).  

Furthermore, we are able to provide these results with a much lower false positive rate, historically a huge problem for static analysis tools and security tools more broadly. 

‍

Available Now in the Almanax security platform

The new model is now available in the Almanax security platform, providing developers with a powerful tool to proactively secure their code early during the development process and before their manual audits. By integrating state-of-the-art AI directly into the development pipeline, we aim to empower developers to catch vulnerabilities early, ensuring the security and robustness of their Web3 applications.

At Almanax, we are committed to pushing the boundaries of AI in Web3 security. This release represents a significant milestone in that journey, and we look forward to continuing to innovate in this space. In addition to releasing ALMX-1, we started working on the creation of new benchmark datasets with other industry players to address some of the limitations of the ones currently available. If you are interested in contributing to the effort, reach out to francesco@almanax.ai.

Stay tuned for further updates as we continue to enhance our tools to address the evolving security challenges of the Web3 ecosystem. 

Get access to ALMX-1 on our website. 

‍