Certified AI/ML Pentester Review

As organizations race to adopt the latest in artificial intelligence (AI), I was looking for training that addressed its unique attack surface and tested my skills in exploiting large language models (LLM). That’s when I discovered The SecOps Group and their Certified AI/ML Pentester, which was one of the first certifications (I’ve seen*) to offer a practical exam on LLM exploitation.

This post outlines my experience passing the C-AI/MLPen and provides helpful resources used to navigate The SecOps Group’s new “exam-only” training model.

Preparation

Initially, I was thrown by The SecOps Group’s new exam-only approach, removing formal training and requiring 100% self-study — especially when navigating a complex topic like LLM exploitation. However, the Exam Syllabus listed each topic covered, along with links to resources from industry leaders in AI such as Microsoft, Nvidia, and IBM.

I spent a significant amount of time brushing up on advanced exploitation techniques such as direct/indirect prompt injection, training data manipulation, RAG poisoning, and vulnerabilities in LLM design. Most of my effort, however, were focused on practical challenges using platforms like Gandalf, Immersive Labs, and Crucible.

Through each CTF challenge, I concentrated on “why” rather than “how”; evaluating the models core functionality and identifying preventative controls such as output generation rules or input/output filters. I equated this to “reconnaissance” in a standard pentest methodology before moving onto “exploitation”. This made it much easier to craft jailbreaks, exploit excessive functionality, and extract sensitive information once I knew the extent of the LLMs current instructions.

I finished studying with SecOps Mock Exam, which contained two challenges and used the same format as the real exam. Anyone looking for a good challenge should checkout level 2, which introduced a chatbot named Ultron communicating only in emojis and prone to hallucinations! 🌍💀🦠

The Exam

Overall the exam was super straightforward. Students are given 4 hours to work through 8 different AI models and extract a secret key. Submit these keys in the exam portal and click submit when done… no reporting, notes, or other documentation requirements.

Each AI model in the exam has a specific point allocation and students are required to get 60% or higher to pass — meaning you should be safe with at least 6/8 challenges but pay attention!!

Conclusion

Overall, I had a great experience with the C-AI/MLPen and would recommend it for anyone looking to expand their knowledge in AI/ML security. It was a great intermediate level primer and I’m excited to see other trainings on the subject.

Full disclosure - The SecOps Group provided me with an exam attempt to review the course. However, my experience was largely positive and I would recommend it regardless! The C-AI/MLPen provided a lot of fun challenges and, with ongoing discounts, can be found for only ~$65!