This project was a collaboration between Rushi, a Data Engineer at Deloitte, and Rowan, an undergraduate student at University of Pittsburgh. With a shared interest in efficient deep learning, they set out to explore how compact neural networks can be enhanced through advanced fine-tuning techniques without relying on heavyweight architectures.

Their project used MobileNet, a streamlined convolutional neural network designed by Google for speed and resource efficiency. Experimenting with the CIFAR-10 dataset of 60,000 images across 10 classes, the goal was to see how far they could push classification performance using minimal compute.

The real game changer was Perforated AI’s dendrite based fine-tuning, which allowed them to optimize models far beyond their expected performance limits. On the MobileNet-Small model, they saw a clear performance gain of 6% relative improvement highlighting how effective Perforated AI's fine-tuning can be, even on models that are already highly optimized.

After seeing these accuracy gains, the team selected parameters to create a MobileNet-Extra-Small variant. Using this new architecture in combination with Perforated Backpropagation™ they brought the parameter count down from 2.54 million to just 1.66 million and still crushed the original performance, pushing the accuracy to 82.25%.

“Seeing a model that small perform that well was honestly surprising,” said Rowan. “It really changes the way we think about deploying AI on limited hardware.”

The technology was easily and quickly implemented, expanding on the base Pytorch training pipeline. After optimization, scaling up only required minimal code changes, drastically reducing barriers and allowing for seamless integration without system overhauls.

“For anyone building on a budget, or targeting mobile and embedded devices, this is a direction worth exploring,” added Rushi.

Their results prove that with the right tools, small models can deliver a big impact.