This case study documents my journey into building a RESTful API to run AI model inference on a CPU-only setup.
My main motivation was to create a cost-effective solution for running AI models—particularly for chat applications—without relying on expensive third-party APIs.
The project was completed on my personal Mac (i5, 32GB RAM) and aimed to deliver real-time responses in under one second, without the need for specialised hardware like GPUs.
The Challenge: Balancing Speed and Hardware Constraints
The biggest challenge was ensuring that AI models could run efficiently on a CPU-only system.
Traditional AI setups typically require GPUs to achieve fast inference speeds, but I wanted to find an alternative that could work on more accessible hardware.
My first attempt involved using Llama 3.1 in the standard PyTorch tensor format, but this resulted in frustratingly slow response times—up to a minute for a single response.
I needed a solution that could meet my target of under one second.
After experimenting with different formats, I found success using the GGUF format with Q4 quantisation.
While Q6 provided better model quality, the increased size led to slower inference times, so I settled on Q4 as the best balance between performance and speed, allowing me to hit that crucial one-second mark.
My Approach: Choosing FastAPI and Navigating Technical Hurdles
To build the API, I chose FastAPI instead of Flask because of its native support for asynchronous programming and websockets, which would allow for better real-time interaction.
This choice proved key in ensuring that the API could handle multiple requests smoothly, especially for chat applications.
One of the initial technical hurdles I faced was getting the model to load into memory before it could handle requests.
I implemented a pre-loading mechanism to “warm up” the model, ensuring that it was ready to infer immediately once a request came in.
However, the trickiest part was handling the model’s end-of-response tokens.
The GGUF format fragmented the <|eot_id|> token into multiple parts, which made it difficult to detect when the response generation was complete.
This was critical because if the end token wasn’t detected, the model would continue generating tokens unnecessarily, impacting performance.
After some troubleshooting, I built a custom logic to detect the fragmented token sequence, which allowed the system to stop generating tokens at the right point.
The Outcome: Meeting Performance Goals and Deployment
After resolving these issues, the API performed as I had hoped. I was able to achieve:
- Response time: Under one second for input sizes up to 200 tokens, making the system fast enough for real-time chat applications.
- Deployment: I successfully deployed the API on both my local Mac and a Digital Ocean droplet using Docker, ensuring portability across different environments.
This project proved that AI models can be run effectively on CPU-only setups without sacrificing performance or incurring significant costs.
For me, the key takeaway was finding the right balance between model optimisation and system constraints—choosing the GGUF format with Q4 quantisation was a game-changer.
Reflections and What’s Next
Working on this project was an invaluable learning experience, especially as it was my first time diving into AI-powered chat models.
One of the main reasons I took on this challenge was to explore alternatives to using expensive APIs from providers like OpenAI and Anthropic.
By building my own system, I can now use AI models freely and share them with others who might not have the budget for high-cost services.
Looking ahead, I plan to enhance this API by incorporating memory techniques, such as those available through Langchain, to maintain conversation history and improve interaction quality.
While that wasn’t part of this case study, it’s an exciting avenue for future development.