Expected Behavior Spring AI should support Jlama as a model provider, allowing applications to run LLM inference directly within the JVM using Spring AI's abstractions. Example configuration:
spring:
ai:
jlama:
model: tjake/Llama-3.2-1B-Instruct-JQ4
working-directory: ./models
Current Behavior Currently, Spring AI primarily supports external API-based models. While these are valuable, they don't address use cases requiring complete data privacy and internal processing. The current version doesn't support Jlama, preventing usage of this Java-native LLM engine within Spring AI's ecosystem.
References - Project: https://github.com/tjake/Jlama - Creator: @tjake
Context - Internal LLM processing is crucial for enterprise clients with strict data privacy requirements - No external API calls or data transmission required - No subscription costs for API-based services - Already proven viable through Langchain4j integration - Jlama supports multiple modern models (Llama 3+, Gemma2, Mistral, etc.) - Distributed inference capability for larger models - Native Java implementation optimized via Vector API - Active development and community support
Alternatives Considered 1. Continue using Langchain4j integration 2. Implement custom wrapper around Jlama 3. Use external API-based models with data filtering
Current Workaround Currently using Langchain4j's Jlama integration, but prefer Spring AI's ecosystem and abstractions for better Spring Boot integration.
Comment From: tzolov
Hi @LiveNathan , this will be a valuable capability to add. Would you be interested to give a try?
Comment From: LiveNathan
Would you be interested to give a try?
Yes! You mean test the implementation? Of course.
Wait, do you give it a try, as in building it?? Oooof, wow, that seems years beyond my skill level. I'd really out to @tjake about that.