Elastic Accelerates Jina Model Inference up to 8x with NVIDIA RTX GPUs

Elastic Accelerates Jina Model Inference with NVIDIA RTX Pro 6000 GPUs

Elastic has achieved significant performance improvements in Jina models using the new NVIDIA RTX Pro 6000 Blackwell Server Edition GPUs, with up to 8x speedups compared to previous NVIDIA L40-based solutions. Initial tests show substantial reductions in latency times for embedding, reranking, and OCR, critical components for search and agentic AI applications.

Quick Answer

Technical Performance Details

The tests conducted by Elastic highlighted significant improvements in various usage scenarios. For embedding, the 99th percentile latency dropped from 5.86 seconds to 0.73 seconds, representing an approximately 8x improvement. In reranking, the 90th percentile latency decreased from 732 milliseconds to 171 milliseconds, with a 40% increase in the average request volume.

A particularly relevant use case was the handling of large reranking workloads. In a controlled test, the completion time went from 702 milliseconds (using three L4 GPUs) to 160 milliseconds (with a single RTX Pro 6000 GPU), a 4.4x improvement. Even more significant, when small reranking requests were run alongside large jobs, the latency dropped from 708 milliseconds to 66 milliseconds, a 10.7x improvement.

Mixed Workload Management Capabilities

One of the most critical aspects evaluated by Elastic was the new GPUs' ability to handle mixed workloads without degrading performance. While Jina models are compact, they often need to share resources with large jobs, which can cause significant latency. With the RTX Pro 6000, Elastic observed that small requests remained fast even when large jobs were running simultaneously on the same service.

Improvements in OCR

OCR is another area where the new GPUs showed significant improvements. With the jina-ocr-v1 model, the time for the first output token dropped from 0.50 seconds to 0.20 seconds, and the page conversion speed increased from 454 to 1,671 pages per hour, a 3.7x improvement. This is particularly important for processing complex documents containing tables, formulas, and intricate structures.

Benefits for Customers

For Elastic's customers, these improvements mean that queries do not slow down due to large jobs running on the same service. The ability to handle mixed workloads without performance degradation is crucial for search and agentic AI applications, where latency can significantly impact the user experience.

Jina Models: Leadership in Their Class

Jina models are known for their precision and coverage, especially in their size class. Jina's Reader and OCR models convert complex documents and websites, including images, tables, and formulas, into clean text with HTML markup and LaTeX for formulas. The jina-ocr-v1 model is the leader in its class for under 600M active parameters, outperforming the olmOCR benchmark.

Jina's embedding models convert content into a multilingual and multimodal embedding space, allowing a single query to reach all content and return results on a single confidence scale. The jina-embeddings-v5-omni-small model is the best-performing openweight omni model under 2B parameters in the MTEB benchmark.

Reranking with Jina

Jina's rerankers re-rank results against the original query, highlighting the best matches. The Jina-reranker-v3.5 model, with 600M parameters, outperforms the Qwen3-Reranker-4B model with 4B parameters in the STaRK benchmark for structured data.

GPU Capacity Considerations

Although Jina models are compact, the GPU capacity required for a workload depends not only on the model size but also on the mix of requests. Jina API and EIS services are shared, meaning large jobs can run alongside smaller requests on the same GPU, potentially causing significant latency. The new RTX Pro 6000 GPUs have demonstrated the ability to effectively handle these mixed workloads without degrading the performance of smaller requests.

Documentation and Resources

For more information on Jina models in the Elastic Inference Service, you can consult the official documentation. To try the Jina API, visit the Jina AI website.