The optimized setup delivered measurable improvements across several workloads. A 32×A10 training job achieved 96.3% average GPU utilization, selected inference and fine-tuning workloads reached 95% to nearly 100% GPU utilization, and an embedding-generation pipeline completed approximately 11% faster than the previous baseline. One vision-language fine-tuning workload was reduced from 3h 5min to 36m 4s.
Meet our client
Client:
Industry:
Market:
Technology:
In a Nutshell
Client’s Challenge
The client needed to migrate training and inference workloads to a managed Ray environment to improve performance and resource utilization. Existing configurations created challenges around multi-node execution, memory pressure, object store behavior, disk spillage, and GPU underutilization, particularly as workloads and data volumes increased.
The objective was not simply to move jobs to a new platform, but to establish a more reliable execution setup that could support demanding workloads across both production and experimental settings.
Our Solution
deepsense.ai migrated selected workloads to distributed execution and optimized the underlying execution and data layers. The team introduced clearer separation between head-node and worker responsibilities, RayJobs, multi-node DDP, and more precise CPU, GPU, and memory allocation.
We also tuned Ray Data pipelines, preprocessing parallelism, input block sizes, DeepSpeed parameters, CUDA memory settings, and inference actor allocation. These changes reduced memory pressure and data-processing bottlenecks while improving accelerator utilization and workload stability.
Client’s Benefits
The optimized environment supported stable execution of a 32×A10 training workload at 96.3% average GPU utilization, while selected inference and fine-tuning jobs achieved 95% to nearly 100% GPU utilization.
An embedding-generation workload completed in 40m 21s, approximately 11% faster than the previous baseline, while a vision-language fine-tuning workload improved from 3h 5min to 36m 4s. The work also eliminated observed disk spillage in the optimized paths and created a stronger foundation for scaling and optimizing additional workloads.
A Deep Dive
Project Context
The engagement was delivered as a short, intensive team augmentation project between March and April 2026. deepsense.ai worked directly on selected ML and LLM workloads, focusing on migration, stabilization, and performance optimization rather than major changes to the underlying model logic.
The scope covered four closely connected areas: migrating training and inference jobs to the managed Ray environment, stabilizing data pipelines and execution configurations, improving GPU and memory utilization, and preparing documentation and handover materials to support continued development after the engagement.
This required looking beyond individual model configurations. In distributed AI workloads, overall performance depends heavily on how compute resources, data movement, preprocessing, memory, orchestration, and multi-node execution work together.
Scaling Distributed Workloads Without Introducing Instability
Moving demanding AI jobs into a multi-node setup introduced several infrastructure and execution challenges.
Some workloads required changes to the distribution of resources between the Ray head node and workers. Others encountered memory pressure as large data blocks moved through preprocessing and inference, increasing object store usage and creating disk spillage. At the same time, GPU availability alone did not translate into consistently high accelerator utilization: inefficient data feeding, resource allocation, and execution parameters could leave expensive compute capacity underused.
The challenge therefore had three interconnected dimensions:
- making training and inference jobs work reliably across distributed GPU infrastructure,
- controlling memory usage and data flow as workloads scaled,
- and ensuring GPUs remained consistently utilized rather than waiting for data or other parts of the execution pipeline.
Optimizing any one of these areas in isolation would have left bottlenecks elsewhere in the system.
Optimizing the Execution Layer, Data Flow, and Resource Utilization
deepsense.ai addressed the problem at the infrastructure, execution, and pipeline levels.
Distributed execution and workload migration.
The team adapted selected training workloads for efficient multi-node execution, introducing clearer separation between head-node and worker responsibilities and using RayJobs to make execution more structured. Multi-node DDP configuration and workload-level resource allocation were tuned to better match the characteristics of the available GPU infrastructure.
Data pipeline stabilization.
Ray Data parameters were adjusted to reduce the impact of large blocks and uneven processing. The team optimized read and preprocessing parallelism, reduced input block sizes where appropriate, and limited unnecessary object growth and duplication in memory. These changes reduced pressure on the Ray object store and made data delivery to compute-intensive stages more predictable.
GPU and memory optimization.
Training and inference workloads were tuned to make more effective use of accelerator capacity. The work included adjustments to micro-batching, DeepSpeed configuration, CUDA memory settings, CPU-to-GPU resource allocation, and inference actor oversubscription. Together, these changes helped keep GPUs supplied with work while reducing memory-related instability.
Importantly, many of the improvements came from configuration, infrastructure, and pipeline changes rather than deep modifications to model code. This allowed the team to address system-level bottlenecks while limiting unnecessary changes to the workloads themselves.
Measurable Performance and Stability Gains
The optimization work produced measurable gains across training, inference, embedding generation, and fine-tuning workloads.
Selected results included:
- A large training workload was migrated to a 32×A10 configuration and achieved 96.3% average GPU utilization.
- An embedding-generation workload completed in 40m 21s, approximately 11% faster than the client’s previous baseline, with 91.4% average GPU utilization.
- A multi-parameter inference workload completed in 59 minutes with 95% average GPU utilization.
- A vision-language SFT LoRA workload improved from 3h 5min to 36m 4s.
- Two additional SFT LoRA workloads completed in 9m 14s and 7m 44s, respectively, with GPU utilization close to 100%.
The optimized inference paths also ran with no observed disk spillage, addressing one of the stability issues identified during the engagement.
Beyond the individual benchmarks, the project established a more predictable approach to running distributed AI workloads: clearer resource allocation, better-controlled data movement, more stable memory behavior, and consistently high GPU utilization.
The engagement concluded with documentation and a technical handover, providing the client with a foundation to continue optimizing existing workloads and applying the same principles to additional jobs over time.
