KubeCon + CloudNativeCon India 2025

As PepsiCo continues to push the boundaries of AI-driven innovation, multiple 70-billion-parameter Llama model has been deployed within our Kubernetes-based AI platform, showcasing our ability to operationalize large-scale LLMs efficiently while optimizing performance, cost, and scalability.

This session will explore our journey of deploying and managing a high-performance LLM on Kubernetes. We’ll share insights on architectural decisions, GPU provisioning, and fine-tuning techniques for efficient inferencing. Attendees will learn how we tackled memory optimization, high availability, cost-performance balancing, and Responsible AI practices.

We’ll also discuss how our infrastructure, orchestration, and resource management evolved to meet large-scale inferencing demands, ensuring AI-driven innovation remains scalable, responsible, and efficient at PepsiCo.

Kueue and Volcano are leading the charge in orchestrating large-scale distributed ML jobs. But are they truly maximizing your GPU resources? Traditional batch scheduling methods often suffer from inefficient queue management, and rigid allocations that fail to adapt to real-time demand resulting in problems that scale with workloads.

This talk dives into how priority-aware queueing and elastic resource allocation can supercharge Kueue and Volcano, making batch scheduling more adaptive and efficient. We’ll break down the scheduler’s architecture, exploring how jobs dynamically move between priority queues, how elastic scheduling adjusts resource allocations in real time, and how these improvements lead to faster job execution and better GPU utilization.

Whether you're managing distributed training, hyperparameter tuning, or large-scale inference pipelines, this talk will provide the tools and strategies needed to unlock smarter scheduling and maximize ROI on Kubernetes GPU workloads.

Do you know about Large Language Models or LLMs? It’s been a while since we’ve seen a technology like this grab everyone’s attention, techies and non-techies alike. Ever since OpenAI’s ChatGPT burst onto the scene, powered by an LLM, AI has become a household name. But for newcomers, especially in the cloud native ecosystem, these concepts can feel overwhelming.
In this talk, we’ll start from the very basics, explaining what LLMs are and busting some common myths. We’ll break down the key terms with simple diagrams and analogies that are easy to digest. You’ll get a clear, beginner-friendly look at how LLMs work, why Kubernetes is perfect for running them, and how they fit together in the cloud-native world.
We’ll even walk through running an LLM on Kubernetes using vLLM for some practical insights and touch on how this opens the door to AIOps. If you’re curious about stepping into the AI ecosystem and understanding its cloud foundations, this talk is for you.

Imagine walking up to a self-service kiosk at an airport, hospital, or hotel, speaking in your native language, and instantly hearing a real-time translation—without cloud delays or privacy risks. Traditional speech translation relies on heavy cloud compute, but what if it could run directly on low-power edge devices like Raspberry Pi or even phones?
This session demonstrates how Kubernetes (k0s) enables AI-powered multilingual speech translation at the edge, eliminating latency, cloud dependency, and high operational costs. With a single k0s control plane, running outside of edge, managing hundreds of kiosks, AI models are deployed, updated, and scaled seamlessly. We’ll showcase a live demo of real-time speech translation running on edge devices, proving how edge-native AI can revolutionize automated customer interactions.
AI-powered customer service—automated, private, and built for scale.

This talk explores how KServe and OPEA are redefining the enterprise AI stack through modular and composable architectures. KServe delivers a scalable, hardware-agnostic Kubernetes native platform for production model deployment. OPEA complements this infrastructure with an open, multi-provider platform that provides optimized GenAI solutions through standardized interfaces. By building on KServe's capabilities, OPEA allows teams to focus on developing innovative applications rather than managing complex infrastructure concerns.
Join us to learn how deconstructing AI systems into specialized Lego blocks leads to more extensible and powerful enterprise AI architectures. We will demonstrate how this separation of concerns speeds up innovation, enabling teams to focus on app development for their app development. We will also discuss practical approaches for implementing this "building block" methodology in your organization's AI strategy.

Modern AI workloads rely on large GPU fleets whose efficient utilisation is crucial due to high costs. However, gathering telemetry from these workloads to optimise performance is challenging because it requires manual instrumentation and adds performance overheads. Further, it does not produce telemetry in a standardised format for commonly used visualisation tools like Prometheus.

This talk explores the potential of leveraging eBPF to capture CUDA calls made to GPUs, including kernel launches and memory allocations. Data from these probes can be used to export Prometheus metrics, facilitating detailed analysis of kernel launch patterns and associated memory usage. This approach offers significant benefits as eBPF imposes minimal overhead and requires no intrusive instrumentation. Our implementation is also open-source and available on GitHub.

Building ML systems that operate on real-time data streams is no easy feat, especially when dealing with complex messaging systems, scaling requirements, and the need for seamless inference. At Intuit, we saw firsthand how these challenges slowed down our ML teams and hindered innovation. That’s why we created Numaflow, a Kubernetes-native open-source platform that empowers teams to easily connect to streaming sources, apply transformations, and run inference at scale—without the typical overhead. In this talk, we’ll share how Numaflow enhances the developer experience, reduces boilerplate, and accelerates deployment of ML workflows. Whether you're a data scientist, ML engineer, or platform builder, this session will offer practical insights into running real-time inference on streaming data, the Intuit way.

As AI-powered cloud-native applications evolve, ensuring trust, compliance, and robustness requires dynamic governance mechanisms that operate seamlessly across distributed environments. This session introduces a multi-layered cloud-native framework that enforces AI guardrails at three critical stages: pre-processing (input validation), inference (real-time bias mitigation), and post-inference (output validation).

By leveraging Kubernetes orchestration, Istio service mesh, and knowledge graphs, the framework enables scalable AI governance that integrates multi-agent coordination, real-time intervention, and traceability to ensure AI decisions remain transparent, auditable, and aligned with compliance requirements.

Attendees will gain insights into cloud-native AI governance patterns, practical deployment strategies, and the role of multi-agent oversight in ensuring compliant, production-ready AI workflows within Kubernetes environments.

AI Agents are autonomously taking decisions, interacting with each other, and ensuring that the user specified deliverable is achieved. In lot of cases, AI Agents are dynamically generating the code to achieve the functionality. This dynamically generated code needs to be guardrailed i.e., an untrusted model could generate malicious code that will have equal access as that of the model itself. The aim of the talk is to create awareness of security issues sorrounding this use-case, explain the existing tooling/frameworks (such as executing in remotely hosted MicroVMs, use of WASM from NVIDIA). Explain the operational issues using such sandboxing mechanism and then put forth an approach leveraging LSM-BPF that combines the power of Linux Security Modules (LSM) with that of eBPF to achieve better sandboxing. KubeArmor, a CNCF Project, would be used to explain how this can be achieved.