Yuanjing Shi

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Comment

Deploying GenAI model inference or generation (as Jensen calls it) can come with significant challenges: - Excessive latency - High cost per million inferences - Low throughput - Accuracy drops You might be tempted to apply brute-force solutions like requesting more NVIDIA GPUs or, even, abandoning your GenAI use case. My suggestion? Take a deeper look at Anklet’s post below, where he details some of the techniques you can use to optimize inference using your existing NVIDIA GPUs and models. You may always consider NVIDIA NIMs to simplify the process even further. Whichever path you choose, we're here to help. Ping me for more details, and feel free to share and comment below.

10

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

I’m excited to announce the release of LLM Compiler Pro, an extension of the paper titled “An LLM Compiler for Parallel Function Calling”. This framework addresses complex problem-solving scenarios that require consideration of various dependencies. - 📋 Handle subtasks generated through plan decomposition with function calling. - 🔌 Enable integration with a wide range of APIs and tools beyond simple Python functions. - 💬 Provide an interactive conversational interface. - ⚡ Ensure users do not experience waiting by offering various streaming interactions. - 🚀 Maintain the fast processing speed of the original LLM Compiler. - 🛠️ Comprehensive refactoring and migration of deprecated modules from Langchain and OpenAI. 😀 Feel free to visit GitHub repository to check out the project’s architecture and demo! Github: https://lnkd.in/gcA9XZmk It currently have a fully functional GUI demo, and the implemented tools include the SerpAPI Google Engine and a web browsing tool utilizing Selenium. Future plans include creating report-style documentation and ongoing project maintenance. Contributions are always welcome, and I highly appreciate any feedback, even if it’s critical! I would also like to express my gratitude to the researchers for their excellent work, which has inspired this exciting side project. Thanks a lot, Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael Mahoney, Kurt Keutzer, Amir Gholami

83

7 Comments

Explore the trade-offs between throughput and latency in #deeplearning models in the new self-paced #course, Sizing LLM Inference Systems. Learn how to measure and optimize performance for on-prem and cloud deployments. 🌟 Take the new course for free for a limited time. Enroll now: https://nvda.ws/4fomyod #NVIDIATraining #DLI #LLM

5

1 Comment

Discover how Perplexity AI built AI architecture to scale its Llama 3 #LLM deployment to serve millions of inference requests. Key takeaways: ✅ 435M queries/month ✅ 20+ AI models, including Llama 3 (8B, 70B, 405B) ✅ 3x cost reduction

7

In this lightning talk at AWS re:Invent 2024, explore how to enhance Apache Spark using Amazon EC2 G6 GPU instances, demonstrating speedups of up to five times and cost reductions of 80% on Spark operations. bit.ly/3VpcBPI

2

Discover DocETL—an open-source, low-code AI system developed by UC Berkeley researchers! This cutting-edge tool utilizes LLMs to tackle unstructured data processing with ease and efficiency. 🚀📊 #AI #dataprocessing #lowcode

8

Excited to announce the public release of the Sortformer model! 🎉 This innovative approach to speaker diarization addresses the limitations of traditional methods, offering seamless integration with ASR and LLMs, efficient token-level training, and a sort-loss mechanism to resolve the speaker permutation problem. No more timestamp-based training for speaker diarization!  Lightweight and highly adaptable, Sortformer simplifies diarization integration into neural architectures 🚀 🎧 Check out this 10-minute NotebookLM podcast explaining Sortformer in detail: https://lnkd.in/gX8S6fPA Paper: https://lnkd.in/gCvPg45C Model weights: https://lnkd.in/ga9SQX6P Code: https://lnkd.in/gB5J2r37 Tutorial: https://lnkd.in/gaH4zhex

49

2 Comments

2025 Prediction: Thanks to advancements in KV cache optimization techniques and the expansion of context size in LLMs, we will witness the rise of "LLM Cache-Augmented Generation". This innovation will start to replace traditional Retrieval-Augmented Generation (RAG) methods, redefining how LLMs handle memory and external knowledge. 🚀

32

💡 Do you know that Hugging Face recently added an option to use open-source LLMs with NVIDIA NIM Serverless Inference? https://lnkd.in/gaZz9E8p Why this matter❓ This is a quiet revolution for small organizations looking to build Gen AI applications without breaking the bank. 💸 Affordable Access: Powerful AI without the hefty infrastructure costs. 👯 Democratized Innovation: Small teams can harness the same cutting-edge AI as the big players. 🧠 Innovation for All: Open-source models are now within reach, letting small teams innovate on a whole new level. 🚀 Bonus 🎯🎯🎯 - If you’re working with Dataiku, you can tap into this potential effortlessly using the NVIDIA NIM Plugin. A subtle shift could make a big difference in your organization's AI journey. NVIDIA NIM Plugin - https://lnkd.in/gie8Ah3s #HuggingFace #NVIDIA #NVIDIAAI #Dataiku #ServerlessInference #GenAI

16

It's interesting to watch. torchtitan is a proof-of-concept for Large-scale LLM training using native PyTorch. It is a repo to showcase PyTorch's latest distributed training features in a clean, minimal codebase. torchtitan is currently in a pre-release state and under extensive development. https://lnkd.in/evNebqu9

36

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

15