Huazhe Wang

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

112

6 Comments

The high-end graphics cards from Nvidia, such as the H100, have catalyzed the emergence of dedicated Rent-A-GPU farms, expanding beyond traditional hyperscale providers like AWS, GCP, and Azure. Initially, supply chain limitations coupled with high demand enabled these specialized farms to purchase these cards—each costing nearly $40,000—in bulk and offer them for rent. By late 2023, rates were around $8 per hour under commitments ranging from one to three years. Today, the landscape has shifted significantly. Some of the largest Rent-A-GPU providers have adapted to changing market dynamics by offering GPU spot markets with commitments as short as 48 hours, with rates as low as $1.8 per hour or even less. This suggests that the supply chain for high-end GPUs has finally caught up with demand, potentially heralding a substantial decrease in costs not only in the rental market but also in capital expenditures for owning these GPUs. For the first time, there is downward pressure on the prices of high-end GPUs. This development is particularly promising for real-world applications of Generation AI. The significant reduction in inference costs is crucial for the viability and value creation of GenAI within enterprises. It represents the first of many indicators that we are approaching the rapid realization of game-changing use cases, transforming how businesses leverage advanced AI capabilities. At KamiwazaAI we believe the start of an 5th Industrial Revolution Enterprise #5IR is around #1trillionInferences a day or in the range of ~$3b dollars a day utilizing chat GPT4 at .003 per Inference (100 tokens). A 1000x decrease in the cost of Inference is only 18 months away or so and its time NOW to begin implementing the technology to keep up!

21

5 Comments

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, Huaming Chen University of Sydney 2024 https://lnkd.in/gVZ3r8G7 "As LLMs have demonstrated significant capabilities in tasks such as code generation and vulnerability detection, researchers are increasingly interested in their potential applications. However, the study highlights the limitations of LLMs, including their lack of autonomy and self-improvement, which LLM-based agents aim to address. These agents represent a promising advancement towards achieving Artificial General Intelligence (AGI) by integrating LLMs for decision-making and action-taking. Despite the growing body of research on LLMs in software engineering, there remains a lack of clarity distinguishing between LLMs and LLM-based agents, as well as a need for standardized benchmarks to evaluate these technologies. The survey systematically investigates current practices and solutions related to both LLMs and LLM-based agents, focusing on six key areas: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. By reviewing and contrasting the contributions of LLMs and LLM-based agents across these topics, the survey provides insights into their respective tasks, benchmarks, and evaluation metrics. It also discusses the models and benchmarks utilized in this domain, offering a comprehensive analysis of their effectiveness and applications in software engineering. Ultimately, the authors aim to illuminate the path forward for LLM-based agents in software engineering, paving the way for future research and development in this promising area."

2

Get a deeper look into the results submitted by NVIDIA in #MLPerf Inference v4.1, highlighted by the first ever submission using the NVIDIA Blackwell architecture, increased performance using the NVIDIA H200 Tensor Core GPU, and more.

1

Huge release by Daekeun Kim and Hyo (HK) Choi from our AI GBB team in Korea, sharing Korean language benchmarks for our latest Azure OpenAI model, gpt-4o-mini - like gpt-4o, the model excels on Korean language tasks, in this case mini completely eclipses gpt-35-turbo and nearly achieves gpt-4-turbo performance. The benchmarks and code are open source, extensible beyond our Azure models, so we encourage our customers and the larger community to try it out and even fork/share your own findings - and please let me in the comments what other Asia languages beyond Korean you may be looking to evaluate with your language models.

52

7 Comments

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

27

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,897

80 Comments

Seeking a full time onsite hybrid AI Architect Engineer Manager for leading RAG and LLM Generative AI role to relocate and lead RLHF, prompt engineering for zero shot learning, one shot learning, few short learning. Engineer Architect or Manage and lead teams on Fine Tuning LLM models on PEFT, RLFL, Prompt Tuning context window, and to work on Reward model using LoRA, and to work on agent to instruct LLM on Context window and lead AI Generative LLM RAG performance for scalability. Engineer Architect or Manage and lead Generative AI LLM Lang Chain Use cases, and work on Hugging faces for encoded/weights. Engineer Architect or Manage and contribute to building RAG Retrieval Augmented Generation to retrieve external library data for tailored to specific models domain in prompt analysis. Additional talents that I have which include People first partnerships, deliver on AI architect accountability and ensure focus on diversity inclusion for delivery business value. Manage stakeholders data projects to improve business revenue. People talents that I offer are people loving manager who is humble and offers critical creative innovative AI thinking with the team. Engineer Architect or Manager the architecture for AI Lang Chain Generative AI Use Cases that need LLM text words. Contribute as Engineer Lead on improved production pipeline supply chain signal processing and be the Engineer Architect for large language text analysis (LLM). Research for teams can be more productive in using Lang Smith for LLM large language models. Assist on curating sandbox to develop bidirectional decoder/encoder transformer (BERT) masking language next sentence (MLM). Lead again on OpenAI GPT for optimized performance to build chat what-if analysis for the business. Manage Engineer to lead scaling the team on development with performing LLM Chain and predictions models and build predict messages that pull from the Hugging Face Hub objects and prompt engineering with prompt templates and human message responses.

3

Brand new paper on September 5th, 2024 on Mamba and Physics-informed machine learning. State-space models are accurate and efficient neural operators for dynamical systems Zheyuan Hu, Nazanin Ahmadi Daryakenari, Qianli Shen, Kenji Kawaguchi, George Karniadakis arXiv: https://lnkd.in/gnVbq5rm Physics-informed machine learning (PIML) has emerged as a promising alternative to classical methods for predicting dynamical systems, offering faster and more generalizable solutions. However, existing models, including recurrent neural networks (RNNs), transformers, and neural operators, face challenges such as long-time integration, long-range dependencies, chaotic dynamics, and extrapolation, to name a few. To this end, this paper introduces state-space models implemented in Mamba for accurate and efficient dynamical system operator learning. Mamba addresses the limitations of existing architectures by dynamically capturing long-range dependencies and enhancing computational efficiency through reparameterization techniques. To extensively test Mamba and compare against another 11 baselines, we introduce several strict extrapolation testbeds that go beyond the standard interpolation benchmarks. We demonstrate Mamba's superior performance in both interpolation and challenging extrapolation tasks. Mamba consistently ranks among the top models while maintaining the lowest computational cost and exceptional extrapolation capabilities. Moreover, we demonstrate the good performance of Mamba for a real-world application in quantitative systems pharmacology for assessing the efficacy of drugs in tumor growth under limited data scenarios. Taken together, our findings highlight Mamba's potential as a powerful tool for advancing scientific machine learning in dynamical systems modeling.

98

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

31