Gyuho Lee

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

Worth bookmarking this one!

25

Meta just dropped Llama 3 (70b) last week. Here's what you need to know. Trained on 15 trillion tokens of data. 😯 Llama 3 is leading the space in capabilities for math and reasoning. Scoring an 85 on the MMLU benchmark. (Massive Multitask Language Understanding) Compared to Claude 3's impressive 86.9 score. However, with 25 trillion less training tokens. Llama 3 definitely has the potential to exceed Claude 3. The most exciting development comes in the form of Meta's approach to AI. 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲. Llama 3's being released to developers, Along with a "Getting Started" guide, as well as new trust/safety features. - Llama Guard 2 A beefier version of their Llama guard content classification system. - Code Shield A screener for potentially malicious, or improper code outputs. - CyberSec Eval 2 A system for protection against prompt injections and code interpreter abuse. 💡 Benefits of Open Sourcing Llama 3 Llama 3 is comparable to GPT-4 in terms of capabilities. Tech hardware companies like Nvidia and AMD are supporting the model, As well as Cloud services like AWS, and Hugging Face. By it becoming publicly available to whomever wants to design with it. ✅We'll quickly have many GPT-4 models entering the market. The competition is on! - I'm really excited to announce Hank and I's collaboration with Christoph Rosenboom Keep an eye out next Monday! #llama3 #AI #LargeLanguageModel

17

19 Comments

LLMs are trained on internet-scale data. But to make AI work for work, models need to be grounded in your enterprise data -- a much harder problem than meets the eye! How do you connect to all your data sources and ingest them frequently enough while managing compute/storage costs? How do you make sure that you only show users results from documents they have access to? How do you rank results without internet-scale click and link data? How do you process the content of documents that are often hidden in PDFs and presentations? These are tricky problems, and it's why enterprise search didn't work for nearly a decade, until Glean came along. Turns out it's the exact same set of challenges needed to give LLMs the right context to work their magic on your enterprise content. Glean spent 5 years building the ingestion, search, and ranking infra needed to solve this problem well. This has given them the right to run the "Work AI" race and delight users at work. CEO Arvind Jain shares more on the latest episode of Training Data. Listen here, or wherever you get your podcasts: YouTube: https://seq.vc/ytd Apple: https://seq.vc/pit Spotify: https://seq.vc/jnz 

203

10 Comments

I am incredibly excited to announce Arch gateway https://lnkd.in/ePAf-VK8, an open source project for developers to build genai apps faster. It is engineered with purpose-built LLMs, Arch handles the critical but undifferentiated tasks related to the handling and processing of prompts, including detecting and rejecting jailbreak attempts, intelligently calling “backend” APIs to fulfill the user’s request represented in a prompt, routing to and offering disaster recovery between upstream LLMs, and managing the observability of prompts and LLM interactions in a centralized way - all outside business logic. Here are some additional key details of the project, * Built on top of Envoy and is written in rust. It runs alongside application servers, and uses Envoy's proven HTTP management and scalability features to handle traffic related to prompts and LLMs. * Function calling for fast agentic and RAG apps. Engineered with purpose-built fast LLMs to handle fast, cost-effective, and accurate prompt-based tasks like function/API calling, and parameter extraction from prompts. * Prompt guardrails to prevent jailbreak attempts and ensure safe user interactions without writing a single line of code. * Manages LLM calls, offering smart retries, automatic cutover, and resilient upstream connections for continuous availability. * Uses the W3C Trace Context standard to enable complete request tracing across applications, ensuring compatibility with observability tools, and provides metrics to monitor latency, token usage, and error rates, helping optimize AI application performance. This is our first release, and would love to build alongside the community. We are just getting started on reinventing what we could do at the networking layer for prompts. Do check it out on GitHub at https://lnkd.in/ePAf-VK8 Please leave a comment or feedback here or in GH and I will be happy to answer. #gateway #envoy #rust #genai #developers #ai cc Salman Paracha | Muhammad Ikramul Haq | Syed Anas Hashmi | Fahim ul Haq | Farheen Omar | Umar Farooq | Arslan Ahmad | Abdul Rafay Abbasi |

60

9 Comments

LLMs' capability follows a "bathtub curve" of usefulness. At the lowest level, they excel as a super-autocomplete, generating concise code snippets. Conversely, at the peak, they effectively convey high-level task-relevant concepts. However, in the middle ground, they struggle.

1

🚨🚨 🚨 Unlocking Innovation: The Power of the Adjacent Possible Innovation programs can stall when the team tries to ideate way beyond their collective perspective and frame of reference. Like a software engineer brainstorming genetic modification of mycelia, or a senior executive ideating process improvements which they haven't been participated in for decades. It's challenging, exhausting, and ultimately self-defeating. Enter the concept of the "Adjacent Possible" - a game-changer for innovation. What is it? The Adjacent Possible, coined by Stuart Kauffman, represents the realm of new ideas within reach based on our current knowledge, resources, and capabilities. Why it matters: ➡️ Realistic Innovation: it focuses on achievable advancements, not far-fetched concepts. ➡️ Incremental Progress: each innovation opens doors to new possibilities. How to leverage it: • Constantly explore the boundaries of your adjacent possible. What's in your line of sight is what you know best. • Combine existing ideas in novel ways. Even small perturbations from status quo can generate large innovation leaps. • Stay curious about developments in adjacent industries. Cross-domain patterns create fresh sources of ideas, while also expanding the adjacent possible. Most of all, it takes the 'genius' pressure off the team. No one knows everything, but each one of us knows a lot about something, and we can all drive plenty of innovation potential by focusing on the spaces we know best. Are you exploring your Adjacent Possible? Share your thoughts below! 👇 #Innovation #innovationsystems #AdjacentPossible

8

2 Comments

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

Last year, Huggingface deprecated their InferenceApi client and switched inference calls to InferenceClient. Simple enough. But see, Langchain's HuggingfaceHub package still called InferenceApi. So anyone running langchain.llms.huggingface_hub received an future warning error that InferenceApi was depracated and to use InferenceClient instead. The problem was that unless you went searching around into the details of the code langchain.llms.huggingface_hub package, you had NO idea what this error meant and whether it was an important warning to heed. So one would think Langchain would resolve this quickly, right? No. This was not resolved universally until a January 2024 update. Until then, people were using patches to get around this error. AND if you're thinking, "Wow! Well, glad that's no longer a concern" think again because out of curiosity, I asked ChatGPT 4o list imports for me to create multi-agent AI using langchain and inferencing the mistral API and wouldn't you know it, it suggested I use the deprecated InferenceApi client. Now, can you use the InferenceApi client and it still work? Sure. However, that future warning error gives lots of engineers who have not seen that error before pause AND it would likely be an issue when considering if one's code is production ready. When one AI company changes a client name and another AI company is using their own client to call on the original AI's client, this creates another dependency. One that a LOT of engineers building AI are not aware of unless they get used to seeing these errors over and over and over again. Little things like this are keeping great AI model demos from becoming great AI products.

45

6 Comments

Meta researchers develop ‘thinking’ LLMs  Meta researchers just introduced a new method called Thought Preference Optimization (TPO) to train large language models to ‘think’ before responding to general instructions — not just reasoning tasks. The details: →TPO prompts models to generate internal thoughts before responding to user instructions, similar to how humans think before speaking. →The AI's thoughts are kept private, with only the final answer shown to users with the AI using trial-and-error without direct supervision to optimize outputs. →TPO outperforms standard models on key benchmarks for non-reasoning tasks like marketing and creative writing but declines in math-related tasks. →The approach builds on the recent OpenAI ‘Strawberry’ research and o1 model release, which takes time to reason. Why it matters: Meta’s own Yann LeCun may scoff at the notion, but this method shows the potential of AI to ‘think’ for broader tasks than just math and reasoning. By allowing models to learn useful thought processes independently, TPO could also enable more capable and flexible AI assistants across a wider range of domains.

6

1 Comment

~40% of Frec’s customers are engineers working in tech, many of whom expect frequent and large capital gains. They also prefer to self-manage their money, which makes Frec an ideal platform. Meet Evan: an engineering leader who upgraded from investing in ETFs to direct indexing. Direct indexing appealed to Evan because it offers comparable returns to his ETFs while saving him money on advisory and ETF fees. “ETFs are true and tested, and giving folks a different way of investing in the market while saving more money is really appealing.” He’s used many other robo-advisors and other fintech tools, but “Frec seems to maximize tax loss harvesting opportunities better than any other product that I’ve used.” He also likes Frec's transparent fee structure.  “I feel like with a lot of other companies, that’s muddled in with the cost of investments,” he told us, “and you don’t really know how much you’re paying or how much you’re not saving as a result. I think Frec does a really good job of laying all of that out and giving you a true understanding of the cost of these investment products.” Read more about his Frec journey here ⬇

45

2 Comments

To all the founders about to quit and dreaming of a FAANG job or equivalent in pay & prestige, listen up: 1. What you're doing now-- trying to build & grow something new-- is the only "job" that AI and robots won't be taking. 2. The job you are dreaming about is already gone. Push through to fight another day. Be relentlessly resourceful as PG puts it. Your problem needs to be solved and no one is going to solve it for you. If you want help/advice on cutting expenses, sharing upside with community in exchange for build & grow collaboration, and raising capital from your users-- DMs open. There is no greener grass than the type that grows on PMF. Using Slyk IO to launch, grow, and raise fast puts you on the fast track to finding out. Your community can help you get there. I know because I'm doing this with Cadoo . #founders #startups #community

5

3 Comments

Nice highlight on Utah's tech cluster, aka Silicon Slopes. Not much silicon -- a lot of software, but a fine example of what can be done with persistent efforts, in this case beginning decades ago that I am aware of. My first business trip to SLC was in the 1980s and the potential was obvious. Interestingly, my first trip to NM was even earlier, and the potential was obvious here as well, but UT has done a much better job in diversification. In UT the LDS was a dominant feature, in some respects like the state in NM -- only game in town with deep pockets. UT was just much more business oriented and business friendly in their culture. The results speak for themselves. We covered UT in our VC firm during the 2000s (NM as well), during which time they were gaining momentum. Interestingly, UT is still under-served by local VCs when compared to the largest markets, though much healthier than NM. I hope UT never becomes like SV -- pricing locals out of the market, creating bubbles that require extreme predatory tactics to keep bubbles inflated, monopolistic behavior in tech giants, etc. Much better to learn from WA and CA's mistakes, and maintain a sustainable, diversified cluster with ongoing creative destruction and generational change in tech leadership.

1

1 Comment

🔥GITHUB SIGNALS 🔥 Out of all the signal sources we added, Github has the best APIs. Not surprised at all, but it's been fun to see their APIs paired with our best-in-class ID resolution engine. The result? Speedy lead lists. "Woah, the sync was so fast (sync). Thank you!! We're really excited to get going!" "Showing the info from stargazers is already very helpful, especially for those with personal email on GitHub profile." Been really fun seeing Netto Farah push the limits on what's possible - turning repos with 10k+ stars into a fully-enriched lead list in a matter of minutes! This is one of MANY signals we just added this week, check out the whole catalog here: https://lnkd.in/gzrhNZ9a

48

5 Comments

Curious about what we are working on at Kinship AI? Here's a sneak peek

26

Attached is NVDA's price chart for the last year - up c.175% (up 2500%+ in L5Y period!!) 🚀 I nonetheless have conviction that it currently remains undervalued... Among other things, due to strong competitive positioning as a GPU-maker, AI-driven demand fuelled by investor and consumer interest, and moats it has built. This is mirrored by equity analysts – GS’ latest price target was $800, BofA’s $1,500. I do believe the growth is capped, and driven in equal measure by under-supply and bubble-like demand as it is by fundamentals. Even so, Nvidia remains an attractive medium term investment. ⚖️ Demand – Supply Dynamics: AI is a generational innovation, and agents /apps are already upending SaaS, or at least so think VCs who deployed $64.6B in AI start-ups in 2023 alone. As early movers and now the industry standard (c.80% market share), Nvidia's A100 and H100 chips are not only the most efficient option, but also crucial for large scale AI training and inference (chips are the biggest cost head for LLM companies). With unprecedented interest in AI (ChatGPT acquired 1M users in 5 days of launch, OpenAI reported $3.4B ARR in Apr’24, AI agents are widely recognised as the tech of the future), Nvidia stands to capture a large portion of the $250-$500B data centre conversion market over the next 5 years​. This is proven by the $22.6B in data centre revenue Nvidia brought in during Q2’24, up 23% from the previous quarter and up 427% from a year ago 🥇Competitive positioning: While Intel, AMD, Arm have been active, the only credible competitor, AMD, has a relatively smaller market share in the high-performance AI GPU segment, and Nvidia’s consistent technological advancements keep it further ahead, e.g., Nvidia’s Rubin which will succeed Blackwell in 2026 is orders of magnitude better than anything on-market. 🤝Moats via ecosystem building: Collaborations with hyperscaler cloud providers like AWS and Microsoft Azure have expanded Nvidia’s reach. Moreover, acquisitions like Mellanox have strengthened its position in high-performance networking, critical to AI infrastructure. Coreweave, a prominent cloud services partner / distribution channel for chip capacity is an NVIDIA investee and is built almost entirely on Nvidia chips. Finally, they continue investing in networking, software tools, data centre solutions but also purpose-built AI PC and smartphones to create an ecosystem that not only locks in customers and enhances switching costs, but pivots more of their revenue to more predictable recurring contracts. 📈 Financial underpinnings: Nvidia’s expected FCF for next 2 years stands at $100B, providing huge ammo for R&D, strategic investments and M&A (only $30 to $35B earmarked for buybacks) and in turn, growth. Even so, their shares trade at a P/E ratio of c.20x  NTM earnings, far below historical median of 35x+, indicating the market expects them to grow in the medium term. KEEN TO HEAR THOUGHTS! #Nvidia #AI #Compute #Chips #Tech #Growth

10

1 Comment

🎯Data was, is, and will be the only fuel. Analogy with the fossil fuel is a good one, since the data we have today is as dirty as oil and coal. What big tech companies tried is to put a lot of "garbage" data, and what they did get was fantastic. Yet, we are at the point of diminishing returns. Data is the beginning and the end of this cycle.” Great thread, particularly this comment 👆from Ilija Lazarevic

20

2 Comments