Archit Sharma

Celebrity LLMs unable to count the number of ‘r’s in ‘strawberry is a nice reminder that they do not think like humans do, and that perhaps a refresher of the concept of tokenization is due. Tokenization is a process used by LLMs to break down text into smaller chunks called tokens. LLMs cannot just process raw text. Tokens can be words, characters, spaces or subword units used in a prompt, depending on the tokenization strategy used. Behind each token is a unique numerical identifier. If the model's primary function is understanding and generating natural language, not performing precise calculations, it understands the task at a conceptual level rather than a computational level. For counting tasks, it might rely on patterns from training data, leading to incorrect counts. The prompts for counting the number of ‘r’s in "strawberry" likely treat the word as one single token i.e. the model doesn't directly see each individual character. Rather, it sees "strawberry" as a conceptual whole. If "strawberry" is treated as a single token, the model might use context and patterns it has learned to approximate an answer. This can lead to errors like undercounting the occurrences of 'r'. For an LLM, generating an answer typically involves predicting one token at a time in a sequence until it completes the output. Each generated token influences the prediction of subsequent tokens. The model predicts the next token (word) in its output sequence by assigning probabilities to all possible tokens in its vocabulary. The token with the highest probability is typically chosen as the next word, but depending on the decoding strategy, other tokens might also be considered. For example, in the case of answering the strawberry prompt, when the model is at the point where it's about to state the number, it will consider all possible numerical tokens ("1", "2", "3", “4”, “5”, etc.) and assign probabilities to them. If it sees that saying "2" leads to a coherent response based on its training data, it might stick with that. Tokenization is a foundational aspect of how LLMs understand and generate text. When tasked with precise operations like counting, their responses may reflect general language understanding rather than exact computational accuracy, especially if the input isn't tokenized in a way that facilitates character-level processing. The use of probabilities allows LLMs to generate varied and contextually relevant responses that are, as per their training, coherent and meaningful even in the face of complex or vague queries. LLMs don't think like humans. They have a bunch of numbers behind the scenes working to answer your prompt.

8

9 Comments

Large language models are wonderful at ingesting large amounts of content & summarizing. Uploading an academic paper and I can pester it with an infinite list of questions & it will respond with equally infinite patience. In comparing the two most recent Microsoft earnings calls, Claude highlighted: - faster than expected Azure growth (29% vs 27%) - AI contributing 8% of Azure revenue up from 7% - higher CapEx spending & greater capacity constraints for data centers - much better commercial bookings growth : 17% vs 13% Excellent analysis. Benn Stancil described LLMs as great averagers of information. But what if I don’t want the average? If I seek the data point two standard deviations out? If I’m channeling my inner Anthony Bourdain & I’m seeking the fermented shark or cobra heart within my query? At Google, we ranked web pages & ads with many signals. An engineer taught me that of them found rarity of a word across a set of documents. For example, if there’s a document containing 10 instances of the word ghotiacross a collection of documents, where the average document has 0 ghotis, that document is likely the best to answer a search about fish. I haven’t found a way to goad an LLM to produce the rare result. Why did one user decide to use a particular piece of software when 10 others did not? Maybe I haven’t learned how to prompt an LLM well. Ideogram launched a feature called Magic Prompt that expands a basic prompt into richer instructions that elicit better results, marrying the language of a user & a computer. A user might write : A boy with a dog in a park. Magic Prompt replies : A heartwarming scene of a young boy playing with his happy, wagging dog in a lush, green park. The boy has a playful smile on his face while he tosses a bright yellow ball for his furry friend to fetch. The park is filled with beautiful flowers and trees, creating a serene atmosphere for this bonding moment between boy and dog. Magic Prompt surprised me in three ways. First, unlike Google search where terse queries tend to work very well, LLMs relish verbosity. Less Hemingway & more Faulkner. Second, maybe there’s a prompt for rarity. If you know what it is, please share with the rarity shibboleth. Third, I wonder what other prompt techniques are out there that have yet to be discovered to wrest the best from an LLM. This technique is called TF-IDF. It means Term Frequency Inverse Document Frequency.

21

6 Comments

https://lnkd.in/esZgE24d Seven years ago I gave this talk at IIT Kanpur. Back then it seemed inevitable that we would have self-driving cars by now, and some predictions I made were wrong. Overall, though, I think this talk has aged reasonably well. Little did I know back then that the reference to the 2016 MaxSAT I gave would spark the creation of a new company, InsideOpt. The Seeker Solver that we sell is based on the technology outlined, and it is changing the optimization landscape. On the social implications, we since edited a special issue of the AI Magazine on the social disruption that AI brings. More work is needed. We need to have a wide societal discourse on how we are going to share the increasing wealth that AI brings to the world. #insideopt.com

8

India is not just catching up in the AI race; we're innovating and leading in ways that are uniquely ours. The Economic Times recently highlighted how our approach to Generative AI goes beyond just Large Language Models (LLMs), addressing the specific challenges and opportunities within our diverse market. Rahul Agarwalla, Managing Partner at SenseAI Ventures, shared his invaluable insights into this AI revolution. His leadership and vision are truly driving innovation in the field, and it's an honour to be part of this journey with him. For anyone interested in the developments in AI and how India is making its mark on the global stage, this is a must-read.

46

3 Comments

Great course by Andrew Ng and Shreya Rajpal on guardrails. And the best model on Guardrails hub for minimizing hallucinations is none other than our Bespoke-MiniCheck model. It is better and cheaper than frontier models. Get started with it today: https://lnkd.in/gKzciipz PS: If you are wondering how we beat GPT-4 or other frontier models for this task, it's basically cool synthetic data creation. We are cooking a library so you can also fine-tune models that beat frontier models on your narrow task. DM if you are interested in testing an early beta.

39

Edge AI is the future! Are we prepared for the data privacy tornado that brings? Every device will have AI compute power and this will open up a new horizon for consumer (and business) applications. But who can reliably own the data? Businesses should build their own models now. #AIOwnership #DataPrivacy #EdgeAI

7

What should we be aware of when using LLMs? Google just released Gemma 2 to developers and the community: https://lnkd.in/gBTXZaZ2 This is Google’s next version of their open-weight LLM (different from Gemini their closed LLM). There have been great posts already about what Google has done differently with this model, but I wanted to highlight another part of this release. In the paper (https://lnkd.in/gYvS-Q_w) Google published along with Gemma 2, it covers the technical design of Gemma 2, but it also addresses 4 problems LLMs still face. 1. Catastrophic forgetting With an imbalance between the amount of data used for pre-training LLMs and fine-tuning LLMs, there are common cases where fine-tuning loses aspects from the initial data and "forgets" it in their responses. 2. Excessive verbosity When rewarding models for better structure in their answers to questions, this tends to deviate the entire LLM when having casual conversations. 3. Refusing to answer A problem that stems from the question: is no information better than incorrect information? 4. Exploitation *Note: this is actually from a different paper Google references: https://lnkd.in/gRigUZpt Policies dictate how LLMs learn using a reward system. But these systems, even if “perfect,” can be exploited using adversarial means. When we know how LLMs are designed to answer, we can find ways to exploit them. - I believe with more intelligent data and smaller language models, we can begin to address these issues!

7

An alternative view on 'AI for good' by Payal Arora describing what I refer to as the tyranny of good intentions, in this case regarding AI, though as Payal discloses, intentions are not always as good as they claim to be. I've experienced this several times when we either hired outsourcing firms claiming to provide 'AI for good', or we were asked by others to assist for altruistic endeavors, including by some of the largest and wealthiest organizations in the world. What I've found is other agendas are usually lurking beneath the surface, including age-old greed and lust for power in NGOs, governments and companies. I've concluded that we can contribute the most to society by providing more efficient and secure systems that empower customers (KISS). "Meanwhile, we’ve seen philanthropies such as the Bill & Melinda Gates Foundation launch grand challenges for AI to help alleviate burdens on African healthcare systems. This has resulted in winners such as IntelSurv in Malawi, an intelligent disease surveillance data feedback system that computes data from smart wearables. Yet, even with hundreds of patents for such devices being registered every year, they’re not yet capable of consistently capturing high-quality data. In places like Malawi, these devices may become the single source of training data for healthcare AI, amplifying errors in their healthcare system." "Tech altruism has increasingly become suspect as AI companies are now facing an acute data shortage. They’re scrambling for data in the Global South, where majority of tech users live. Take, for instance, the case of Worldcoin, co-founded by OpenAI CEO Sam Altman. It plans to become “the world’s largest privacy-preserving human identity and financial network, giving ownership to everyone.” Worldcoin started as a nonprofit in 2019 by collecting biometric data, mostly in Global South countries, through its “orb” device and in exchange for cryptocurrency. Today, it’s a for-profit entity and is under investigation by many countries for its dubious data-collection methods."

4

1 Comment

It took us < 30 mins to integrate Llama3 to our health AI stack. It takes our customers < 3 mins. Within a week Llama3 went live with all the 101 GenAI healthtech customers. We’ve curated and integrated 15+ models over last 7+ months to enhance healthcare workflows. Don’t spend weeks or months integrating with just one model; instead have a “team of models” power your “network of copilots”. Great post by Prashanth Aditya Susarla about how to build a high performance AI product with team of LLMs.

16

If you’re looking for some LLM-based tools inspo, we got you. Our friend Anmol Baranwal has written about 17 LLM tools that went viral after launch. I’m sure you’ll get inspired by these cases. And hopefully, you start building your own thing! https://lnkd.in/d3Ck3wVr

15

1 Comment

LLMs are great at summarization, but summaries are not great for learning ⚡ Many engineers, across all levels, have started relying solely on summaries generated by LLMs to understand any concept. But it does more harm than good. Learning from summaries is quick and easy. It makes you think that you understood the concept, but one probing question and you will realize how surface-level your understanding was. Summaries are lossy, which means there is a loss of information (crucial details) when your favorite LLM is summarizing stuff for you. Uncovering that information requires you to ask deep questions to yourself and subsequently to LLMs. Asking tough questions is tougher, and it requires you to have a deeper understanding of the subject. Deep understanding cannot be built just by reading summaries, which is the new habit in town. Do you see the cyclic dependency here? Don't get me wrong, summaries are important as they act as a great starting point, but solely on them is catastrophic. So, whenever you find time, grind it out - read books, blogs, papers, docs, and good codebases. Once in a while build projects and prototype the concept. The depth of understanding separates the best from the better, the better from the good, and the good from the average. ⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff. youtube.com/c/ArpitBhayani #AsliEngineering #CareerGrowth

577

15 Comments

Most LLMs(ClaudeAI by Anthropic or GPT by OpenAI or Gemini by Google) use invalidated code snippets(from GitHub Issues, Stackoverflow and other forums) which originally could have been shared to seek correction or clarification. As I resume full-stack development and spend more time trying to get some meaningful work done(not boilerplate stuff), it not only wastes time but suck a lot of attention and cognitive bandwidth. Is there someone who has been able to get some real work done using these tools(consistently)? If you have achieved something by cleverly tweaking and constraining a prompt for a very particular niche use case, I believe that's you doing the work, not the model. #llm #ai #gpt #generativeai

13

3 Comments

I'm super excited to launch ⌘ Langbase.com 🥳 ⌘ Langbase – Serverless AI developer platform to ship AI agents, memory, and tools in minutes, not months. Deploy AI Pipes: Hook any LLM to any data, hyper-personalized API AI Memory: Managed search engine API with RAG tools BaseAI.dev — the first Web AI Framework. ❯❯ So what is ⌘ Langbase? Langbase is a composable AI infra and developer experience to build, collaborate, and deploy any AI products and features. We are a team of nine engineers who love building tools for other engineers. We asked, can AI infra empower every developer? Imagine AI as accessible as `npm install`. ❯❯ Our mission AI for all. Not just ML wizards. Every. Single. Developer. (you too!) Since the soft launch in January, over 23,000 developers have signed up. ❯❯ We're accelerating 🔹 58+ Billion AI messages tokes 🔹 107 Million AI API requests/runs 🔹 Written over 2.1 million lines of code … and we're just getting started!! ツ ⌘ Langbase closed an oversubscribed pre-seed round in Jan. We intentionally focused on raising money from founders and operators in the DevTools space: We're backed by incredible investors: Several @a16z scouts and Firestreak Ventures via Walter Kortschak Amir Rustamzadeh, who backed world-class companies like Stripe, Anthropic, Cohere, Hugging Face, OpenAI, Perplexity, and Groq. Tom Preston-Werner - founder GitHub Amjad Masad - founder Replit Guy Podjarny - founder Snyk Feross Aboukhadijeh - founder Socket Paul Copplestone - founder Supabase Zeno Rocha - founder Resend Ian Livingstone - CTO Manifold Luca Maestri/Amirteymour Moazami/Enea - CFO of Apple Ahmad Nassri - ex CTO npm, Inc. TELUS Anand Chowdhary - CTO FirstQuadrant As well as incredible angel investors like Michele Catasta (VP AI Replit, PhD Standford), Logan Kilpatrick (Google · OpenAI), and @Faraz Ahmad (Netflix) — the complete list of investors is here: https://langbase.com/about We've also acquired LangUI.dev, which helps you build and deploy a custom ChatGPT with any LLM. It recently crossed 2,300 stargazers on GitHub. A million visitors and 200K+ component copies by developers to build their custom AI UIs. Join ⌘ Langbase Discord: https://lnkd.in/g_ytC_xs So where are we going? We're building the best composable AI infrastructure with first-principle thinking, breaking down all primitives where incumbents charge 100x more. We're building the developer-friendly future of AI Infra, where you don't need a PhD or $100m to see ROI. I also want to thank our incredible community of users and customers. Thank you for your trust, feedback, and incredible support throughout the year. Starting today, there's no waitlist. ツ Hopefully, you'll like our new site as well. ツ ⌘ Langbase is available today. https://langbase.com You can start with the docs and solutions. ↳ https://langbase.com/docs Explore 100+ public open-source AI pipes by our team and community. ↳ https://lnkd.in/gVG-9m7A Let's go! AH! #Langbase #AI

333

78 Comments

200$ is 10x the Plus tier. Plus tier is possible for people to purchase and not worry about company policy of reimbursement etc. But 200$? Few individuals will take it. Executives and senior leaders will most likely take this on their own. But vast majority will gladly use Pro if their employer is ready to reimburse it. And remember, many large companies have a learning fund of 10k USD per year. 20% of this to equip your employees with o1 like capabilities is a force multiplier.

1

The Urim and the Tumim. 1.57/1.58bit vectorization is Null, True or False. In CPP, quite lossless if I am to trust the rumors, free and on device.

1

This feels like a game changer. NVIDIA is accelerating past Moore's law. The 5070 will be priced at $550, but deliver the same performance as the 4090 (currently at $1500), ie at 1/3 of the price. Bonkers. But I am not just joining into the universal fan-boying. It matters for RADiCAL and other startups whose vision is to bring 3D graphics to everyone, everywhere. RADiCAL currently optimizes its browser-based 3D graphics to render smoothly across most consumer devices (incl Apple), and we run our AI loads (for real-time, markerless motion capture) in the cloud. We can reliably factor into our roadmap that, within 2-3 years, a much larger number of users will have highly performant machines that support the compute and memory needs of high-end grapics and AI inference loads.

54

13 Comments

There is no such thing as a Trained LLM - https://lnkd.in/gbVv_C3K What I mean here is that traditional LLMs are trained on tasks irrelevant to what they will do for the user. It’s like training a plane to efficiently operate on the runway, but not to fly. In short, it is almost impossible to train an LLM, and evaluating is just as challenging. Then, training is not even necessary. In this article, I dive on all these topics. ➡️ Training LLMs for the wrong tasks Since the beginnings with Bert, training an LLM typically consists of predicting the next tokens in a sentence, or removing some tokens and then have your algorithm fill the blanks. You optimize the underlying deep neural networks to perform these supervised learning tasks as well as possible. Typically, it involves growing the list of tokens in the training set to billions or trillions, increasing the cost and time to train. However, recently, there is a tendency to work with smaller datasets, by distilling the input sources and token lists. After all, out of one trillion tokens, 99% are noise and do not contribute to improving the results for the end-user; they may even contribute to hallucinations. Keep in mind that human beings have a vocabulary of about 30,000 keywords, and that the number of potential standardized prompts on a specialized corpus (and thus the number of potential answers) is less than a million. ➡️ Read the full article at https://lnkd.in/gbVv_C3K, also featuring issues with evaluation metrics and the benefits of untrained LLMs. #llms #rag #openai

234

25 Comments

Recently there has been debate between RAG and 'long-context' LLMs. What are RAG and long-context LLMs in the first place? LLMs have traditionally had a limit in something called the "context window". The context window is like the "maximum word length" for prompting LLMs. Since the "maximum word length" was short for LLMs, we had to come up with ways to shorten the input length if we wanted to use something like a long PDF document as an input to the LLM. RAG emerged as a technique to solve this. RAG takes your query and lets you 'search' for _parts_ of the document that could be related, and attaches that to your query to give the LLM _some_ of the information about the long document to the LLM. The problem with RAG is that since the LLM isn't actually given the entire document, it relies heavily on the accuracy of the search mechanism. Even just defining "parts of the document that is related to the query" is not trivial and hence there's a whole array of techniques you need to learn about to deal with this. Because of this, RAG can have accuracy issues in many cases. Recent LLMs have a much longer context window, so in some cases you are able to just use the whole document as a prompt. In this case you don't have to do any tuning of the search mechanism so it's straightforward, and the accuracy usually increases too. This "just put the whole thing in the prompt" approach is referred to as "long context". The problem with "long context" is that the computational costs & latency are high, because the LLM needs to process the entire document instead of just chunks of the document. A technique that is recently becoming popular to deal with the high cost of "long context" is "KV cache reuse". The simple explanation for this is that when you process a PDF document with an LLM, the LLM builds up an internal state known as a "KV cache". If you then save this KV cache somewhere, you don't have to re-process the PDF document if you just load this KV cache up. Although many LLM providers implement some flavor of this internally, this is generally still something that is up-and-coming for general open source implementation. So how do you then choose between "long context" and "RAG"? If you need an understanding of the entire document (for example, the document is a manual that the LLM always needs to refer to) and you want to do repeated processing on the document (hence you can benefit from KV cache reuse), then long context could be a good fit. If you know you only need parts of the document and you have a good sense of how to find that one part (and can design a search method for it), then RAG is a good fit. (Or alternatively, if your document is too long to fit in the long context). At Outerport we're developing technology to make KV cache re-use straightforward & easy & extremely performant- reach out if this interests you at all!

53

2 Comments