Michael Schmitz

Its about solving the problem, not 'having AI' in the system. This is from last March, but the Wave 1 vs. Wave 2 distinction does seem to be playing out. Less 'generate me content', more 'synthesize vast amounts of relevant data and aid in decisonmaking or signal extraction'. https://lnkd.in/ehFBFC32 They don't talk about multi-agent systems here, but looking forward to more startups with a focus on leveraging agentic design patterns (https://lnkd.in/ed5rMmFe) to streamline real world workflows.

4

1 Comment

Ever wonder how exactly your knowledge sources are being used during RAG? We did too. I'm excited to share our early work in explaining LLMs, "RAGE Against the Machine: Retrieval-Augmented LLM Explanations"! 🎸✨ 📄 Paper: https://lnkd.in/eV4qF8YV 📺 Video: https://lnkd.in/e6mNV8tt Last week at ICDE 2024, I introduced RAGE, an interactive tool designed to explain the outputs of LLMs augmented with retrieval capabilities (RAG). RAG needs explaining since it is unclear how the presence and order of knowledge sources (among other things) affects the LLM's answer! 🤖💡 RAGE helps users understand how LLMs generate answers by identifying parts of the input context that, when moved or removed, change the LLM's response. This counterfactual approach makes AI decision-making more transparent, and allows us to derive a form of citation. 🔄📊 Key features include: - Combination Tests: Which sources lead to which answers? 🔍 - Permutation Tests: How does the answer change if sources are reordered? 🔀 - Interactive Demo: Explore behaviors of real LLMs using real data. 📚 Whether it's determining the greatest tennis player or answering your own unique questions, RAGE supports various use cases. Our code (coming soon) is designed with interfaces, making it highly adaptable to any LLM, retriever, or text data source! 🎾🏆 For technical details and implications of our findings, please check out our paper. Along with my co-authors at University of Waterloo, York University and AT&T, we are working on several extensions to explain many unexplained aspects of LLMs, RAG, and beyond. Stay tuned for more updates and an open-source RAGE code release in the near future! 🚀💡 #ExplainableAI #LLM #RAG #Research

15

2 Comments

How do the costs compare of implementing a low-latency write-ahead-log (WAL) on S3 Express One Zone and one implemented as a State-Machine-Replication (SMR) system (such as Paxos/Raft/Kafka)? I built a cost model for both to find out. https://lnkd.in/dwczVutW

43

5 Comments

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

23

Well, my day ended up taking a different path than I'd thought, yesterday! Instead of my primary research, I ended up down a literature review path. I'm sure by now you've heard that OpenAI announced it's recent model release called o1 so I thought I'd provide some interesting papers that I think might align with what is implemented within the reasoning module outlined in the o1 system card. If not, it's at least a foray into some newer methods being explored today. 😊 📄 o1 System Card: https://lnkd.in/gwSRs46w 📄 Large Language Monkeys: Scaling Inference Compute with Repeated Sampling: https://lnkd.in/gSTe43Bq 📄 ReFT: Reasoning with Reinforced Fine-Tuning : https://lnkd.in/gKPzfhfN 📄 Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning: https://lnkd.in/gqMy_dKu 📄 Reinforced Self-Training (ReST) for Language Modeling: https://lnkd.in/gTeaCqrM

22

3 Comments

Insightful how multi-agent AI can provide a better solution to many problems.

2

2 Comments

I've dusted off my personal blog after years of neglect and published a new post: How to add natural language AI data filters to your app. Take a look! https://lnkd.in/gxT3NfRr

22

1 Comment

Anthropic released another paper on their ongoing work towards interpretability of large foundational models like their Claude 3 family, built on previous work by many. Their work continues to be enlightening and inspiring. The findings in this work are generally similar to previous findings on smaller models but scaled up to Claude 3 Sonnet: 💡 they find large numbers of "features" corresponding to combinations of neuron activations and these features map to concepts that we recognize/understand: for example, features for transit infrastructure, particular people, concepts (like bias and "objects lacking sentience"). This might sound familiar from classical machine learning, but the interesting thing is that there is no single thing/neuron encoding each of these concepts, it's a particular combination of neuron activations that embeds these concepts in "superposition" in the network. 💡they observe a sort of "distance" of these feature activations that arrange themselves "near" semantically similar features. When they use their sparse autoencoder approach to learn more and more features they observe that the level of abstraction in the detected features becomes more granular, in ways that are interpretable and meaningful. 💡when they artificially fire these activation patterns during inference, it affects the output in ways that relate to the "feature". For example, asking Claude "what's the most interesting science" normally it responds Physics, but if you clamp the feature activations for the feature "Brain Science" to 10x their max values then Claude responds: Neuroscience. Ways to interpret _and_ influence the model. Great readable work [1] along with an interactive explorer of some of the "features" themselves [2] 🎂 There's an easter egg too: if you scroll in the feature explorer tool and try and select one of the "Randomly selected features from 1M" you get a message "if you want to see the rest of the features, we're hiring!" with a link to their recruiting page :) [1] https://lnkd.in/gZbparwp [2] https://lnkd.in/gC9eus53

19

There is a surprising amount of complexity to choosing colors to represent data. Here are some of my favorite resources: https://lnkd.in/gg_YHqGV https://lnkd.in/g-priQZB

3

If you're working in ed-tech or building AI tools, making sure language models (LLMs) give accurate info is a big deal. One challenge is "hallucinations," where models sometimes make stuff up or get things wrong. Big Kudos to Ryan for creating this innovative solution : using language-based methods to spot errors in LLMs. It’s an exciting way to investigate AI tools and how to create more reliable and effective solutions for students and teachers. #ai #edtech #logprobs

4

Something that only became obvious to me after I had been writing software for a couple of years is that there are different types of bugs, with different characteristics, impacts, and mitigation strategies. One class of bug that doesn't often get talked about is what I call the Semantic Bug - a Semantic Bug is basically an "oops I made some assumptions about this data/API/module that aren't really true" bug. For example: - "oops I thought this method was idempotent but it's not" - "oops I didn't think users could be active in two teams at once but they can" -"oops I didn't think these events could happen in that order but they did" I find it helpful to think of a dichotomy between logic bugs ("the code doesn't do what I intended") and Semantic Bugs ("the code does what I intended, but in the context of the larger system the thing I intended is wrong"). You might not run into Semantic Bugs too often when, say, working on a personal project, but they can be a real problem when developing larger software systems - especially when you have to reason about models and interfaces owned by other teams. Some types of bugs are fairly benign - if some unicode formatting looks weird in production, for example, you should fix it, but it's not going to break any functionality. It's generally ok to just squash these bugs when they show up and move on. You don't want to let that happen with Semantic Bugs, though, because they can be pretty nasty and unpredictable: -They don't always lead to a simple crash. In the worst cases, they can cause faulty components to exhibit essentially undefined behavior, potentially lying to other components about the state of the world in a way that’s hard to detect. Bad behavior can spread through the system, causing it to corrupt data or take inappropriate action. -They can be hard to catch with tests, if only for the reason that you’re apt to make the same faulty assumptions writing your test cases that you made writing your code. -They’re pernicious. Sometimes your wrong assumptions may only be wrong under very specific conditions, and you may be (un)lucky enough that it takes months for those conditions to occur in production. This can be maddening - when bugs occur directly after deploying a code change, you can at least point to a probable culprit, but when these “sleeper” bugs hit it feels like your happy, stable system keels over and dies for no reason. Semantic Bugs are really a product of our fallible human brains more than any technological constraint, so it might seem like there's not much we can do to prevent them besides making sure we're communicating well with our partner teams. The good news is that there are, in fact, architectural techniques you can use to make your software naturally resistant to Semantic Bugs! We put up a survey of existing approaches (along with a couple new ideas) at https://lnkd.in/g4mjqw2a - check it out if you're working on a large codebase!

15

One potentially really good use for AI (for the rich person who wants to contribute back to US society): a model which asks for someone's opinions on general topics, and then correlates that with all available data for election choices, and fills out a sample ballot for the person based on the data. Each item could have a "drill down" link, to explain the available trade-offs, and why the option was selected. Rationale: A lot of voting is inefficient (in terms of "selecting the best option per people's desired outcomes"), because voters are bad at correlating large amounts of data, and propaganda people are good at confusing people and misrepresenting causal actions. But ML models are good at the former, and if trained properly, effectively immune to the latter. My assumption is that most voters/people are generally well-meaning, and trying to do the "right thing" for the country, but that's increasingly difficult in today's world of disinformation and propaganda. An unbiased model which aligned suggested choices with desired outcomes and trade-offs, based on actual data, would be potentially tremendously beneficial for getting better outcomes from elections in general. Note: This would not be an easy problem to solve by any stretch... but it could be a really positive change, imho.

1

I couldn't recommend this more - I was reading through the other day, and the arguments resonated quite strongly with the intuitions around why a purely preferentist (or alternatively, purely utilitarian) approach to AI alignment may be flawed. This is especially important in the coming era where we are likely to see advanced AI assistants more widely deployed than ever before - and where aligning them to human values and user well-being remains of utmost importance. https://lnkd.in/dyWqkTQm

30

FlashInfer v0.2 is here, a lot of technical upgrades, great news for anyone are doing custom attention improvement or just benefit it implicitly from integration of LLM frameworks

39

You don’t normally see LLM researchers and engineers talking about their day-to-day work. Big shoutout to this YouTube video from the Anthropic LLM Interpretability team, the team behind the Golden Gate Claude. It’s amazing to see that most of the team members have worked for the team for only one year, and yet they have made such a big breakthrough in interpretability research. Another thing I want to highlight is the unique challenges facing the team and how they approach these challenges, that is, through fast trial-and-error iterations and constant evaluation of what may work and what to prioritize next. #artificialintelligence #machinelearning #deeplearning https://lnkd.in/eUaPAYch

97

1 Comment

🚀 Meta just released Llama 3.3 70B with some exciting updates: 128K context, multilingual capabilities, and enhanced tool calling. It’s reported to outperform Llama 3.1 70B and even compares favorably to Llama 405B! 🎉 But let’s not jump to conclusions—these broad generalizations can be misleading. 🤔 Here are a few critical questions to consider: - Which models does it NOT outperform? (For instance, Qwen seems to be missing from comparisons.) - On which benchmarks did it struggle or show deterioration? I’d love to hear your insights, thoughts, or comments. Let’s dig deeper into this release together! #AI #ML #LLM

2

This blog post comprehensively presents fundamentals of neural sparse search and measurement of its effectiveness. https://lnkd.in/gJ6yUFEB

29

"The programming language is now human. Everybody in the world is now a programmer." - Jensen Huang Prompts are the new programs. GPT agents the new apps. Those of us trained in classical deterministic programming need to adjust our mental models to get better at leveraging GenAI tools. Only way to get better is to practice. #GenAI

8