Jeffrey Arnold

This is the final part of the 3 part series of posts that shows how to Evaluate your LLM based RAG application. In the last part what I would like to show us how to Evaluate the Answer (Actual Response) faired against the Contexts Provided after Retrieving it from a Vector Database. The only metric that is available for this is Faithfulness. Where the Claims made in the Answer (Actual Response from the LLM) are checked against the Claims present in the Contexts Provided to the LLM to answer the user query. The way the Claims are obtained is by asking the same LLM or another LLM to present all the Claims made in the Answer, and then get all the Claims made in the Contexts that was provided to the actual LLM to generate the Answer. Then calculate the ratio between the two. #llm #evaluation #metric #ai

4

Question for engineering leaders (managers, directors, VP's, etc): I'm curious about the current landscape of developer productivity analysis + management tools - things like LinearB and Jellyfish. Which tool, if any, are you using? Any opinions/pros/cons on the tool you're using (or the entire domain overall)?

21

This is more of a "talking to myself in public" thing than any kind of fully fleshed out hypothesis, so take this with the appropriate grain(s) of salt. But if I were interested in AI and were looking for an angle to explore that might yield something interesting, I'd start working on learning "Multiscale Mathematics" (also known as "Multiscale Modeling"). This is rooted in my personal belief that there has to be a "multi-scale" aspect to intelligence because there is such an aspect inherent in our underlying physical reality.

2

The best use case for overseas development I've heard: We have customers in the eastern hemisphere. To support those customers better, we're standing up product, dev, and support teams to help serve those customers needs. Two key ingredients in making that successful: commensurate pay and the same level of performance and accountability as US peers. #software #international #teammanagement #overseasdevelopment #startup #leadership #coaching

6

🤔 Looking at YC's latest Request for Startups struck me: Every category—gov tech, manufacturing, fintech, space—has AI humming under the hood. 🔄 Maybe it's not about having AI (like it's not about "having electricity" ⚡). It's about reimagining whole industries with it. 🚀 Checkout the whole article: https://lnkd.in/gcQkKMja

8

Interesting conundrum: A kebab menu is more visible using white dots on the blue background than black dots on the same blue background. But the contrast ratio is bad with the white (2.76:1) and good with the black (7.6:1). I’ve asked three separate folks with low vision to tell me which was better for them. They all say “white dots”. Granted, it’s a very small sample. Still. This leaves my client with an interesting choice: pick the option that is more accessible, but not standards conformant, or stick to the standards but end up with an element that is harder to perceive. What would you do? #Accessibility #WCAG

12

23 Comments

With Redpanda Data's recent acquisition of Benthos I've gotten some questions about how it compares to Arroyo. In short: Benthos is *stateless*, while Arroyo (like Flink or RisingWave) is *stateful*. This may sound small, but adds up to a fundamental difference in how these systems are designed, built, and used. In a streaming engine, statefulness combines two inter-related features: * Repartitioning data (GROUP BY in SQL) * Remembering previously seen events With just these two features, we can support most of normal SQL (window functions, joins, aggregation, etc.) plus time-oriented streaming operators (time windows, event-time semantics, watermarks). Stateless engines on the other hand are limited to simple transformations and filters (SELECT and WHERE in SQL terms) that operate on individual events. But all of this is a tradeoff; stateful systems are much harder to operate. So if you don't need joins, aggregates, or windows, a stateless system can be a better choice. They also work well as infra glue, routing data between different systems—Benthos in particular has a huge connector ecosystem. For a deeper dive on this, I wrote down some thoughts on our blog:

57

4 Comments

Meta researchers develop ‘thinking’ LLMs  Meta researchers just introduced a new method called Thought Preference Optimization (TPO) to train large language models to ‘think’ before responding to general instructions — not just reasoning tasks. The details: →TPO prompts models to generate internal thoughts before responding to user instructions, similar to how humans think before speaking. →The AI's thoughts are kept private, with only the final answer shown to users with the AI using trial-and-error without direct supervision to optimize outputs. →TPO outperforms standard models on key benchmarks for non-reasoning tasks like marketing and creative writing but declines in math-related tasks. →The approach builds on the recent OpenAI ‘Strawberry’ research and o1 model release, which takes time to reason. Why it matters: Meta’s own Yann LeCun may scoff at the notion, but this method shows the potential of AI to ‘think’ for broader tasks than just math and reasoning. By allowing models to learn useful thought processes independently, TPO could also enable more capable and flexible AI assistants across a wider range of domains.

6

1 Comment

Parsing tables from long pdfs is hard today, but getting easier everyday. Retrieval of mid sections of the document is hard today, but getting easier everyday. Taking a judgement call, getting everyone on the same page, decision making with imperfect information, communicating to inspire, asking the right questions, having an agenda is hard today, and will remain hard tomorrow. #Skills #AIAge #HumanEdge

17

6 Comments

If you're working in ed-tech or building AI tools, making sure language models (LLMs) give accurate info is a big deal. One challenge is "hallucinations," where models sometimes make stuff up or get things wrong. Big Kudos to Ryan for creating this innovative solution : using language-based methods to spot errors in LLMs. It’s an exciting way to investigate AI tools and how to create more reliable and effective solutions for students and teachers. #ai #edtech #logprobs

4

~40% of Frec’s customers are engineers working in tech, many of whom expect frequent and large capital gains. They also prefer to self-manage their money, which makes Frec an ideal platform. Meet Evan: an engineering leader who upgraded from investing in ETFs to direct indexing. Direct indexing appealed to Evan because it offers comparable returns to his ETFs while saving him money on advisory and ETF fees. “ETFs are true and tested, and giving folks a different way of investing in the market while saving more money is really appealing.” He’s used many other robo-advisors and other fintech tools, but “Frec seems to maximize tax loss harvesting opportunities better than any other product that I’ve used.” He also likes Frec's transparent fee structure.  “I feel like with a lot of other companies, that’s muddled in with the cost of investments,” he told us, “and you don’t really know how much you’re paying or how much you’re not saving as a result. I think Frec does a really good job of laying all of that out and giving you a true understanding of the cost of these investment products.” Read more about his Frec journey here ⬇

45

2 Comments

I do a lot of AI automation work, much of it involving batch processing and reprocessing of vast amounts of text. The low cost and high quality of Claude 3 Haiku (25 cents per million tokens in and $1.25 per million tokens out) has been fantastic. But today, OpenAI released GPT-4o-mini for use in API calls, driving the cost of AI automations even lower: 15 cents per million tokens in and 60 cents per million tokens out! To give you a sense of the savings: I once accidentally ran my batch processor using the high-end GPT-4o model. With 12 million tokens submitted and 2.5 million tokens returned, it cost me $97.50. With GPT-4o-mini, today that cost is only $3.30. That’s over a 96% savings! 😲 I have reviewed the Llama 3 8b + 70b Meta open-source models for these automations. The cost is lower still (the model is free, but hardware time is not). However, in my tests, the quality just isn't the same. Plus, considering the additional effort to host an API to access the Llama 3 models, my engineering time and money have so far been better spent using the low-cost models from OpenAI and Anthropic. Anyone else finding similar results in savings + quality?

36

1 Comment

I want to thank Ethan Mollick for the post below. Check out https://lnkd.in/eBXSssFH and his book "Co-Intelligence: Living and Working with AI." Considering Marshall McLuhan’s tetrad of media effects, let’s explore some questions about the implications of the WildChat research: 1. Retrieve: Are LLMs replacing human mentors and teachers and collaborations? How do we feel about relying on LLMs for creative inspiration instead of real people? 2. Enhance: AI speeds up our thinking and makes us more efficient, but is there a neurological cost? Are we sacrificing depth and reflection for a brain rut paved with ΔFosB? (https://lnkd.in/eh2cRTme) 3. Reverse: Could our dependence on LLMs erode our ability to think independently? Could creativity reverse into inability? 4. Obsolesce: As AI makes traditional thinking obsolete, are we losing the art of thought itself? Will LLMs obsolesce us?

2

2 Comments

"We don't yet know...what the great product experiences with AI are going to look like. I think those are yet to be created." - Brad Lightcap (OpenAI COO) This statement comes courtesy of our president's (Tobias Dengel) post (linked in comment), and I think it's worth reiterating: AI is NOT the product. The product is the experiences we facilitate, and we still have a lot to figure out about how to make those valuable for people. I'm working with an AI engineer now on a multimodal experience, and it's immediately apparent how much we need to understand about users' use contexts, preferences, abilities, etc. to create something truly useful. Otherwise we're just tacking on ignorable or annoying features. If we're indeed on the cusp of a profound upset with our digital interfaces (which seems overdue...), researchers and those with a socio-technical lens are going to be the ones to help it take hold in a substantive way. What's your take? Is Apple Intelligence a tipping point in profound human-computer interactions? What will make the difference in execution? (Link to original post in comments.)

14

1 Comment

What's going to happen to NVIDIA now that they've reached 3tn market cap? For the current employees and long timers... I've read stories of mid-level engineers who reportedly earned more than $60M via stock appreciation. Imagine the numbers for the more senior folks. How will NVIDIA keep folks around who are post-economic? And for all the new folks... The value proposition is complex. The company is the most valuable (by market cap) in the world. If they are heavily compensated via RSUs, the assumption is that there is significant remaining upside to be gained. It's incredibly difficult to continue orders of magnitude stock appreciation after reaching the absolute zenith... NVIDIA is an obvious tech darling - it looks and feels like a version of Google in the mid to late 2000's - but I'm really interested to see if the immense financial success of the stock will end up driving out the core talent group (eg, the long timers who are now mega rich) while also limiting access to new upstart talent who don't see a path to outsize financial success. Curious what others think!!

36

10 Comments

Last year, Huggingface deprecated their InferenceApi client and switched inference calls to InferenceClient. Simple enough. But see, Langchain's HuggingfaceHub package still called InferenceApi. So anyone running langchain.llms.huggingface_hub received an future warning error that InferenceApi was depracated and to use InferenceClient instead. The problem was that unless you went searching around into the details of the code langchain.llms.huggingface_hub package, you had NO idea what this error meant and whether it was an important warning to heed. So one would think Langchain would resolve this quickly, right? No. This was not resolved universally until a January 2024 update. Until then, people were using patches to get around this error. AND if you're thinking, "Wow! Well, glad that's no longer a concern" think again because out of curiosity, I asked ChatGPT 4o list imports for me to create multi-agent AI using langchain and inferencing the mistral API and wouldn't you know it, it suggested I use the deprecated InferenceApi client. Now, can you use the InferenceApi client and it still work? Sure. However, that future warning error gives lots of engineers who have not seen that error before pause AND it would likely be an issue when considering if one's code is production ready. When one AI company changes a client name and another AI company is using their own client to call on the original AI's client, this creates another dependency. One that a LOT of engineers building AI are not aware of unless they get used to seeing these errors over and over and over again. Little things like this are keeping great AI model demos from becoming great AI products.

45

6 Comments

So many friends that were part of those layoffs, maybe this can help avoid future surprises. https://lnkd.in/eFkq7sib #WARNTracker #Layoffs #EmployeeRights #JobMarket #DataAnalysis #EmploymentLaw #JobSearch #CareerTransition #HRTech #EconomicTrends #WorkforceManagement #BusinessInsights #TechLayoffs #CorporateStrategy

1

Who would our solutions engineer role be great for at Mapistry? My likely uncommon take on the "normal" company job description ☑ You love figuring out problems....you like digging into spreadsheets and asking well articulated questions based in curiosity (e.g. why are you doing it that way?) ☑ You want to solve problems (technical and user/workflow) that have an environmental impact using software ☑ You like working with people, maybe you consider yourself an "extroverted engineer"....you like some time working with customers and internally with your team but also like the head's down time to do deep work and solve problems. The variety is key to your happiness! ☑ You want to work in-person some or most of the time in Berkeley, because you learn quickly and fuel yours and others creativity that way ☑ You like being organized....maybe your use project management software in your personal life or color code sticky notes or like organizing your work photos/files and developed your own naming conventions or ??? ☑ You value direct, frequent communication.....you communicate well within in a team, to clients, and to managers. You love giving and receiving continuous feedback. You follow up to emails quickly, you send meeting summaries, and you ask about next steps ☑ You are ambitious in learning/growing and having an impact in your role.....small company and working in-person with myself and Allie Janoch means you get direct access but also like the pressure :) If this sounds like not much fun, please don't apply :) there are lots of jobs out there that will be great for you If you are excited after reading this and say "yes" to each of these then please apply! https://lnkd.in/eZuvcjQN If you are not sure what I mean with any (or all of these), but are intrigued by the role...send me a message!

6

2 Comments