Utku Demir

This research from Facebook efficiently hard-codes large amounts of information into model memory layers, without increasing the number of required computations. It matches the capability of models roughly 4x larger. It uses key-value pairs within the actual model parameters that can be recalled highly efficiently, this is especially important for fields where factual accuracy is important (100% improvement in factual recall on QA benchmarks). There was also some custom CUDA Kernels to overcome memory bandwidth bottlenecks for the EmbeddingBag operation (used to recall the most appropriate information based on the query), and other cool stuff. Facebook clearly has multiple independent teams converging with all this research, I think a combination of all these breakthroughs -rather than a single event- is really going to move us forward. https://lnkd.in/ecC7MW_U

4

1 Comment

How to lie with statistics : This chart gives a very distorted view of the software industry. C/C++ code is used in real products, so most people / companies won't put them up on GitHub. I have a million lines of high-quality C code in Equinox3D, and I'm not putting it on GitHub. Python is a very inefficient toy language that's only useful for prototyping. You can't use it when performance and efficiency matter (and when does it not?). So, Python code doesn't have as much value and folks don't mind sharing it. If something is free, there's a reason for it! Of course, this means that when we try to use that code, we have to translate it to C/C++. But this is the one thing LLMs are pretty good at. Python is used in machine learning, but even there it's just the "glue" / scripting framework during model tweaking and training. The heavy lifting is done on GPUs using C/C++, CUDA etc. And the final application on the inference side is also in C/C++ and it just loads the trained ML model as a binary blob, then runs the inference code on a GPU or dedicated hardware, again not written in Python.

10

4 Comments

🚀 Exciting news! My Pull Request to the PyDriller library has just been merged! 🎉 In this PR, I introduced a new method, get_added_and_removed_lines(), to the CodeChurn class. This feature provides a more granular view of code churn by offering the number of lines added and removed per file across commits in a repository. Why is this important? 🤔 The value of code churn alone can sometimes be misleading. For example, there are two common ways of calculating code churn: Delta (added - removed): If the number of lines added and removed is the same, code churn equals 0, which could mistakenly suggest that no changes were made, even though there was significant modification. Sum (added + removed): While more precise, knowing both added and removed lines provides more context about the nature of the change. With this enhancement, developers can now easily access both values, making it clearer how much code was actually changed and ensuring a more accurate interpretation of churn metrics. 🔍 Key Changes: New method: Returns a dictionary where file paths are keys, and the value is a tuple (added_lines, removed_lines). Refactored Code: Updated the _initialize() method to store both added and removed lines for each modified file. Documentation: Updated the Code Churn documentation to reflect the new feature. Unit Tests: Added tests to verify the accuracy and reliability of the get_added_and_removed_lines() method across multiple commits. 💡 Impact: This change makes it easier for developers to track code churn details at the file level, offering deeper insight into how much work was actually done in a project. Check out the full details of the PR here: 👉 PR #299 in PyDriller: https://lnkd.in/ebmMQXT9 A big thanks to Davide Spadini, the PyDriller maintainer, for the review and merge! 🙏 #softwareengineering #opensource #pydriller #codechurn #python #versioncontrol #git #contribution

8

People rightly state that there is a lot of hype surrounding #AI. This reflects our obsession with applying new tools to *every* problem, fit for purpose or not. 😄 Here's the good part, though: when applied properly, AI is a huge force multiplier. Imagine a lever ^ 1000000. I've been sharing, any time I have the opportunity, the "good parts": areas in which AI can have real utility now. Not someday. Not maybe. Absolutely yes, and now. One area in particular is #RAG, Retrieval Augmented Generation. RAG == Turning an AI loose on YOUR DATA. Imagine all of those documents in which vital information is squirreled away, hard to search, even harder to correlate *across* those myriad documents. What a treasure trove if only you could leverage it! With RAG, you can...and pretty easily, too. Using AI and RAG, 1+1 pretty quickly adds up to 500. Or more. Even now, the power of your AI-enabled application is more limited by your imagination than the capabilities of the tools. If you want to know more, get just a taste of what you can do with AI using RAG, join me this Monday at #RAGHack. See below for more details and to save your spot! https://lnkd.in/gYAsiSP4

13

1 Comment

New post! Probably the most desired enhancement for database privilege management is finally here in 23ai. #database #sql https://lnkd.in/gXkXGWWH

31

Combine contextual retrieval and reranking to maximize retrieval accuracy in RAG systems 👾I converted it to a podcast using NotebookLM and heard it on my commute https://lnkd.in/gXRKBgPs this is my usual way now to read papers/tech blogs 🃏Highlights: 1. Embeddings + BM25 is better than embeddings on their own 2. Passing the top-20 chunks to the model is more effective than just the top-10 or top-5 3. Adding context to chunks improves retrieval accuracy a lot 4. Reranking is better than no reranking Summary: Combine contextual embeddings with contextual BM25, plus a reranking step, and adding the 20 chunks to the prompt 🔗https://lnkd.in/g2PGg_ca

8

I wrote a short tutorial on using Porcupine to check for linearizability (without needing to deal with the JVM). https://lnkd.in/eW6Y-QQQ

63

✨ New release for Au, Aurora's units library --- and it's a big one! Check out the (voluminous) 0.4.0 release notes (https://lnkd.in/emKCAwRx). Here are some of my favorite highlights: * 🧑‍🔬 Specific physical constants (c, etc.) get included out of the box. And our `Constant` machinery has been upgraded too: it's the most capable implementation I've seen in any units library. * 🔬 Runtime conversion checking functions let you check for overflow or truncation for _any_ unit conversion, for each _individual_ value. AFAIK: a first for any units library anywhere. * 🏷️ We generate labels for ad hoc units, including more informative labels for common-units that "do the math for you" and express in every input unit. `[UNLABELED_UNIT]` is almost entirely gone! * 🏗️ You can now use Quantity _values_ as template parameters, even before C++20! * ➗ Improved syntax for integer division: it's both more natural and readable, and it will make generic programming far easier, while preserving our safety checks. This is just a fraction of the user-facing improvements, and doesn't even touch on under-the-hood stuff near and dear to my heart, such as our improved factoring. Read the release notes for any breaking changes, and give it a spin! 🚀

56

3 Comments

#RAG with advanced retrieval technique RAPTOR for long contexts. When working with long-context documents, we cannot just chunk the documents and embed them. Instead we would want to have a good approach for minimalist document splitting for long context LLMs. This is where RAPTOR comes into picture. Recursive Abstractive Processing for Tree Organized Retrieval [RAPTOR] is a new and powerful indexing and retrieving technique for LLM in a comprehensive manner. It adapts a bottom-up approach by clustering and summarizing text segments(chunks) to form a hierarchical tree structure. → The leaves are a set of starting documents. → Leafs are embedded and clustered. → Clusters are then summarized into higher level (more abstract) consolidations of information across similar documents. → This process is done recursively, resulting in a “tree” going from raw docs (leaves) to more abstract summaries. We can apply this at varying scales; leaves can be: → Text chunks from a single doc → Full docs With longer context LLMs, it’s possible to perform this over full documents. This tree structure is key to RAPTOR function as it captures bot high level and detailed aspects of text, which is particularly useful for complex thematic queries and multi-step reasoning in questioning and answering tasks. This process involves segmenting documents into shorter texts called chunks and then embedding the chunks using an embeding model. These embeddings are then clustered by a clustering algorithm. Once clusters are created, the text associated with each cluster is summarized by using a LLM. The summaries generated form nodes in a tree with higher level nodes providing more abstract summaries. Know more about RAPTOR in this research paper: https://lnkd.in/gDgZP6Vj Also, read this article by Plaban Nayak: https://lnkd.in/g-7HpADg ----------------------------------------------------------------- BTW, you also need a robust database in RAG. I recommend using SingleStore Database. Sign up and get $600 worth free credits. Try SingleStore database for free: https://lnkd.in/gCAbwtTC

229

1 Comment

Anshu Jain Well. I Agree. And with demand in integration more and more, each LLM subscription price will increase so at the end of the day, it's all about requiremnets, usability and benefits over time. Majority in major MAANG and it's affiliates have utilized open sources and trapped it in a ecosystem. Some who are still exploring more use cases are keeping it open so that demand can cover up for low market capture over time. #TBH #GoGetterAttics

1

How I Squeeze 60K RPS out of SQLite on a $5 VPS: There are tons of tutorials on how to set up SQLite, and sometimes they contradict each other. Here's what worked for me: 1. Configuring PRAGMAs: We need to send the following PRAGMA commands right after opening the connection: PRAGMA journal_mode = WAL; - Enables write-ahead log so that your reads do not block writes and vice-versa. PRAGMA busy_timeout = 5000; - SQLite will wait 5 seconds to obtain a lock before returning SQLITE_BUSY errors, which will significantly reduce them. PRAGMA synchronous = NORMAL; - SQLite will sync less frequently and be more performant, still safe to use because of the enabled WAL mode. PRAGMA cache_size = -20000; - Negative number means kilobytes, in this case 20MB of memory for cache. PRAGMA foreign_keys = true; - Because of historical reasons, foreign keys are disabled by default; we should manually enable them. PRAGMA temp_store = memory; - Moves temporary tables from disk into RAM, speeding up performance a lot. Do NOT use cache=shared! Some tutorials recommend configuring it, but this is how you get nasty SQLITE_BUSY errors. It is disabled by default, so you don't have to do anything extra. 2. Use immediate transactions: If you know that a transaction can possibly do a write, always use BEGIN IMMEDIATE or you can get a SQLITE_BUSY error. Check your framework; you should be able to set this at the connection level. 3. Open two connection pools: Another trick is to open two connection pools: one for reads only and another for reads/writes. Set the connection limit of the write pool to 1, and the connection limit of the read pool to some reasonably high number, e.g., the number of your CPU cores. Hope this helps!

13

1 Comment

Neural Magic ML team is introducing 2:4 Sparse Llama: The first sparse foundation model built on Llama 3.1 8B. We show that it is possible to combine sparsity and quantization for extreme compression.

20

2 Comments

Has anyone else noticed that #Huggingface’s #TRL’s #PPO trainer doesn’t work well with DeepSpeed Zero 3? While it performs fine in multi-GPU DDP settings, scaling up with larger model sizes becomes a real challenge. 🚧 On a related note, although we managed to make it work, using policy gradient methods with model.generate from the Hugging Face ecosystem can be quite slow for practical applications. This is because in policy gradient methods like PPO, we need to generate text from the current policy on the fly during training. Generating on the fly is essential to ensure that the trajectories collected reflect the most recent policy, allowing the model to interact dynamically with the environment and compute rewards based on updated behaviors. Perhaps integrating VLLM for efficient text generation and frameworks like Ray for distributed training could optimize the Hugging Face ecosystem for scalable on-policy RL methods? 🤔 This raises another interesting question: How can frameworks like VLLM and Ray be seamlessly integrated with Hugging Face to address these scalability and efficiency challenges? So far, I’ve come across Tulu from #AllenAI, which seems to have addressed some of these issues through Open-Instruct. https://lnkd.in/gviiJsny Do you know any other similar solutions? #Tülu3 #TRL #PPO #RL

31

4 Comments

Best Friends: OpenAI and Stack Overflow I must admit, I have learnt a good deal from using Stack Overflow. It is there that true experts often post excellent responses to questions. While there can be terrible answers, often, that is a prompt for a true expert to add their response. In cryptography, to me, there are few better places to go. And, so, OpenAI knows this, and, in places where you ask a technical question from ChatGPT, it just looks like it has scraped its knowledge directly from the site. Underneath ChatGPT, there’s not really much intelligence going on, as it is just reconstructing the answers from standard templates and filling in the knowledge from the sites it has gathered data from. And, so, now it becomes more obvious that AI is crawling human generated answers with the partnership between OpenAI and Stack Overflow. Many don’t like this and believe that the credit for their responses is being taken away. For this, some have removed their response but are faced with a stern message from Stack Overflow: https://lnkd.in/egWfdbnZ

43

11 Comments

# Melbourne Compose - will it prove a worthy successor to Melbourne Scala? Many of you will know that I organised Melbourne Scala User Group meetings for about 12 years. We had some great times! This post is about Melbourne Compose, a successor group to Melbourne Scala that I have started with John Walker, who runs the Melbourne Haskell meetup [https://lnkd.in/gzAmbsH7]. Join the group at https://lnkd.in/gFmDh-nz I believe Melbourne Scala expired due to a combination of - Covid and the post-covid rise of WFH work culture which greatly reduced in-person meetup activity across the city - The lack of a sponsor to cover the steadily rising fees charged by Meetup.com. If you don't payup $300+ a year, they don't just disable new events, they delete the entire group! (er.. no thanks, we'll take our business elsewhere) - A decline in activity around Scala, and functional programming generally, in Australia, (eg the loss of Lambdajam conference :cry:). People asked me afterward whether there would be a future Melbourne Scala, and then John and I got into a fruitful conversation about merging our efforts. We talked about what motivated us to run a meetup/user group in the first place. And we looked at what has been working elsewhere, such as Brisbane's BFPG [https://lnkd.in/gWErd4di] and Sydney's SydFP [https://lnkd.in/gdwn4dqB]. And formed some opinions.. Firstly, the world has a surplus of online content. So much community and content have gone online that there's little demand for more. And do people really want to be on zoom after hours when they already spent the workday online? However, the online experience is not the same as in person (IRL). While online is convenient, it is also lacking the spontaneity and social connection that physical company offers. I find communication via zoom etc plain exhausting, even after much practice. It is suited to broadcast model where one presents and others listen. But spontaneous conversation & collaboration feels stunted. Secondly, it seems that multi -language and-technology user-groups prove more resilient than narrower, focused groups. As well as drawing from a wider base, they can better adapt to evolving technology trends. And so, John and I decided we wanted to focus on a group whose charter spanned the breadth of "Typed Functional Programming" and which includes regular in-person gatherings. We realise that in this post-Covid world, IRL gatherings will likely be small. It's more effort to head out from home, than coming on to a city meetup after the office. But when I think back to some of the early meetings of Melbourne Scala, how awesome they were, and how much I learned and grew from those experiences, I do hope "Small is beautiful". (Cut due to word limit. Continued in Part 2)

18

2 Comments

Planning is an important part of work for most knowledge workers including software engineers. It’s not only about what I’m going to do today, but also what the team is going to do in this week/sprint/quarter. We want to come up with a concrete and pragmatic plan to align the timeline within the team or across many teams. Even though we want to be realistic, we always seem to be too optimistic and there are way more times we see ourselves behind the progress than ahead of it. This phenomenon is also mentioned in the book “Effective Engineer”. If I remember correctly, it said 80% of all software engineering projects are not finished on time. It’s interesting that people fall into this trap over and over again, including myself, even if I already read the book and realized this is the exact case for me. The simple fact is things can go wrong. At the personal level, it can be a bug that takes a couple hours to fix, it can be an important metric missing in the evaluation pipeline, it can be unstable CI/CD because of some unknown reason, or it can be the key code reviewer taking days off. At the team level, it can be abrupt priority changes, emergent issues, assumptions found invalid, or just miscommunication. During planning, we usually won’t allocate time for these disturbances, yet it always costs more time than we think. I bring this topic to experienced leaders in the company and here are two directions from their advice: (1) When possible, have checkpoints and rolling planning. Accidents are by definition unexpected; the important thing is to adapt and align. (2) Invest in discovery on a regular basis. As we underestimate the risks, we are also underestimating the value to reduce the risk by discovery. These are good starting points to crawl out yet still vague –  I suppose we all need to learn through experiences. Feel free to share yours! #taboola

10

This is important as a design strategy when looking at agents. An agenting system, is, in essence, a workflow with branch points and conditional operations. We worked with something very similar in the XML space around fifteen years ago with XProc, which used an XML-based language to run transformations, queries, and validators. Such workflows can get to be quite complex. There have similarly been workflows within the RDF space, and you can build SHACL based workflows of some complexity on the semantic side (I've seen some based on prov-o as well).

5