Ramón Fernández Astudillo

This article by Pere Martra explores a structured pruning technique for state-of-the-art models, that uses a GLU architecture, enabling the creation of smaller and more efficient large language models. #Llama #LLM

5

Thrilled to share a groundbreaking paper published by colleagues, Henrik Dreyer and Etienne Granet here at Quantinuum, featured in Nature's NPJ Quantum Information. This innovative research presents a novel method for simulating Hamiltonian dynamics on digital #quantumcomputers, bypassing the need for Trotterization – a standard but computationally expensive technique. Why is this important? The ability to simulate quantum systems more efficiently opens up new possibilities for quantum research and practical applications, especially in fields like #materialsscience, #chemistry, and #drugdiscovery, where understanding complex systems at the quantum level can lead to breakthroughs. Link to the article: https://lnkd.in/etS52Mer

63

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

Proud of Owkin K OS latest work: FedPyDESeq2, a privacy-enhancing tool for differential expression analysis in #federatedlearning, is now available on bioRxiv. FedAI breaks research and data silos for an open ecosystem of research. Differential gene expression analysis (DGEA) is one of the most widely used computational techniques to reveal gene expression differences between two or more conditions, a key step to discover new biology and further our understanding of diseases. However, large-scale gene expression datasets, are often stored behind hospitals and institutes' firewalls, limiting opportunities to run large studies across them. Well, no more. FedPyDESeq2, based on Owkin's Substra, allows to perform DGEA in a real-world, federated setting. We rigorously demonstrate that it does so while maintain almost-identical performances to a pooled setting. And yes, it's open-source - because this is what we do Check out the code here: https://lnkd.in/ghSPXARf Congrats to our amazing scientists Boris Muzellec, Ulysse MARTEAU-FEREY and Tanguy Marchand under Eric Durand scientific supervision. Read all about our method here: https://lnkd.in/gFhN679y Importantly, FedPyDESeq2 is built on PyDESeq2, our open-source python implementation of the famous DESeq2 method first introduced by Love, Huber and Anders (2014). #goowkin

55

1 Comment

CAT-BENCH: Evaluating Language Models Understanding of Temporal Dependencies in Procedural Texts Understanding how LLMs comprehend natural language plans, such as instructions and recipes, is crucial for their dependable use in decision-making systems. A critical aspect of plans is their temporal sequencing, which reflects the causal relationsh... https://lnkd.in/eZwW7CAe #AI #ML #Automation

1

Quantum computing is difficult stuff. I am glad we have colleagues like Paul who can explain it to us. Have a look at this great blog.

14

This morning, I read an interesting article: "Benchmarking Optimizers for Qumode State Preparation with Variational Quantum Algorithms" by Shuwen Kan et al. Abstract: —Quantum state preparation involves preparing a target state from an initial system, a process integral to applications such as quantum machine learning and solving systems of linear equations. Recently, there has been a growing interest in qumodes due to advancements in the field and their potential applications. However there is a notable gap in the literature specifically addressing this area. This paper aims to bridge this gap by providing performance benchmarks of various optimizers used in state preparation with Variational Quantum Algorithms. We conducted extensive testing across multiple scenarios, including different target states, both ideal and sampling simulations, and varying numbers of basis gate layers. Our evaluations offer insights into the complexity of learning each type of target state and demonstrate that some optimizers perform better than others in this context. Notably, the Powell optimizer was found to be exceptionally robust against sampling errors, making it a preferred choice in scenarios prone to such inaccuracies. Additionally, the Simultaneous Perturbation Stochastic Approximation optimizer was distinguished for its efficiency and ability to handle increased parameter dimensionality effectively. Link: https://lnkd.in/dkB8hBHC #quantumalgorithms #quantummachinelearning #quantumcomputing #quantumoptimizers

30

A very interesting аrticle published in Nature about LLMs. https://lnkd.in/dms_2y7e So basically, the more complex the model, the more it pretends to know something rather than actually knowing it. This seems like an interesting amplification of an inherent flaw in LLMs, now scaled up in a way that's worth paying attention to. It will be interesting to see if anyone will manage to mitigate this. #artificial_intelligence #ai #llm #genAI

24

2 Comments

For his debut TDS article, Jeremi Nuer presents a thoughtful walkthrough of a recent paper on model interpretability: "We want to have finer-tuned control over LLMs. Being able to detect when certain features are active, and tune how active they are in the middle of generating output, is an amazing tool to have in our toolbox."

18

Are you interested in vector search, indexing, or database efficiency? Then you won't want to miss this talk by Etienne Dilocker from Weaviate at Carnegie Mellon University's Database Group with Andy Pavlo! Etienne breaks down key challenges and innovations that are powering the future of search: ✨ Bit-Sliced Indexing - tackling complex range queries like price filters with innovative indexing. ✨ BM25 Hybrid Search - combining keyword and vector search to handle nuanced queries. ✨ Graph-Based Indexing (HNSW) - optimizing for high update rates and recalls without latency drops. ✨ Overcoming the Minority Filter Problem - enhancing search relevance through in-place filtering, based on the ACORN paper. ✨ Weaviate's Roadmap - performance, enterprise readiness, and pioneering "Recommendations as a Service." Check out the full talk for an in-depth look at how Weaviate is setting new standards in search technology: https://lnkd.in/gavAZRrn

33

2 Comments

This paper has an interesting concept called PlanRAG, a novel method designed to enhance decision-making capabilities of large language models (LLMs) by structuring the process into two distinct steps: planning and retrieval. Initially, the model generates a decision-making plan, which is then followed by retrieving the necessary data to support those decisions. This structured approach helps the model handle complex tasks more effectively than the traditional agentic approach, where decisions are made directly without a clear plan. By separating planning and retrieval, PlanRAG ensures a more comprehensive and accurate decision-making process. The researchers tested PlanRAG using scenarios from video games and found it outperformed existing methods. They also introduced a new benchmark, Decision QA (DQA), to evaluate the decision-making abilities of LLMs in complex scenarios. The structured plan-then-retrieval method allowed the model to understand what decisions need to be made first and then gather the relevant information, leading to better performance. The study's results demonstrate that PlanRAG is a significant improvement over the agentic approach, making it a promising advancement for using LLMs in complex decision-making tasks. https://lnkd.in/gCbG9ie7

10

We've just published a new paper in the journal Physics Essays. It shows how the case is not closed for information processing of a quantum nature in the neurons of a brain. We argue for multi-dimensional quantum information processing based on the ultrastructure of electric near fields surrounding cellular tissue. We also show how the decoherence time may be of the order of the timescale of an action potential and not many orders of magnitude less as thought of earlier. https://lnkd.in/gxQs4dv4 Joint work with Professor Salvatore Domenic Morgera. #quantum, #brain, #information, #decoherence, #AI

1

Researchers at CDS have unveiled that transformer language models can solve complex problems without clear intermediate reasoning, challenging existing assumptions about their operation. Led by CDS PhD students Jacob Pfau and William Merrill, and CDS Associate Professor of Linguistics and Data Science Sam Bowman, this study highlights the use of nonsensical filler tokens that achieve perfect accuracy in specific tasks, raising concerns about  interpretability and accountability in LLMs. Explore the full findings: https://lnkd.in/eERzzKsU

13

Enhancing Mathematical Reasoning in LLMs: Integrating Monte Carlo Tree Search with Self-Refinement

5

#Performance of #LLMs. The adoption of #LLMs is rapidly growing, but the difficulty arises when trying to apply them beyond experimental settings. Deploying LLMs in an actual environment is complex, and one of the significant challenges is effectively evaluating their performance. Evaluation of LLMs can be based on different factors like toxicity, helpfulness, correctness and more. This article by Surya Prakash Pathak from Red Hat explores various evaluation techniques and metrics employed in assessing the performance of LLMs, with a particular emphasis on retrieval-augmented generation (#RAG) use cases. RAG, a pivotal use case for different enterprises, focuses on combining retrieval methods with the generation capabilities of LLMs. This unique approach involves not only generating content based on learned patterns but also leveraging pre-existing information through retrieval mechanisms. Scott Creeley :: Stephen Flaherty https://lnkd.in/gJTBagjv

13

2 Comments

Excellent writeup on ReFT by Ben Dickson. In this article, he discusses a novel approach to fine-tuning large language models, called Representation Finetuning (ReFT). ReFT is research outcome from Stanford University. Unlike traditional parameter-efficient fine-tuning (PEFT) methods that update a small number of weights, ReFT methods operate on a frozen base model and learn task-specific interventions on the hidden representations. The authors present a specific instance of the ReFT family, called Low-rank Linear Subspace ReFT (LoReFT), which is shown to be 10x-50x more parameter-efficient than state-of-the-art PEFT methods while delivering superior performance on a range of tasks, including commonsense reasoning, arithmetic reasoning, and the GLUE benchmark. The authors have released a generic ReFT training library publicly, making their approach accessible to the research community. Good writeup where you can quickly grab the key details and of course more details you can go the paper in arxiv. Happy Learning.

7