Ruoteng LI

Understanding the use of multimodality in RAG based system can help a lot in making business grow and most definitely there could be other uses cases as well. I would like to thank Mr. Andrew Ng for creating this beautiful short course that help understand a lots of things around RAG and its multimodality.

12

1 Comment

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

💥💥💥 DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search Huajian Xin, Z.Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z.F. Wu, Fuli Luo, Chong Ruan Abstract We introduce DeepSeek-Prover-V1.5, an open-source language model designed for theorem proving in Lean 4, which enhances DeepSeek-Prover-V1 by optimizing both training and inference processes. Pre-trained on DeepSeekMath-Base with specialization in formal mathematical languages, the model undergoes supervised fine-tuning using an enhanced formal theorem proving dataset derived from DeepSeek-Prover-V1. Further refinement is achieved through reinforcement learning from proof assistant feedback (RLPAF). Beyond the single-pass whole-proof generation approach of DeepSeek-Prover-V1, we propose RMaxTS, a variant of Monte-Carlo tree search that employs an intrinsic-reward-driven exploration strategy to generate diverse proof paths. DeepSeek-Prover-V1.5 demonstrates significant improvements over DeepSeek-Prover-V1, achieving new state-of-the-art results on the test set of the high school level miniF2F benchmark (63.5%) and the undergraduate level ProofNet benchmark (25.3%). 👉 https://lnkd.in/dgG9eqk7 👉 https://lnkd.in/dRvTVe3p #machinelearning

1

Super excited to be at #CVPR2024 this week to meet colleagues and absorb the latest in computer vision and genAI. Happy to be part of the following contributions by our amazing team Yafei M., Sunil Hadap, Yuelong Li, Brandon Smith, Xuelu Li   1 “MRC-Net: 6-DoF Pose Estimation with MultiScale Residual Correlation”, in main conference, presents a novel single-shot approach to determining 3D pose of multiple objects from a single RGB image, that is robust to severe occlusions and symmetry ambiguity.   2. “A Simple Strategy for Body Estimation from Partial-View Images”, at Workshop on Computer Vision for Fashion, Art, & Design, presents a method to predict 3D human body measurements from a photo, overcoming scale ambiguity when only a part of the body is visible. And check out these other awesome papers by Amazon scientists: amzn.to/3VhrR0i  

119

2 Comments

Microsoft Research and Tsinghua University researchers have introduced a novel architecture: You Only Cache Once (YOCO): Decoder-Decoder Architectures for Large Language Models. Architecture Overview: ·      YOCO consists of two main components: a self-decoder and a cross-decoder. ·      The self-decoder efficiently encodes global key-value (KV) caches, which are then reused by the cross-decoder through cross-attention. Caching Strategy: ·      YOCO caches key-value pairs only once, which significantly reduces memory requirements. ·      The self-decoder handles the initial encoding of KV caches, and the cross-decoder benefits from these cached values during cross-attention. Research Paper: https://lnkd.in/dQWXtpQ8

3