Weishan DONG

Stanford Ph.D. researcher Cheng Chi discusses the development of the UMI gripper and the use of diffusion AI models for robotics. The post Stanford researcher discusses UMI gripper and diffusion AI models appeared first on The Robot Report. https://lnkd.in/gbTiRwh4

1

Label Studio Customized Backend for Semiautomatic Image Segmentation Labeling - Customized backend; GCP Deployment; Data Versioning with GCS Integration by Yuhan Yao

3

💥💥💥 GUI Agents with Foundation Models: A Comprehensive Survey Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, Ruiming Tang Abstract Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), facilitate intelligent agents being capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions by simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data, frameworks, and applications. We begin by discussing representative datasets and benchmarks. Next, we summarize a unified framework that captures the essential components used in prior research, accompanied by a taxonomy. Additionally, we explore commercial applications of (M)LLM-based GUI agents. Drawing from existing work, we identify several key challenges and propose future research directions. We hope this paper will inspire further developments in the field of (M)LLM-based GUI agents. 👉 https://lnkd.in/dtCrG5DB #machinelearning

1

One of the first packages for radar point clouds! It uses dynamic filtering to make usable point clouds for purposes like SLAM! Radar has many benefits over Lidar and cameras, primarily working in heavy weather conditions, like rain or snow. I believe this can be the future of self-driving cars, if the filtering is truly good!

28

1 Comment

💥💥💥 Buffer Anytime: Zero-Shot Video Depth and Normal from Image Priors Zhengfei Kuang, Tianyuan Zhang, Kai Zhang, Hao Tan, Sai Bi, Yiwei Hu, Zexiang Xu, Milos Hasan, Gordon Wetzstein, Fujun Luan Abstract We present Buffer Anytime, a framework for estimation of depth and normal maps (which we call geometric buffers) from video that eliminates the need for paired video--depth and video--normal training data. Instead of relying on large-scale annotated video datasets, we demonstrate high-quality video buffer estimation by leveraging single-image priors with temporal consistency constraints. Our zero-shot training strategy combines state-of-the-art image estimation models based on optical flow smoothness through a hybrid loss function, implemented via a lightweight temporal attention architecture. Applied to leading image models like Depth Anything V2 and Marigold-E2E-FT, our approach significantly improves temporal consistency while maintaining accuracy. Experiments show that our method not only outperforms image-based approaches but also achieves results comparable to state-of-the-art video models trained on large-scale paired video datasets, despite using no such paired video data. Website 👉 bufferanytime.github.io Paper 👉 arxiv.org/pdf/2411.17249 #machinelearning

2

1 Comment

📌 How would you like to know the AI expert, Dr. Andrew Ng's insights on the next wave of AI trends for Taiwan's industries? For those who missed the live forum, you're in luck! Here’s your chance to catch up! The highly influential figure in AI (TIME100 AI), Silicon Valley AI leader, startup pioneer, and Managing General Partner of AI Fund, Dr. Andrew Ng, has brought his keynote speech titled AI, Agents, and Applications online!! 🥳 Don’t miss this chance to explore the latest AI trends — tune in and share your insights in the comments! 💬 🔻 Watch the Keynote🔻 https://reurl.cc/6dpQb6 #GenAI #AIAgents #ai #AIFund #LandingAI #AndrewNg #AgenticAI

25

Check out this #OA paper, “Research on Animation Character Expression Generation Based on Attention Conditioned Cyclegan”, by Chunling Kong (Zhengzhou University of Industrial Technology, China), featured in the Journal of Cases on Information Technology (JCIT). With advancements in computer vision and AI, deep learning is applied in animation and game design to enhance facial expression realism. This study proposes ACCGAN, an innovative method for automated animated character expression generation and transfer using deep learning. Access the full paper here > https://buff.ly/4eMhOsv #ZhengzhouUniversityofIndustrialTechnology #CharacterExpression #GenerativeAdversarialNetworks

1

Huge release by Daekeun Kim and Hyo (HK) Choi from our AI GBB team in Korea, sharing Korean language benchmarks for our latest Azure OpenAI model, gpt-4o-mini - like gpt-4o, the model excels on Korean language tasks, in this case mini completely eclipses gpt-35-turbo and nearly achieves gpt-4-turbo performance. The benchmarks and code are open source, extensible beyond our Azure models, so we encourage our customers and the larger community to try it out and even fork/share your own findings - and please let me in the comments what other Asia languages beyond Korean you may be looking to evaluate with your language models.

52

7 Comments

💥💥💥 LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang Abstract Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. 👉 https://lnkd.in/d7PaajBQ #machinelearning

1

Missed our seminar on Graph Neural Networks for Time Series Analysis? Good news! The full recording of the event, where Ming Jin from Griffith University provided an in-depth review of GNNs and their applications in traffic management, energy forecasting, and finance, is now available on my YouTube channel. If you're curious about how GNNs can elevate your time series analysis or want to revisit some of the key insights, head over to the link below https://lnkd.in/gExkNxCc

16

Iron-shelled robo-snails swarm together for off-road tasks. Groups of these bots can operate independently or join together to perform tasks that would otherwise be impossible. Authors: Da Zhao, Haobo Luo, Yuxiao Tu, Chongxi Meng, Tin Lun Lam Corresponding author: Tin Lun Lam -------------------------------- Stay ahead of the curve! Follow us now on our WhatsApp (https://lnkd.in/euHisPiQ) and Telegram (https://wevlv.co/3sJlFn5) channels and stay updated about the cutting edge.

30

💥💥💥 Instruction-Following Pruning for Large Language Models Bairu Hou, Qibin Chen, Jianyu Wang, Guoli Yin, Chong Wang, Nan Du, Ruoming Pang, Shiyu Chang, Tao Lei Abstract With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model. 👉 https://lnkd.in/dt8AnGYy #machinelearning

1

🌟 Boost Meta's Llama 3.3 70B inference throughput by 3x with Tensor RT-LLM optimizations including speculative decoding, in-flight batching, KV caching, custom FP8 quantization, and more.🦙 Technical deep dive ➡️

8

💥💥💥 A Survey for Large Language Models in Biomedicine Chong Wang, Mengyao Li, Junjun He, Zhongruo Wang, Erfan Darzi, Zan Chen, Jin Ye, Tianbin Li, Yanzhou Su, Jing Ke, Kaili Qu, Shuxin Li, Yi Yu, Pietro Liò, Tianyun Wang, Yu Guang Wang, Yiqing Shen Abstract Recent breakthroughs in large language models (LLMs) offer unprecedented natural language understanding and generation capabilities. However, existing surveys on LLMs in biomedicine often focus on specific applications or model architectures, lacking a comprehensive analysis that integrates the latest advancements across various biomedical domains. This review, based on an analysis of 484 publications sourced from databases including PubMed, Web of Science, and arXiv, provides an in-depth examination of the current landscape, applications, challenges, and prospects of LLMs in biomedicine, distinguishing itself by focusing on the practical implications of these models in real-world biomedical contexts. Firstly, we explore the capabilities of LLMs in zero-shot learning across a broad spectrum of biomedical tasks, including diagnostic assistance, drug discovery, and personalized medicine, among others, with insights drawn from 137 key studies. Then, we discuss adaptation strategies of LLMs, including fine-tuning methods for both uni-modal and multi-modal LLMs to enhance their performance in specialized biomedical contexts where zero-shot fails to achieve, such as medical question answering and efficient processing of biomedical literature. Finally, we discuss the challenges that LLMs face in the biomedicine domain including data privacy concerns, limited model interpretability, issues with dataset quality, and ethics due to the sensitive nature of biomedical data, the need for highly reliable model outputs, and the ethical implications of deploying AI in healthcare. To address these challenges, we also identify future research directions of LLM in biomedicine including federated learning methods to preserve data privacy and integrating explainable AI methodologies to enhance the transparency of LLMs. 👉 https://lnkd.in/dRPWGtBH #machinelearning #biomedicine #innovation

2

💥💥💥 Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts Ming Wang, Yuanzhong Liu, Xiaoyu Liang, Yijie Huang, Daling Wang, Xiaocui Yang, Sijia Shen, Shi Feng, Xiaoming Zhang, Chaofeng Guan, Yifei Zhang Abstract LLMs have demonstrated commendable performance across diverse domains. Nevertheless, formulating high-quality prompts to assist them in their work poses a challenge for non-AI experts. Existing research in prompt engineering suggests somewhat scattered optimization principles and designs empirically dependent prompt optimizers. Unfortunately, these endeavors lack a structural design, incurring high learning costs and it is not conducive to the iterative updating of prompts, especially for non-AI experts. Inspired by structured reusable programming languages, we propose LangGPT, a structural prompt design framework. Furthermore, we introduce Minstrel, a multi-generative agent system with reflection to automate the generation of structural prompts. Experiments and the case study illustrate that structural prompts generated by Minstrel or written manually significantly enhance the performance of LLMs. Furthermore, we analyze the ease of use of structural prompts through a user survey in our online community. 👉 https://lnkd.in/d5Tkz7E2 #machinelearning

4

1 Comment