FlashInfer: Kernel Library for LLM Serving
-
Updated
Mar 7, 2026 - Python
#
distributed-inference
Here are 16 public repositories matching this topic...
Performance-optimized AI inference on your GPUs. Unlock superior throughput by selecting and tuning engines like vLLM or SGLang.
cuda inference openai llama maas rocm ascend llm llm-serving vllm genai llm-inference qwen deepseek sglang distributed-inference high-performance-inference mindie
-
Updated
Mar 7, 2026 - Python
一個基於 llama.cpp 的分佈式 LLM 推理程式，讓您能夠利用區域網路內的多台電腦協同進行大型語言模型的分佈式推理，使用 Electron 的製作跨平台桌面應用程式操作 UI。
-
Updated
Aug 24, 2025 - JavaScript
Code for paper "JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks"
compression deep-learning distributed-computing inference dnn neural-networks pruning quantization dnn-optimization edge-computing mobilenet edge-cloud ai-models dnn-partition distributed-inference
-
Updated
Aug 24, 2025 - Python
Analyze and generate unstructured data using LLMs, from quick experiments to billion token jobs.
csv s3 pandas data-engineering parquet data-processing data-pipelines observability synthetic-data unstructured-data mlops batch-inference polars llm-inference evals distributed-inference
-
Updated
Mar 6, 2026 - Python
Accelerate reproducible inference experiments for large language models with LLM-D! This lab automates the setup of a complete evaluation environment on OpenShift/OKD: GPU worker pools, core operators, observability, traffic control, and ready-to-run example workloads.
-
Updated
Jan 21, 2026 - Python
Source code of the paper "Private Collaborative Edge Inference via Over-the-Air Computation".
machine-learning ensemble-learning wireless-communications differential-privacy multi-view-learning edge-inference over-the-air-computation collaborative-inference distributed-inference
-
Updated
Jan 14, 2025 - Python
Super Ollama Load Balancer - Performance-aware routing for distributed Ollama deployments with Ray, Dask, and adaptive metrics
python caching machine-learning streaming performance ai multi-node http2 load-balancer rpc asyncio ray model-serving gpu-monitoring inference-optimization llm llama-cpp ollama distributed-inference hybrid-routing
-
Updated
Nov 14, 2025 - Python
Official impl. of ACM MM paper "Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds". A distributed inference model for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system. Jointly training of pedestrian attribute recognition and Re-ID.
mec re-identification edge-computing re-id attribute-recognition mobile-edge-computing distributed-inference
-
Updated
Apr 26, 2020 - Python
Turn any Kubernetes Cluster into a private LLM endpoint. One Helm command deploys distributed inference across commodity hardware. Raspberry Pi's, old servers, mixed architectures. OpenAI-Compatible API Powered by llama.cpp RPC
docker kubernetes open-source raspberry-pi machine-learning ai gpu helm decentralized distributed-computing inference self-hosted homelab cluster-computing edge-computing large-language-models llm llm-inference ollama distributed-inference
-
Updated
Mar 4, 2026 - Dockerfile
Encrypted Decentralized Inference and Learning (E.D.I.L.)
deep-neural-networks deep-learning he federated-learning decentralized-training distributed-inference ratio1
-
Updated
Jan 20, 2026 - Python
A comprehensive framework for multi-node, multi-GPU scalable LLM inference on HPC systems using vLLM and Ollama. Includes distributed deployment templates, benchmarking workflows, and chatbot/RAG pipelines for high-throughput, production-grade AI services
hpc rag ai-inference vllm llm-inference ollama distributed-inference vllm-serve ai-inference-server multinode-training
-
Updated
Dec 10, 2025 - Python
Web UI for orchestrating distributed llama.cpp RPC GPU clusters with auto node discovery, telemetry, and one-click deployment.
python flask rpc rpc-server rpc-client gpu-cluster llm llama-cpp gguf distributed-inference ai-cluster gguf-models ai-cluster-automation
-
Updated
Mar 7, 2026 - JavaScript
Pool your CUDA + ROCm GPUs into one OpenAI-compatible API. Speculative decoding proxy gives you 2-3x faster inference — for free, using hardware you already own. Stop renting GPU clouds. Be a tightwad.
python open-source machine-learning ai amd gpu cuda inference self-hosted nvidia homelab rocm inference-engine llm llm-inference local-ai speculative-decoding distributed-inference openai-compatible ollama-alternative
-
Updated
Feb 20, 2026 - Python
🚀 Master GPU kernel programming and optimization for high-performance AI systems with this comprehensive learning guide and resource hub.
android java docker opengl gpu interview perf performance-analysis gpuimage puzzles profiling rocm ascend llm llm-serving genai llm-inference qwen distributed-inference mindie
-
Updated
Mar 7, 2026
Improve this page
Add a description, image, and links to the distributed-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the distributed-inference topic, visit your repo's landing page and select "manage topics."