#

llm-inference

Here are 35 public repositories matching this topic...

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

ai-chat llm-inference

Updated May 27, 2025
C++

openvinotoolkit / openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

nlp natural-language-processing ai computer-vision deep-learning transformers inference speech-recognition yolo recommendation-system performance-boost good-first-issue openvino diffusion-models stable-diffusion generative-ai llm-inference optimize-ai deploy-ai

Updated Jun 21, 2025
C++

SJTU-IPADS / PowerInfer

High-speed Large Language Model Serving for Local Deployment

llama large-language-models llm local-inference llm-inference

Updated Feb 19, 2025
C++

distributed-llama

b4rtaz / distributed-llama

Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

neural-network distributed-computing llm llms open-llm llm-inference llama2 distributed-llm llama3

Updated Jun 16, 2025
C++

lean-dojo / LeanCopilot

LLMs as Copilots for Theorem Proving in Lean

machine-learning theorem-proving lean formal-mathematics lean4 llm-inference

Updated Jun 12, 2025
C++

cactus-compute / cactus

Framework for running AI locally on mobile devices and wearables. Hardware-aware C/C++ backend with wrappers for Flutter & React Native. Kotlin & Swift coming soon.

Updated Jun 21, 2025
C++

Lizonghang / prima.cpp

prima.cpp: Speeding up 70B-scale LLM inference on low-resource everyday home clusters

llama-cpp llm-inference on-device-llms distributed-ai distributed-inference

Updated Jun 20, 2025
C++

zhihu / ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

cuda pytorch llama gpt inference-engine model-serving llm llm-serving llm-inference deepseek-r1

Updated May 16, 2025
C++

foldl / chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

llm llm-inference

Updated Jun 21, 2025
C++

deeppowers / deeppowers

DEEPPOWERS is a Fully Homomorphic Encryption (FHE) framework built for MCP (Model Context Protocol), aiming to provide end-to-end privacy protection and high-efficiency computation for the upstream and downstream ecosystem of the MCP protocol.

ai accelerator llms llm-inference

Updated Apr 25, 2025
C++

vectorch-ai / ScaleLLM

A high-performance inference system for large language models, designed for production environments.

performance gpu model production cuda efficiency inference transformer llama speculative serving llm llm-inference llama3

Updated Jun 21, 2025
C++

andrewkchan / yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

machine-learning cpp cuda llama mistral inference-engine llm llamacpp llm-inference

Updated Jun 7, 2025
C++

zjhellofss / KuiperLLama

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

cpp cuda inference-engine llm llm-inference llama2 qwen qwen2 llama3

Updated Jun 18, 2025
C++

intel / neural-speed

An innovative library for efficient LLM inference via low-bit quantization

Updated Aug 30, 2024
C++

andrewkchan / deepseek.cpp

CPU inference for the DeepSeek family of large language models in C++

machine-learning cpp transformers llama llm llm-inference deepseek

Updated Jun 5, 2025
C++

inferflow / inferflow

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

bloom falcon moe gemma mistral mixture-of-experts model-quantization multi-gpu-inference m2m100 llamacpp llm-inference internlm llama2 qwen baichuan2 mixtral phi-2 deepseek minicpm

Updated Mar 15, 2024
C++

bytedance / ABQ-LLM

An acceleration library that supports arbitrary bit-width combinatorial quantization operations

research cuda mlsys quantized-networks llm-inference

Updated Sep 30, 2024
C++

sophgo / LLM-TPU

Run generative AI models in sophgo BM1684X/BM1688

large-language-models llm generative-ai llm-inference llama2 qwen2 bm1684x llama3 qwen3 qwen2-5-vl bm1688

Updated Jun 20, 2025
C++

Adriankhl / godot-llm

LLM in Godot

gamedev game-development godotengine godot godot-engine gdextension llamacpp llm-inference

Updated Jun 23, 2024
C++

powerserve-project / PowerServe

High-speed and easy-use LLM serving framework for local deployment

smartphone llama npu llm llm-serving llm-inference qwen smallthinker

Updated Mar 18, 2025
C++

Improve this page

Add a description, image, and links to the llm-inference topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-inference topic, visit your repo's landing page and select "manage topics."