colibri
The colibri README describes a lightweight C inference engine that runs the 744-billion-parameter GLM-5.2 mixture-of-experts model on consumer machines with roughly 25 GB of RAM. The engine targets developers seeking to execute frontier-scale models without cluster infrastructure. It keeps dense components—such as attention layers and embeddings—resident in memory at 4-bit integer precision (~9.9 GB) while dynamically streaming routed experts from an NVMe drive on demand. The architecture supports multi-token prediction speculative decoding, compressed key-value caching, and an optional single-process HTTP gateway compatible with the OpenAI chat completions format.
Adoption requires significant local hardware preparation and patience with throughput limitations. Users must supply Linux or WSL2 with AVX2 support, GCC with OpenMP, and roughly 370 GB to 400 GB of NVMe storage on an ext4 filesystem. Cold token decoding is heavily disk-bound, with project documentation noting cold rates around 0.05 to 0.1 tokens per second on restricted setups. Runtime execution is purely C-based, though the initial one-time model conversion script requires Python and PyTorch. An experimental CUDA backend supports offloading resident tensors, while macOS testing remains community-reported. The project is released under the Apache 2.0 license.
rowboat
rowboat is a local-first desktop assistant designed for technical creators who need an integrated workspace that connects daily desktop context with language models. The software organizes communication, meeting transcripts, and project notes into an Obsidian-style backlinked knowledge graph stored entirely on the user's filesystem. Built-in workspaces include an isolated web browser for scoped assistant logins, an email triage interface, markdown meeting notes that capture system audio, and parallel coding agent drivers compatible with Codex and Claude Code.
The system emphasizes privacy and interoperability, storing context in standard Markdown files and connecting with local model engines like Ollama and LM Studio as well as remote provider APIs. However, extended features require individual third-party integrations and API keys. Meeting transcription, voice generation, and specialized web discovery require separate configurations for services such as Deepgram, ElevenLabs, or Exa. External tool integration relies on the Model Context Protocol (MCP) or Composio. Binaries are provided for macOS, Windows, and Linux under the Apache 2.0 license.