gigatoken
The marcelroed/gigatoken repository targets high-throughput tokenization for language model pipelines. Built primarily with Rust and exposed through Python bindings, it suits machine learning engineers processing large-scale text corpora. The project provides a native API that accepts text file sources or Hugging Face model identifiers, as well as compatibility wrappers designed to mirror existing interfaces from Hugging Face tokenizers and tiktoken.
Practical adoption comes with documented constraints. The compatibility mode incurs performance penalties to match Hugging Face outputs, while the native API still carries Python iteration overhead and lacks file sinks. SentencePiece tokenization is unoptimized, WordPiece is unsupported, and Windows environments are not thoroughly validated, leaving the author to recommend WSL. The repository is published under the MIT license.
palmier-pro
The palmier-io/palmier-pro repository provides a Swift-native desktop video editor built for Apple hardware. Geared toward technical creators and developers automating media production, the application allows users to edit alongside AI agents. Running locally, it exposes an HTTP-based Model Context Protocol server at port 19789, enabling external agents like Claude Code, Codex, or Cursor to connect directly to the timeline.
Operating requirements and licensing divide the application's surface. Palmier Pro requires macOS 26 (Tahoe) running on Apple Silicon. While the core video editor, MCP server, and local agent chat are distributed under the GPLv3 license, the generative media processing features remain closed-source and require a user account and paid subscription.