Show HN: Tune LLaMa3.1 on Google Cloud TPUs https://ift.tt/hfwpZkF

Show HN: Tune LLaMa3.1 on Google Cloud TPUs Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla integration with pytorch is clunky, missing libraries (bitsandbytes didn't work), and cryptic HuggingFace errors. We then took a different route and translated Llama 3.1 from PyTorch to JAX. Now, it’s running smoothly on TPUs! We still have challenges ahead, there is no good LoRA library in JAX, but this feels like the right path forward. Here's a demo ( https://ift.tt/GIPDeVS ) of our managed solution. Would love your thoughts on our repo and vision as we keep chugging along! https://ift.tt/NtfgFRy September 11, 2024 at 08:44PM

Comments

Post a Comment

Popular posts from this blog

Show HN: Agent File (.af) – An open file format for agents https://ift.tt/fzI5HcG

Show HN: Agent File (.af) – An open file format for agents Hi HN - We’re building Agent File (.af), which makes it possible to re-create the exact same agent (including memories, tools, message history, configs, etc.) across different machines. A big difference between LLMs and agents is that agents have associated state: system prompts, editable memory (personality and user information), tool configurations (code and schemas), and LLM/embedding model settings. While you can run the same LLM as someone else by downloading the weights, there’s no “representation” of agents that allows you to re-create an instance of an agent across services. Agent File (.af) is an open standard file format for serializing stateful agents. Originally designed for the Letta framework, .af is a human-readable representation of all the associated state of an agent to reproduce the exact behavior and memories. To demonstrate .af, we also made a few example agents with download links to .af: - MemGPT: An agen...
Read more

Show HN: Sort lines semantically using llm-sort https://ift.tt/7vEeHKP

Show HN: Sort lines semantically using llm-sort This is a small plugin I made for Simon Willison's llm utility. You can do things like: cat names.txt | llm sort -q "Which one of these names is best for a pet seagull?" cat books.txt | llm sort -q "Which book is more related to basic vs. advanced CS topics?" I see a lot of potential marrying LLMs with classic UNIX interfaces. https://ift.tt/9GDceOu February 11, 2025 at 08:55AM
Read more

Show HN: AgentKit – JavaScript Alternative to OpenAI Agents SDK with Native MCP https://ift.tt/H4Kz5Vi

Show HN: AgentKit – JavaScript Alternative to OpenAI Agents SDK with Native MCP Hi HN! I’m Tony, co-founder of Inngest. I wanted to share AgentKit, our Typescript multi-agent library we’ve been cooking and testing with some early users in prod for months. Although OpenAI’s Agents SDK has been launched since, we think an Agent framework should offer more deterministic and flexible routing, work with multiple model providers, embrace MCP (for rich tooling), and support the unstoppable and growing community of TypeScript AI developers by enabling a smooth transition to production use cases. This is why we are building AgentKit, and we’re really excited about it for a few reasons: Firstly, it’s simple. We embrace KISS principles brought by Anthropic and HuggingFace by allowing you to gradually add autonomy to your AgentKit program using primitives: - Agents: LLM calls that can be combined with prompts, tools, and MCP native support. - Networks: a simple way to get Agents to collaborate wit...
Read more