Challenges/Opportunities to Enable Dependable Scale-out System with Groq Deterministic Tensor-Streaming Processors

Abstract

The Groq tensor streaming processor (TSP) architecture is an hardware accelerator that provides a new paradigm for achieving both flexibility and massive parallelism without the limitations and communication overheads of conventional CPU or GPU architectures. In particular, the Groq TSP execution and performance is deterministic as the flow of instructions through the the hardware is completely orchestrated and scheduled, making processing both efficient and predictable. When scaling out to multiple TSPs, the same deterministic execution needs to be provided in a distributed system. In this work, we explore the challenges and opportunities to scale such deterministic architecture across multiple processors to ensure a dependable scale-out system. In particular, maintaining synchronization across multiple TSPs to enable a software-scheduled network is a challenge; yet, the high-radix, low diameter topology enables N + 1 redundancy to improve the reliability of the system.