Reduce speech transcription costs by up to 90% using interactive artificial intelligence (CAI)

Figure 4: 2D NoC with 20 Tbps total bandwidth

Unlike competing solutions, the 2D NoC eliminates any bottlenecks between the memory and compute engines in the Speedster7t ASR solution, enabling optimal utilization of the hardware accelerators at these very low batch rates.

Putting all of these features together in a roofline diagram clearly illustrates the advantages of the Achronix Speedster7t devices over other competing FPGA solutions in low-latency ASR applications. The roofline diagram uses verified TOPS data published by each manufacturer to show what these devices can achieve in real-world applications.

The figure below shows a roofline model of effective TOPS using a subset of the code built by Achronix for microbenchmarks (GEMV and MLP) and tests, as well as published data from Company A [4] [5] and Company B (based on the architecture). The orange vertical line represents the sweet spot for batch sizes of 8 ms and 80 ms for audio blocks used in low latency, real-time ASR data streaming applications. At this sweet spot, Achronix achieves a 44% improvement in effective TOPS over Company A and a 260% improvement over Company B’s solution.

Figure 5: Roofline model of effective TOPS

Achieve a 90% reduction in ASR processing costs within one year

Most ASR solutions are provided by large cloud service providers such as Google, Amazon, Microsoft Azure, and Oracle. As operations scale and these products become more successful in the market, service providers building products on top of these cloud APIs face increasing cost pressures. Larger ASR providers publicly advertise costs ranging from $0.01 to $0.025 per minute [6], [7], [8], [9]. Industry reports indicate that the average call time in a call center is approximately 5 minutes. Consider a large enterprise data or call center service company that handles 50,000 calls per day, each lasting 5 minutes. Using these rates, the ASR processing cost would be $1,500 to $6,000 per day or $500,000 to $2 million per year. Solutions from Achronix and Myrtle.ai can support 4,000 RTS on a single accelerator card, which can handle over a million calls per day.

There are many factors that determine the cost of a standalone ASR device. In this specific example, let’s assume that the Achronix ASR acceleration solution is delivered via an FPGA-based PCIe card and integrated into a 2U server based on an x86 architecture. This device might be sold from a system integrator for $50,000, and the annual cost of running the server might be twice that. This would bring the first year cost of an on-premises ASR device to $100,000. Comparing this on-premises solution to a cloud API service, the end user could save 5 to 20 times the cost in the first year.

Table 1: Summary of comparison between Achronix ASR solution and cloud API service

The highly compact system enables enterprises to scale as their business grows without having to rely on increasingly expensive ASR cloud APIs or build out massive data center infrastructure to deliver on-premises solutions.

Summarize

The ASR function in CAI requires low-latency, high-throughput computation for RNN machine learning algorithms, which challenges modern AI accelerators. FPGA hardware accelerators that claim to have inference speeds of up to 150 TOPS encounter bottlenecks when transferring data between large computing engines and high-speed memories. These bottlenecks can cause hardware utilization to be as low as 5%. Achronix and Myrtle.ai have joined forces to launch an ASR platform consisting of a 200W, x16 PCIe Gen4 accelerator card and related software that can support up to 4,000 RTS simultaneously and process up to 1 million 5-minute transcription files every 24 hours. Comparing the cost of a PCIe accelerator card on a single x86 server to the cost of a cloud ASR service, the first year's capital expenditure (CAPEX) and operating cost (OPEX) can be reduced by up to 90%.

References: