Opening a New AI Era with Super-efficient AI Chips

Whether it’s facial recognition-enabled mobile devices or the driverless vehicles that are increasingly ferrying us safely from A to B, AI-empowered applications are transforming nearly every aspect of our life. While significantly making our life smarter, AI carries a hefty price tag. Large computing and memory resources, plus lengthy training periods for AI models, being just two of the major expenses that are limiting the technology’s more widespread uptake.

Founded in 2020 by HKUST in collaboration with Stanford University, The University of Hong Kong, and The Chinese University of Hong Kong, the pioneering multidisciplinary AI Chip Center for Emerging Smart Systems (ACCESS) is committed to making AI applications more ubiquitous. Led by HKUST’s Vice President for Research and Development, Prof. Tim CHENG Kwang-Ting, ACCESS brings together a group of R&D specialists to fuel AI’s global expansion by building novel AI chips that improve performance and energy efficiency by up to 1,000 times.

Optimizing hardware

To stay ahead of the rapid proliferation of AI applications, ACCESS’ chip designers are now working 24/7 to innovate chips capable of processing these incredibly complex algorithms. However, the ordinary computing chips we use on our personal computers, such as CPUs and GPUs, have proved incapable in this regard – they are either too slow to properly run an AI algorithm, or are overheating and energy inefficient to an extent that undermines application’s durability and performance.

Also known as AI accelerators, customized computing chips for AI applications have now emerged as the technology’s gold standard hardware solution.

“Companies want to cram more and more intelligence into today’s sensors and devices to enhance their functions,” says Prof. Cheng, the Center’s founding director and an internationally recognized leader in electronic design automation, integrated circuit design, and computer vision. “They need powerful but small, energy-efficient AI chips to carry out specific and ubiquitous tasks. But such embedded intelligence is not yet widely available,” he adds.

Prof. Cheng says while tech giants such as Apple, Tesla, and drone-maker DJI have the financial clout needed to customize AI chips for exclusive use in their own devices, such resources are beyond most companies. Smaller firms seeking AI-enabled chips have to share their algorithms and specifications with external chip developers, but, rather than optimizing for a certain AI application and algorithm, those chips tend to cover a wide range of applications.

“This means client companies may only use a part of the purchased chip’s functionality and potential performance, resulting in mismatch, wasted resources, and inefficiency,” he says. “It also makes it harder for companies to differentiate themselves from their competitors’ AI products.”

Ultra-efficient customized AI chips     

This is where Prof. Cheng believes ACCESS’ chip designers can make a difference by pioneering open-source methodologies and AI-assisted automation tools early on. Such an edge will simplify startups and smaller companies’ sourcing of chips more precisely tailored to their specific needs. In addition to offering an alternative to the current AI ecosystem, ACCESS also intends to slash chip design cycles and so better meet stringent time-to-market requirements.

Take the team’s recently built AI acceleration Field Programmable Gate Array (FPGA) prototyping system as an example. By swiftly developing and debugging prototype designs, this innovative solution delivers fully ten times the energy efficiency and treble the productivity of existing methodologies. “Our goal is to produce customized AI chips with a team of tens of people within one to two quarters, instead of requiring years and hundreds of people,” Prof. Cheng says.

And speeding the pace of electronic design automation is just one of ACCESS’ four major technical arms. The Center is also busy researching three other key technical areas: enabling technology for emerging computer systems, architecture and heterogeneous system integration, and hardware-accelerated AI applications. With 14 projects now underway, ACCESS’ target of eliminating AI hardware design and performance bottlenecks in areas such as memory/data bandwidth limitations, speed and energy efficiency is already on track.

Another major advance the team is developing is a new generation of ultra-low-power computing in memory (CIM) chips. When marketed, the finished products will serve as independent data processors and eliminate time- and energy-consuming off-chip data communications.

“Currently, data have to be sent to the cloud or a central server for analysis and processing before they are returned to the device,” Prof. Cheng explains. “By confining the analysis and processing to the device itself, the CIM chips can operate hundreds of times faster and with higher energy efficiency.” Recent literature and results indicate that ACCESS chips are three times more efficient than today’s best-performing CIM chips.

Prof. Cheng says such ultra-low-power chips will be ideal for a wide range of daily applications. Specific examples include the traffic-counting cameras that indoor venues and public areas use to monitor footfall levels, plus smartwatches that track variability in wearers’ heart rates and other key health indicators.

A vast potential global market

With the global market for AI estimated to exceed US$290 billion by 2026, Prof. Cheng believes ACCESS’ transnational R&D consortium status will firmly establish Hong Kong’s standing on AI hardware’s world stage. The pioneering center is being supported under the Hong Kong government’s InnoHK Clusters initiative, receiving HK$443.9 million in initial funding.

In all, some 10 local and global companies in key fields such as medical, health and consumer electronics, smart manufacturing, and smart city design have already expressed interest in ACCESS’ AI chip technologies. “Just think of all the sectors wanting such AI acceleration, from transportation to fintech, medicine to education,” Prof. Cheng says. “It represents a massive opportunity for research, business, and social impact.”

Prof. Cheng firmly believes that ACCESS’ biggest strength remains its fully integrated, end-to-end approach to R&D. “Other research centers are not connecting all the different dots together as a unified center and can only do part of what ACCESS can do,” he points out.

The Center’s full spectrum ethos also extends to its broad talent recruitment strategies. The end result is an unusually broad array of upstream and downstream expertise spanning everything from software, hardware, chip structure and design to AI algorithms and applications. Some 20+ seasoned researchers and engineers have already signed up and they are supported by around 140 top faculty members and doctoral researchers from the project’s four participating universities. By 2027, the Center is looking to recruit some 100 graduate students and provide them with the opportunity to enhance their cross-domain research and innovation capabilities.

“The full-time staff members work closely with international experts and industrial partners on the leading-edge AI chip technologies, and will widen and deepen their skills through activities including seminars, workshops, and training sessions hosted by the Center and other experts,” says ACCESS’ Executive Director, Dr. LIANG Luhong. Dr. Liang joined the Center in November 2020 after a nearly ten-year career as principal engineer and director of Machine Learning Platforms at the Hong Kong Applied Science and Technology Research Institute (ASTRI). “The Center has set up complete ladders for talents with various interests, capabilities, and work experiences to reach their full potential of innovation and career aspirations.”

Dr. Liang adds that an engineer-in-residence initiative via which industrial partners can assign key talents to ACCESS to work on particular projects for a certain period is also in the pipeline. Aimed at upgrading participants’ technical capabilities, the program will be a key part of ongoing efforts to foster mutually beneficial ties between ACCESS and AI industry stakeholders.

Prof. Cheng adds that closer collaboration promises significant benefits for both parties. Companies will be able to more easily source ACCESS researchers’ expert opinion on AI algorithms and chip architecture, while ACCESS itself can learn more about the latest developments and requirements of the AI application market. “The need is mutual. We need to reach out for new applications and impact, and companies seeking applications already have customers that will be the beneficiaries,” the professor says.

Such co-operation will also see cutting-edge integrated circuit research from the university being commercialized to power AI chip design. “ACCESS has been conceived to remove the ‘valley of death’ — the gap between university research and industrial deployment,” he adds. “Universities can turn their concepts into demos and production, while industry can find ways to apply the demos.”

“There are great customers, great technologies, and the next stage is to tie these together with the AI hardware,” Prof. Cheng concludes. “We are in the right place at the right time!”