A computer tablet forms an AI cluster, which can run a 400B model at home, and has 2.5K stars on GitHub
Cressy from Aofei Temple
Quantum Bit | Public Account QbitAI
Without H100, three Apple computers can drive the 400B large model.
The hero behind this is an open source distributed AI reasoning framework on GitHub, which has received 2.5k stars.
Using this framework, you can build your own AI computing cluster using everyday devices such as iPhone and iPad in just a few minutes.
This framework is called exo. Different from other distributed reasoning frameworks, it uses a p2p connection method, and the device can automatically join the cluster by connecting it to the network.
The developer used the exo framework to connect two MacBook Pros and a Mac Studio, achieving a computing speed of 110TFLOPS.
At the same time, the developer said that he is ready for the upcoming Llama3-405B.
EXO officials also announced that they will provide support for Llama3-405B as soon as possible (day 0) .
And it's not just computers. Exo can also allow iPhones, iPads and other devices to join the local computing network, and even Apple Watch can also be absorbed.
With the iteration of versions, the exo framework is no longer limited to Apple (initially only supported MLX) , and some people have also pulled Android phones and 4090 graphics cards into the cluster.
Complete configuration in as little as 60 seconds
Unlike other distributed reasoning frameworks, exo does not use a master-worker architecture, but connects devices point-to-point (p2p) .
As long as the device is connected to the same local area network, it can automatically join the exo computing network to run the model.
When splitting the model across devices, exo supports different partitioning strategies, the default is ring memory weighted partitioning.
This runs inference in a ring, with each device running a number of model layers proportional to the device memory.
And the whole process Almost no manual configuration is required . After installation and startup, the system will automatically connect to devices running in the local area network, and will support Bluetooth connection in the future.
In one of the author's videos, the configuration was completed on two new MacBooks in just about 60 seconds.