A neural network made up of billions or even billions of parameters requires resources in the tens of millions of Euros.
With such huge costs, the only actors able to build and train a great model of artificial intelligence are the multinationals.
With these characteristics, the cost of AI research constitutes a barrier to entry.
Over the last few years we have witnessed the proliferation of smaller and less complex models, however far from the large LLM linguistic models.
In recent years we have seen a turnaround. For example, Meta's release of OPT-175B (Open Pretrained Transformer), a language model trained with public datasets and made available to researchers in “quasi” open mode.
But the news of the moment is the release of BLOOM LM by BigScience.
BLOOM is an open access multilingual language model that contains 176 billion parameters and has been trained for 3,5 months on 384 A100–80 GB GPUs.
A BLOOM checkpoint takes up 330GB of disk space, so it seems impossible to run this model on a desktop computer.
However, you just need enough disk space, and at least 16GB of RAM to run this model on your computer.
BLOOM is a collaborative effort of over 1.000 scientists.
It is important that such a broad multilingual model is openly available to all.
BLOOM is a causal model language, which means it has been trained as a predictor of the next token.
This seemingly simple strategy of predicting the next token in a sentence, based on a set of previous tokens, has been shown to capture a certain degree of reasoning power for large language models.
This allows BLOOM and similar models to connect multiple concepts in a sentence and to be able to solve non-trivial problems such as arithmetic, translation and programming with fair accuracy.
BLOOM uses a Transformer architecture consisting of an input embedding layer, 70 Transformer blocks, and an output language modeling layer, as shown in the following figure.
Article extracted from the Post of Luca Sambucci, if you want to read theNews.AI
An ophthalmoplasty operation using the Apple Vision Pro commercial viewer was performed at the Catania Polyclinic…
Developing fine motor skills through coloring prepares children for more complex skills like writing. To color…
The naval sector is a true global economic power, which has navigated towards a 150 billion market...
Last Monday, the Financial Times announced a deal with OpenAI. FT licenses its world-class journalism…
