Mixtral 8x7B: A very interesting and powerful Language Model by Mistral AI

What is Mixtral 8x7B?

A new open source model called Mixtral 8x7B has been revealed in the research paper “Mistral of Experts” by a French company called Mistral AI. This model showcases significant improvement in natural language processing by making use of Sparse Mixture of Experts (SMoE) technology. Mixtral 8x7B is an extension of the Mistral 7B model.

Why 8x7B, not 56 B parameters? 

Mixtral 8x7B has eight feedforward blocks (experts) in each layer that help it process tokens efficiently and dynamically. This unique architecture makes Mixtral a powerful tool for language modeling. Using a subset of parameters for each token, Mixtral achieves faster inference at low batch sizes and higher throughput at large batch sizes.

It is interesting to see that though this model has fewer parameters than LLama and similar models, it outperforms them. 32k Context window is huge 

Key Features of Mixtral 8x7B

Before exploring the details of the Mixtral 8x7B model, let’s quickly explore What a Sparse Mixture of Experts (SMoE) 

A Sparse Mixture of Experts (SMoE) is a complex neural network architecture. It is composed of numerous specialized sub-networks, called ‘experts’, each designed to handle specific types of data or tasks. In an SMoE model, input is routed to only a few relevant experts, rather than all of them. This selective routing is usually managed by a ‘gating network’, which identifies the most suitable experts for a given input.

The sparse utilization of experts makes a model efficient and scalable, particularly for large-scale tasks, as it enables the model to handle a wide range of tasks without overwhelming computational resources. This architecture is particularly valuable in situations where diverse, specialized knowledge is required.

• Model Architecture: Mixtral is based on a transformer architecture and supports a fully dense context length of 32k tokens. The feed-forward blocks are replaced by Mixture-of-Expert layers.

• Enhanced Token Processing: The model uses a Sparse Mixture of Experts (SMoE) and has a router network at each layer to select two out of eight experts for processing each token. By doing so, the model can access a large pool of parameters (47B) while actively utilizing a smaller subset (13B) during inference. This mechanism ensures efficient computation.

• Routing Mechanism: The router network dynamically selects two experts per token at each layer. This selective process efficiently combines their outputs for token processing.

• Superior Performance: Mixtral 8x7B has shown better performance than major models such as GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B. This model also demonstrates reduced biases and a more balanced sentiment profile in benchmarks like BBQ and BOLD.

Why Mixtral 8x7B is Significant

• Benchmarking Excellence: Mixtral sets new benchmarks in language modeling, offering top-tier performance with lower active parameter count, making it a highly efficient model.
• Versatility and Multilingual Capabilities: Its ability to excel in tasks requiring long-context understanding and multilingual proficiency opens doors to diverse applications.
• Open Accessibility: Released under the Apache 2.0 license, Mixtral is available for broad usage, encouraging development across various fields.

Model Page: Mixtral 8x7B
Research Paper : Mixtral of Experts

Paper Authors : Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed