DeepSeek Popular Science! How Domestic LLMs Are Overtaking on the Curve?

Today, the hottest discussion in the financial circle is undoubtedly Deepseek, an AI application that has caused quite a stir in the tech world, directly causing $NVIDIA.US to take a tumble and potentially threatening major model manufacturers like $Meta Platforms.US, $Alphabet - $Citigroup(C.US), and $Alibaba.US.

Everyone's impression of it is "domestic LLM achieving GPT O-1 level at low cost." So how does Deepseek achieve this? No one has mentioned the underlying principles, so let me explain: the technical principles behind DeepSeek's ability to overtake on a curve boil down to three points:

Mixture-of-Experts (MoE) is a core technology of the DeepSeek-V3 model. It optimizes computational efficiency and resource usage by intelligently selecting which parameters to activate, thereby improving the model's performance. This architecture allows the model to activate only a portion of its parameters when processing inputs, significantly reducing computational costs.

To put it in layman's terms:

Imagine a large company's customer service center, which has many different departments. Each department has its own area of expertise, such as technical support, billing inquiries, and complaint handling. Whenever a customer calls, the receptionist (Gate Network) will transfer the call to the most appropriate department (expert) based on the customer's issue. In DeepSeek's MoE architecture, the model functions like this customer service center, having multiple "experts" to handle different types of information. When an input arrives, the system selects a few of the most relevant experts to process it, rather than involving all experts, thus saving resources and improving efficiency 2. Dynamic Redundancy Strategy In DeepSeek-V3, the system dynamically selects suitable "experts" for processing based on the current task. This flexibility ensures that the model maintains optimal load balancing during inference and training, thereby improving efficiency and response speed.

Continuing with the analogy of a customer service center, suppose some departments are often overwhelmed while others are quite idle. To balance the workload, the main receptionist dynamically adjusts the distribution of incoming calls based on the busyness of each department. For example, if the technical support department has already received many calls, the receptionist will reduce the transfers to that department and direct more calls to the billing inquiry department. DeepSeek's dynamic redundancy strategy is such a mechanism that intelligently adjusts the load of each expert, ensuring that every expert can work efficiently, thus avoiding resource waste.

Multi-Head Potential Attention (MLA) DeepSeek-V3 also introduces a multi-head potential attention mechanism, allowing the model to better capture contextual information when understanding and generating text. This mechanism enables the model to perform better in handling complex problems, especially in areas such as mathematical calculations and code generation.

Still using a human analogy:

Imagine a band performing, where each instrument has its unique timbre and role. During the performance, the conductor activates different instruments based on the needs of the piece. For instance, in a soft melody, only violins and flutes may be needed, while in a vigorous section, all instruments play together. DeepSeek's multi-head potential attention mechanism is like this conductor, allowing the model to dynamically select the most suitable "instruments" (attention heads) based on the input content, thereby enhancing overall performance and efficiency.