7 13 4

ROHITH VENKATA REDDY

knight7561

AI & ML interests

Deep learning, Autonomous Driving

Recent Activity

commented on an article about 1 month ago

SmolLM3: smol, multilingual, long-context reasoner

upvoted an article about 1 month ago

SmolLM3: smol, multilingual, long-context reasoner

commented on an article 3 months ago

DABStep: Data Agent Benchmark for Multi-step Reasoning

View all activity

Organizations

commented on SmolLM3: smol, multilingual, long-context reasoner about 1 month ago

Thats a great write.
Definitely adding to the Weekend read..!

upvoted an article about 1 month ago

Article

SmolLM3: smol, multilingual, long-context reasoner

and 22 others •

Jul 8

• 614

commented on DABStep: Data Agent Benchmark for Multi-step Reasoning 3 months ago

Great benchmark to start with .. Awesome work .. Kudos to the team ..!

upvoted an article 3 months ago

Article

DABStep: Data Agent Benchmark for Multi-step Reasoning

and 5 others •

Feb 4

• 99

updated a Space 3 months ago

Demo Mcp

📚

demo of MCP

published a Space 3 months ago

Demo Mcp

📚

demo of MCP

New activity in CogwiseAI/llama2_7b_hiring 3 months ago

Dataset and more details on Fine tuning

#1 opened 3 months ago by

knight7561

updated a collection 4 months ago

LLM and Reasoning Papers

Collection

Papers dump of LLM Reasoning domain • 15 items • Updated Apr 8 • 1

liked a Space 5 months ago

2.99k

The Ultra-Scale Playbook

🌌

The ultimate guide to training LLM on large GPU Clusters

New activity in nanotron/ultrascale-playbook 5 months ago

Questions?

👀 1

#84 opened 6 months ago by

nouamanetazi

reacted to Kseniase's post with 🔥 5 months ago

Post

7980

15 types of attention mechanisms

Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.

Here is a list of 15 types of attention mechanisms used in AI models:

1. Soft attention (Deterministic attention) -> Neural Machine Translation by Jointly Learning to Align and Translate (1409.0473)
Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1.

2. Hard attention (Stochastic attention) -> Effective Approaches to Attention-based Neural Machine Translation (1508.04025)
Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything.

3. Self-attention -> Attention Is All You Need (1706.03762)
Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation.

4. Cross-Attention (Encoder-Decoder attention) -> Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation (2104.08771)
The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources.

5. Multi-Head Attention (MHA) -> Attention Is All You Need (1706.03762)
Multiple attention “heads” are run in parallel. The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values.

6. Multi-Head Latent Attention (MLA) -> DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2405.04434)
Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations.

7. Memory-Based attention -> End-To-End Memory Networks (1503.08895)
Involves an external memory and uses attention to read from and write to this memory.

See other types in the comments 👇