Standard Transformer attention does not encode distance or structure — which makes it a poor fit for graphs.
In this AI Tech Experts Webinar, Kamil Czerski, Senior Edge Staff Engineer, explains how attention mechanisms can be adapted to graph-structured data.
You’ll learn:
- why vanilla self-attention fails on graphs,
- distance bias attention and its limitations,
- rotary position embeddings (RoPE) adapted from sequences to graphs,
- generalization to unseen and continuous distances,
- practical trade-offs and implementation constraints.
Timeline
01:02 Motivation: graphs, distances and real use case
01:52 Vanilla attention explained
06:45 Multi-head attention basics
07:44 Why attention fails on graphs
08:46 Distance bias attention for graphs
12:48 RoPE intuition and theory
18:46 RoPE applied to graphs
Speaker
Kamil Czerski
Senior Edge Staff Engineer at deepsense.ai
