What is Attention?

Attention computes, for every token being generated, a weighted relevance score against every token in the context. This lets models connect a pronoun to its noun ten paragraphs back or a bug to a function defined in another file. It is the core operation of the transformer and its main computational cost.