dm.cs.tu-dortmund.de/en/mlbits/neural-nlp-decoders/
Decoder Models – Lecture Notes
\frac{z_i}{\theta}} \]
\(\theta=1\) yields the regular softmax operation.
Because \(\exp (z_i/\theta) = (\exp z_i)^{1/\theta} = (\exp 1/\theta)^{z_i}\) we see that:
for \(\theta<1\) , large \(z_i\) get boosted [...] called causal (e.g., CausalLM in transformers ), as this matches the idea of \(P(w_N \mid w_1,\ldots, w_{N-1})\) , where the next word is caused by the previous words.
Localized RoPE
Positional Encodings [...] = (w_0, \ldots , w_t)\) , \(k\) context window size, predict the next word:
\(L_1(w_{0:t}) = \sum_i log~ P(w_i|w_{i-k:i-1}; \Theta)\)
Finetuning: Discriminative (supervised); given a labelled data set …
