dm.cs.tu-dortmund.de/mlbits/neural-nlp-intro/
Neural Models for Word Similarity – Lecture Notes
[ LeMi14 ; MCCD13 ] )
One hidden layer neural network:
Use context words \(w_{i-c},\ldots ,w_{i-1},w_{i+1},\ldots ,w_{i+c}\) to predict word \(w_i\) .
Intuition: sum the rows of every input word in \(W [...] \) are iteratively optimized to best predict the neighbor words \(w_j\) for \(j\in i-c,\ldots ,i-1,i+1,\ldots ,i+c\) and \(c\approx 5\) .
References
[BDVJ03]
Bengio, Y., Ducharme, R., Vincent, P. and Janvin [...] look-up table \(C(w)\) for word vectors
context by concatenating the previous \(c\) words: \(x=(C(w_{i-1}),\ldots , C(w_{i-c}))\)
optional direct connection layer \(Wx\)
hidden layer with sigmoid activation …
