Neural Representations of Natural Language by Lyndon White & Roberto Togneri & Wei Liu & Mohammed Bennamoun

Author:Lyndon White & Roberto Togneri & Wei Liu & Mohammed Bennamoun
Language: eng
Format: epub
ISBN: 9789811300622
Publisher: Springer Singapore

(3.52)

Most words do not co-occur

Some simple reasoning can account for this as a reasonable consequence of Zipf’s law (Zipf 1949) and a prior of the principle of indifference, but there is a further depth to it as explained by Ha et al. (2009).

The question is then: how is the negative sample to be found? One option would be to deterministically search the corpus for these negative samples, making sure to never select words that actually do co-occur. However that would require enumerating the entire corpus. We can instead just pick them randomly, we can sample from the unigram distribution. As statistically, in any given corpus most words do not co-occur, a randomly selected word in all likelihood will not be one that truly does co-occur – and if it is, then that small mistake will vanish as noise in the training, overcome by all the correct truly negative samples.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.