Use nltk.tokenize.word_tokenize instead of string.split

[kaushiksk]

CodeComb/CodeComb_Core/embeddings.py

Line 15 in ecc5a7b

w2v_input = [w2v_input_i.split() for w2v_input_i in w2v_input]

word_tokenize is a much more sophisticated tokenizer and is recommended for NLP tasks