utf-8 to utf-16 conversion in ChunkWordReader.next() is incorrect. #33
Description
This code casts each byte to char, ignoring all multi-byte characters. The only reason this works is that the gov2 corpus is mostly ASCII.
{{ message }}
This code casts each byte to char, ignoring all multi-byte characters. The only reason this works is that the gov2 corpus is mostly ASCII.