langstat2candidates.py requires large amounts of RAM

[achimr]

langstat2candidates.py, particularly when used with the -candidates parameter uses up large amounts of RAM (needing 32-64 GB of RAM for large language pairs). This is because it reads the entire candidates file into memory (dictionary with the URLs as keys and the entire candidates file line as values). Retaining all this data seems unnecessary.
This reduces the parallelizability and leads to crashes.

[achimr]

Matching candidates from some language into English with recent CommonCrawls (2016_50) requires 60+ GB of RAM