feat: add server functionality #41
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @bede
This adds server functionality, controlled by use of the feature
server. Sorry, this PR is a little long but a vast majority of line changes are reworking existing logic rather than adding new logic.I've checked with the test suite and
rsviruses17900.fastq.gz, and both give the same results as previously in my testing.Compiled without the
serverfeature:Compiled with the
serverfeature:filter::runfor a version based on 0.7.0, so it's a little slower for runningdeacon filter...deacon server some/index.binto preload an index into memory for later usedeacon client http://some.address ...to function equivalent todeacon filer some/index.bin ...clientandfilterproduce the same results as 0.10.0In a bit more depth, compared to 0.10.0:
FilterProcessorremote_filter.rswhich is based on the implementation from 0.7.0, but updated to use the newer logicfilter.rsa simple file which either importsrunfromremote_filter.rsorlocal_filter.rsdepending on if theserverfeature is enabledclientandservercommands to the CLI.clientis essentially a mirror offilter, but swapping the index path for a server addressclient, mirroring all tests forfilterclippylints, although these should not impact functionalityIn terms of performance, larger input files will process slower (as sending to the server has overheads), but for smaller input files, there is significant speed improvement. Especially useful for removing human reads from ONT fastqs as they are basecalled.
On my laptop (Linux, x86), just loading the panhuman index take ~6s, so processing a small fastq (4,900 reads) with
deacon filtertakes a total of <7s. I've seen a lot of variation with how long index loading takes, so there's a margin of +- 1s on how long this takesPre-loading an index with
deacon server, then processing with the same settings, but withdeacon clienttakes ~200ms.