perf: Speed up creating archives when pusing big codebases #1011
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR somewhat relates to #982. The main goal is to speed up creating ZIP archives for codebases larger than 2MB when using
apify push.The issue was in using
archive.glob()function within the loop. This approach forces the library to go through the entire current working directory (and its children). Using this function is also redundant because thegetActorLocalFilePathsalready usesglobbyto get all of the valid file paths to be archived.This issue is easy to measure while working on a generic actor for
eu-monitoring-toolwhich has roughly 1200 source code files and largenode_modules.Here is comparison of the compression times:
@B4nan or @vladfrangu Could you please have a look?