Parallelize the scan pr flow #1028
Conversation
- Run SCA and JAS scans in parallel for better performance - Use LogCollector for isolated log capture per scan - Logs dumped sequentially after scans complete (no interleaving) - Add SetLogCollector to ScanDetails for passing collector to CLI - Export FrogbotUploadRtRepoPath for result upload
- Add JF_PARALLEL_PR_SCAN env var (default: true, set to 'false' for sequential mode) - Remove verbose/redundant comments - Remove gradle cache files from tracking
…ing, support FailUponAnyScannerError
…t, remove comments
- scanPullRequestBranches: scanning + partial results filtering - auditPullRequestAndReport: output flags + issue conversion
|
frogbot-pr-info.txt - regular log mode frogbot-pr-debug.txt - debug mode |
| diffResults := results.UnifyScaAndJasResults(scaResult.source, jasDiff) | ||
|
|
||
| log.Info("Uploading results to platform...") | ||
| if _, err := output.UploadCommandResults(scanDetails.ServerDetails, utils.FrogbotUploadRtRepoPath, diffResults); err != nil { |
There was a problem hiding this comment.
was this operation exist in frogbot before ? (uploading the results to RT) or was it cli responsibility before?
There was a problem hiding this comment.
cli
it uploaded after the second scan finished, and it knew it was in a diff context
now scans in parallel and it doesnt know its in diff really (other than sca)
There was a problem hiding this comment.
so if i understand - it means that for scan-pr -> frogbot uploads results, but for scan-repo - cli uploads the results
| targetLogCollector := audit.NewLogCollector(log.GetLogger().GetLogLevel()) | ||
| targetScanDetails.SetLogCollector(targetLogCollector) | ||
|
|
||
| targetResults = targetScanDetails.Audit(targetDir) |
There was a problem hiding this comment.
why are the Audit commands in runScaScans are not running in parallel but in run JasScans they are executed as a go routine?
There was a problem hiding this comment.
because in sca , for diff:
We calculate the target sca sbom
delta it with source, during source scan
and then what is left we do sca scan and contextual when we scan the source
running the sca on target takes milliseconds becasue sbom calculation is almost immediate with the new engine
| if e != nil { | ||
| err = errors.Join(err, fmt.Errorf("failed to get issues for pull request. Error: %s", e.Error())) | ||
| log.Debug("Scanning source branch code...") | ||
| scanDetails.SetUploadCdxResults(true) |
There was a problem hiding this comment.
can you explain to me what is the purpose of this line scanDetails.SetUploadCdxResults(true) ? and in general why did you add SetUploadCdxResults(upload bool) ?
There was a problem hiding this comment.
https://github.com/jfrog/frogbot/pull/1028/changes#r2693788773
evolved from the issue in this thread
in the past we sent results in scan repo always
and in sca pr only on the second scan, and we knew if we were the second scan because of value of DiffScan and , resultsToComapre see this:
SetUploadCdxResults(!sc.diffScan || sc.ResultsToCompare != nil)
now because scanning logic changed for scan pr,it became a lot more trivial to just send true or false to whether we need to upload results, than to let the function figure it out on its own
scanpullrequest/scanpullrequest.go
Outdated
| scaResult := <-scaChan | ||
| jasResult := <-jasChan | ||
|
|
||
| if scaResult.target != nil && scaResult.target.GetStatusCodes().ScaScanStatusCode != nil { |
There was a problem hiding this comment.
there is no possible scenario where for example scaResult.target==nil and also err==nil ? because in this case there is no log at all
There was a problem hiding this comment.
only case is maybe if sca is turned off , which is basically what im checking here
will validate
There was a problem hiding this comment.
i updated the condition for this and sca considering all the edge cases (turned off by config, entitlements make jas not run)
| dependencySubmissionFrogbotDetector = "JFrog Frogbot" | ||
| frogbotUrl = "https://github.com/jfrog/frogbot" | ||
| frogbotUploadRtRepoPath = "frogbot" | ||
| FrogbotUploadRtRepoPath = "frogbot" |
| } | ||
| } | ||
|
|
||
| // CloneForBranchScan creates a clone configured for branch scanning with isolated logging. |
There was a problem hiding this comment.
ill proably remove comment say if its clear
- Use log.BufferedLogger directly instead of audit.LogCollector wrapper - Replace HasLogs() with Len() > 0 - Remove duplicate diff log (already in cli-security) - Reorder log statement for parallel scan start
Update jfrog-cli-security to bccd932 which fixes the issue where scan status codes were not preserved during SCA/JAS result merging, causing 'Not Scanned' to appear for JAS scans that actually ran but produced 0 new findings after diff.
3 participants
Uh oh!
There was an error while loading. Please reload this page.