Improving Performance - Maven Artifact Caching

[goneall]

Question

I'm running some very large Trivy scans with several hundred Maven POM files being analyzed.

It is taking over 11 hours to complete.

From analysis, almost all of the time is waiting for Maven artifact downloads.

I was wondering if I could configure Trivy to use a local artifact repository which would essentially be a very large cache or mirror of Maven to speed things up. Something like Sonatype Nexus.

Is it possible to configure Trivy to utilize such a repository?

I did find an old article on implementing a caching forward proxy to solve a slightly different problem. This is something I could try if you think it would help.

Target

Filesystem

Scanner

License

Output Format

SPDX

Mode

Standalone

Operating System

Ubuntu

Version

No response

[knqyf263]

@DmitriyLewen Can you please take a look? You already have many tasks, so anytime you have a minute is fine.

[DmitriyLewen]

I will try to check that on this weak.

[DmitriyLewen]

Hello @goneall
I looked at your case.

I did find an old article on implementing a caching forward proxy to solve a slightly different problem

That instruction was for jar files.
No, it doesn't work now because we use trivy-java-db.

About pom.xml files:
Trivy doesn't have an env (like MAVEN_CENTRAL_URL) to set repositories to look for pom files.
Trivy only checks repositories from pom.xml files (so you can temporarily add your local repository to the scanned pom file (but I'm not sure if that's ok for you)).
I think the best solution is to use the local repository from the settings file. We already have #7807 with a request to add this functionality.

[goneall]

Completed some analysis.

Here's a few conclusions along with some reasoning and data for the conclusions:

• We're impacted by rate limiting on one or more public Maven repositories based on:
  • Running the scan with the offline option runs in 8 minutes whereas running the scan without the option takes 9+ hours
  • Most of the time we're blocked waiting for network IOs
  • The log files shows a consistent 30 seconds between long attempts to resolve POM file locations
• The caching implemented in Trivy does not help - it looks like we're always going out to the internet based on:
  • Running the same scan more than once does not significantly change the time
  • The log files shows the same failures attempting to fetch from remote repositories
  • Based on this comment - it looks like Trivy doesn't cach in FS mode
• There are several remove repositories involved that may be rate limiting
  • It is difficult to tell exactly which remotes are rate limiting since we aren't currently logging successful fetches from the repo
  • The log messages show several repos with failed fetches, but little or no failed resolves, so one of the alternative remotes are successful
• Using an artifact repository doesn't seem practical for our particular use case
  • The local repository needs to be populated
  • Running a Maven build will populate the local cache / repo, but we don't do builds before we run Trivy - Note: this may be a somewhat unique use case
  • Based on experiments and looking a the code, Trivy will not populate the local repo
  • Mirroring the entire remote repository is possible, but would likely involve a lot of resources - for example, the Maven Central repo is over 45 Terabytes in size

Based on the analysis, implementing a cache for the POM files which persists across runs of Trivy may be the best option. It looks like much of this is implemented. If there was some protocol for a "pluggable" cache, I could implement something outside of Trivy.

@DmitriyLewen - how hard would it be to enable the caching for the file system scans? Is there some way I can help?

Also - feel free to challenge any of my conclusions or provide any alternative solutions.

I have a lot of data from the logs of Trivy and Maven runs with debug enabled I can share if interested including a spreadsheet which analyzes the Trivy debug statements.

[DmitriyLewen]

Running the scan with the offline option runs in 8 minutes whereas running the scan without the option takes 9+ hours

In this case, the report doesn't include most of the artifacts 😞

how hard would it be to enable the caching for the file system scans?

Caching for fs mode doesn't make sense, because we can't create safe key for bucket:

• file can be changed, so we can't use filepath, file info, etc.
• we can use hash of file, but for that we still need to read file (it doesn't make sense)

We can think about adding cache for pom file fetched from remote repository:
But there is still a problem with the modified file.
Only Sonatype (and perhaps maven central) forbids modifying artifacts (aquasecurity/trivy-java-db#48 (comment)), but we can't be sure that other repositories won't modify the artifact (in which case Trivy will use incorrect/old artifact info).

[goneall]

Thanks @DmitriyLewen for the reply.

We can think about adding cache for pom file fetched from remote repository:
But there is still a problem with the modified file.
Only Sonatype (and perhaps maven central) forbids modifying artifacts (aquasecurity/trivy-java-db#48 (comment)), but we can't be sure that other repositories won't modify the artifact (in which case Trivy will use incorrect/old artifact info).

From what I can tell from the logs and code, caching the POM files would be sufficient to avoid the rate limiting. I've always assumed the maven central artifacts to be immutable, so I believe caching POM files from Maven central and Sonatype should work. I can check with some folks I know at Sonatype to verify.

For the other repos, we could adopt some aging protocol (any artifact older the XX days), but that would complicate the configuration.

Is there any way for me to implement an external caching module for POM files without having to modify the project POM files? From looking at the debug traces and code, it didn't look possible, but I thought I would check.

[DmitriyLewen]

For the other repos, we could adopt some aging protocol (any artifact older the XX days), but that would complicate the configuration.

This may be unreliable, as changes may happen at any time.
Also people will have to figure out how this caching works, repository specifics, etc. Not all users are ready to deal with this.

But I think we can create a new TRIVY_EXPERIMENTAL_* environment (like for RPM archives (https://trivy.dev/latest/docs/coverage/others/rpm/#sbom)) with a list of repositories.
The user will choose the repositories they want to cache (for example, if the user is sure that the artifacts in this repository are immutable).
This will give more options to advanced users and will not affect users who want Trivy out of the box.
@knqyf263 wdyt?

[goneall]

But I think we can create a new TRIVY_EXPERIMENTAL_* environment (like for RPM archives (https://trivy.dev/latest/docs/coverage/others/rpm/#sbom)) with a list of repositories.

I think this will work for my particular use case