Kubeflow Pipelines: Pipeline archive upload allows decompression bomb to exhaust server memory

Summary

The Kubeflow Pipelines API server decompresses uploaded pipeline archives without adequately bounding expanded data in affected versions. A small ZIP or gzip-compressed tar archive can consume disproportionately large amounts of memory while the server locates or reads the pipeline file, potentially exhausting API-server resources and interrupting service.

Multipart pipeline and pipeline-version upload processing decompresses the archive before namespace authorization, so a caller who can reach those endpoints does not need a successful pipeline-create authorization decision to trigger the issue. External authentication and network controls can restrict reachability. URL-import handlers authorize before fetching and decompressing a remote archive.

The amount of expansion needed to exhaust memory depends on deployment resource limits and concurrent activity; no particular archive is guaranteed to crash every deployment.

Remediation

Upgrade the API server to Kubeflow Pipelines 2.18.0. Versions through 2.17.3 are affected; the first affected release has not been established.

PR #14171 bounds compressed input and extracted pipeline data. PR #14186 strengthens the total tar traversal budget, including metadata and skipped entries. The 2.18 release also includes configurable upload and specification limits; operators should review the shipped settings and resource budget. These controls address pipeline archive amplification and do not establish general HTTP-body, temporary-disk, concurrency, or CPU limits.

Until upgrading, restrict access to upload and import endpoints and apply deployment-level request and resource controls. A compressed upload-size limit alone does not prevent decompression amplification. Regression tests cover compressed input, selected-entry boundaries, and tar traversal budgets; no deployed-server OOM test was performed for this assessment.

Credit

bugbunny.ai discovered and reported the issue. Ayush Gupta (@Ayush-kathil) authored the archive fixes in #14171 and #14186, with Jeff Spahr (@jeffspahr) contributing regression coverage to #14171.

@BrianWillows later submitted a mixed report that included source analysis of pipeline-archive decompression.