Find a lesson, question or project

Start typing to search published content and planned structures.

↑ ↓ moveEnter openEsc close

Stop Sending Entire CI/CD Logs to LLMs: Building an Intelligent Failure-Aware Log Reduction Pipeline

Reduce noisy pipeline logs into a small, safe, failure focused context before asking an LLM to explain what went wrong.

MediumNotes

A failed pipeline can produce thousands of log lines. Sending all of them to a language model looks convenient, but it usually gives the model more noise than evidence.

Most failures have a much smaller useful context. There is an initial error, the command or test that produced it, a short sequence of related messages, and a few environment details. Everything else is often repeated progress output, successful setup, download logs, retry noise, or a second error caused by the first one.

Why complete logs are a poor input

Large log payloads increase cost and response time. They can push useful evidence away from the model’s attention, and they make accidental secret exposure more likely.

The goal is not general log summarization. The goal is to preserve enough evidence to explain a specific failure.

The reduction pipeline

Text
raw pipeline logs
        ↓
parse and normalize events
        ↓
detect the first meaningful failure
        ↓
collect related context
        ↓
redact, deduplicate, and rank evidence
        ↓
build a bounded diagnostic package
        ↓
ask the LLM for an explanation

Parse the log into events

Convert raw lines into a consistent record with a timestamp, stage, command, severity, source, and message. Keep stack traces and multiline compiler errors together. Structured events make later decisions easier than repeated regular expressions over one large string.

Find the first meaningful failure

The last error is not always the cause. A test runner might report a failed job after a compilation error occurred several minutes earlier. Rank failure candidates using exit codes, known tool patterns, exception boundaries, test failures, and transitions between pipeline stages.

Build a context window

Keep the failing event, nearby lines, the active command, the relevant stack trace, and any earlier event referenced by the failure. Context should follow relationships, not only line distance.

Remove noise safely

Collapse repeated messages, successful progress output, dependency download details, and identical retry failures. Redact credentials, tokens, private URLs, and sensitive environment values before any model request is created.

Create a diagnostic package

The final model input should be explicit and small.

Json
{
  "pipelineStage": "integration-tests",
  "failedCommand": "./gradlew integrationTest",
  "exitCode": 1,
  "primaryFailure": "Connection refused while starting the test container",
  "evidence": [
    "Container health check failed after 30 seconds",
    "Port 5432 was not accepting connections"
  ],
  "removedLines": 18420,
  "redactions": 3
}

Ask the model to distinguish evidence from inference. It should explain the likely cause, cite the supplied evidence, suggest the next checks, and say when the context is insufficient.

Measure the reducer, not only the model

Track the percentage of log lines removed, token reduction, diagnostic latency, secret detection, and whether the primary failure was preserved. Keep a reviewed set of real pipeline failures and run it whenever a reduction rule changes.

A useful reducer is conservative. It can leave a little extra context, but it should never remove the line that explains the failure.

A practical first version

Start with one build system and a small set of known failure patterns. Preserve compiler errors, test failures, stack traces, exit codes, and the command that was running. Add redaction before the model integration, then compare the reduced diagnosis with the original logs during review.

The model should be the final reasoning step. The pipeline should do the evidence collection first.

Continue learning

Back to Notes →

Clarity check

Did this explanation help?

Your quick rating stays in this browser. Use the email option if you want the feedback to reach Rahul.