In this part we fix two of those cracks simultaneously because they are deeply connected:

1. CI logs are enormous — sending raw traces to an LLM is expensive, slow, and counterproductive

2. LLM output is unstructured text — json.loads() on a Claude response is not a production strategy

By the end of this post your app will process logs of any size efficiently and return a typed Python object you can actually work with downstream — not a string you hope parses correctly.

The Problem With Raw Traces

Real CI/CD job traces are not the clean 50-line snippets in tutorials. A typical GitLab pipeline job produces:

Docker layer pull logs     → 300-500 lines of layer hashes
npm/pip install output     → 200-400 lines of package resolution
Test setup and fixtures    → 100-200 lines of framework output
Your actual error          → 3-10 lines buried somewhere in the middle

A moderately complex pipeline job produces 5,000 to 50,000 characters of output. Some jobs — particularly those running test suites or security scanners — produce 100,000 characters or more.

Sending this raw to Claude has three real costs:

The solution is not to hope Claude figures it out. It is to extract the signal before the LLM call.

Strategy 1 — Smart Truncation

The simplest and most effective intervention. CI failure traces follow a predictable pattern: the error is almost always in the last 20-30% of the log. The first 70% is setup noise.

def prepare_trace_for_llm(trace: str, max_chars: int = 8000) -> str:
    """
    Intelligently truncates a CI trace to fit context limits.
    Preserves the head (job context) and tail (where errors live).
    Drops the middle noise.
    """
    if len(trace) <= max_chars:
        return trace

    # Keep a small head for job context, large tail for the error
    head_chars = 1000
    tail_chars = max_chars - head_chars

    head = trace[:head_chars]
    tail = trace[-tail_chars:]

    return f"{head}\n\n[... {len(trace) - max_chars} chars truncated ...]\n\n{tail}"

This one function cuts token cost by 60-80% for most jobs while preserving everything the LLM actually needs to classify the failure.

Strategy 2 — Error Signal Extraction

Smart truncation handles size. Signal extraction handles quality. Instead of sending the full trace — even truncated — we pre-process it to pull out the lines that actually contain diagnostic information.

import re

def extract_error_signals(trace: str) -> dict:
    """
    Extracts structured error signals from a raw CI trace.
    Returns a dict with the most diagnostic content isolated.
    This is what we pass to the LLM — not the raw trace.
    """
    lines = trace.split("\n")

    # Lines containing error markers
    error_lines = [
        line for line in lines
        if any(marker in line.lower() for marker in [
            "error", "exception", "failed", "fatal",
            "traceback", "exit code", "killed", "timeout",
            "not found", "permission denied", "cannot"
        ])
    ]

    # Last 100 lines — where failures always surface
    tail_lines = lines[-100:]

    # Python traceback if present
    traceback = None
    tb_match = re.search(
        r"Traceback \(most recent call last\):.*?(\w+Error[^\n]*)",
        trace,
        re.DOTALL
    )
    if tb_match:
        traceback = tb_match.group(0)[-2000:]

    # Exit code
    exit_code = "unknown"
    exit_match = re.search(r"exit(?:ed)? (?:with )?(?:code )?(\d+)", trace, re.I)
    if exit_match:
        exit_code = exit_match.group(1)

    return {
        "exit_code":   exit_code,
        "error_lines": "\n".join(error_lines[-50:]),   # last 50 error lines
        "tail":        "\n".join(tail_lines),
        "traceback":   traceback or "None detected",
    }

The error_lines field alone typically reduces a 50,000-character trace to 500-800 characters of pure signal. The LLM now has exactly what it needs to classify the failure — nothing more.

Strategy 3 — Two-Pass Summarisation for Massive Traces

For traces over 100,000 characters (security scanners, full test suite runs), even smart truncation may not be enough. Use a fast cheap model to summarise first, then pass the summary to your main model.

import boto3, json

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

def summarise_large_trace(trace: str) -> str:
    """
    Uses Claude Haiku (fast, cheap) to compress a massive trace
    into a diagnostic summary before the main analysis call.
    Reduces a 100K char trace to ~500 chars of signal.
    """
    prompt = f"""Summarise this CI/CD job log in under 400 words.
Focus ONLY on: what failed, the exact error message, what line or file caused it.
Ignore all successful steps. Be precise.

LOG:
{trace[-15000:]}"""

    response = bedrock.invoke_model(
        modelId="anthropic.claude-haiku-4-5-20251001",
        body=json.dumps({
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens":        600,
            "messages": [{"role": "user", "content": prompt}]
        })
    )
    body = json.loads(response["body"].read())
    return body["content"][0]["text"]


def prepare_trace_input(trace: str, job_name: str) -> str:
    """
    Routing function — picks the right preparation strategy
    based on trace size. Returns a clean string ready for the LLM.
    """
    signals = extract_error_signals(trace)

    if len(trace) > 100_000:
        # Two-pass for massive traces
        summary = summarise_large_trace(trace)
        return f"SUMMARY:\n{summary}\n\nKEY SIGNALS:\n{signals['error_lines']}"

    elif len(trace) > 8_000:
        # Extract signals + tail for medium traces
        return (
            f"EXIT CODE: {signals['exit_code']}\n\n"
            f"ERROR LINES:\n{signals['error_lines']}\n\n"
            f"TRACEBACK:\n{signals['traceback']}\n\n"
            f"LOG TAIL:\n{signals['tail']}"
        )

    else:
        # Small trace — send as-is
        return trace

Practical size guide:

The Second Problem — Unreliable Output

With signal extraction solved, we now address output reliability. In Part 1 we asked Claude to return JSON and used json.loads() on the response. This works most of the time.

Most of the time is not acceptable in a system that triggers merge requests.

The failure modes are real:

# Claude responds with a preamble
"Here is the JSON you requested:\n```json\n{\"type\": \"dependency\"}"
# json.loads() → fails

# Claude uses slightly different key names
{"failure_type": "dependency"}  # you expected "type"
# KeyError downstream

# Claude adds an extra field you didn't ask for
{"type": "dependency", "confidence": "high", "suggested_action": "..."}
# Your downstream code breaks on unexpected fields

The solution is Pydantic with LangChain's output parser. The parser handles JSON extraction, validation, and instantiation into a typed Python object in one step.

Introducing LangChain LCEL and PydanticOutputParser

LangChain's LCEL (LangChain Expression Language) is a composable way to build LLM pipelines using the | pipe operator. Each component in the chain has a defined input type and output type, and they connect only when those types are compatible.

The pattern we are building is:

prompt_template | llm | parser

Where each | represents a type-safe handoff:

prompt_template  →  outputs ChatPromptValue (formatted messages)
llm              →  consumes ChatPromptValue, outputs AIMessage
parser           →  consumes AIMessage, outputs your Pydantic object

Let's build it step by step.

Step 1 — Define Your Output Schema With Pydantic

from pydantic import BaseModel, Field
from typing import Literal

class JobTriage(BaseModel):
    """
    Structured output from the triage LLM call.
    Every field is typed and validated — no dict key surprises.
    """
    status: str = Field(
        description="Always 'failed' for failed jobs"
    )
    failure_category: str = Field(
        description=(
            "Category: dependency | syntax | test | "
            "config | infra | security | flaky | unknown"
        )
    )
    failed_reason: str = Field(
        description="One clear sentence explaining what failed and why"
    )
    suggested_fix: str = Field(
        description="One actionable sentence describing how to fix it"
    )
    confidence: Literal["high", "medium", "low"] = Field(
        description="Confidence in the classification"
    )

Pydantic gives you:

• Type validation — confidence can only be "high", "medium", or "low"

• Attribute access — result.confidence instead of result["confidence"]

• IDE autocompletion — your editor knows the shape of the object

• Downstream type safety — passing JobTriage objects around is self-documenting

Step 2 — The PydanticOutputParser

from langchain.output_parsers import PydanticOutputParser

parser = PydanticOutputParser(pydantic_object=JobTriage)

# This generates a JSON schema instruction string
# that tells Claude exactly what to return
format_instructions = parser.get_format_instructions()

get_format_instructions() produces a JSON schema string that gets injected into your prompt. Claude reads it and knows precisely what keys, types, and values to return. The LLM is constrained by schema — not just by a casual "return JSON" request.

Step 3 — The Prompt Template

from langchain_core.prompts import ChatPromptTemplate

SYSTEM_PROMPT = """You are a DevOps expert analysing CI/CD pipeline failures.
Analyse the provided CI log signals and classify the failure.
{format_instructions}"""

prompt_template = ChatPromptTemplate.from_messages([
    ("system", SYSTEM_PROMPT),
    ("user",   "Job: {job_name}\n\nLog signals:\n{log_trace_signal}")
])

# Bind format_instructions as a partial — it never changes between calls
# so we do not pass it at invoke() time
prompt_template = prompt_template(
    format_instructions=format_instructions
)

Step 4 — The LLM

from langchain_aws import ChatBedrock

llm = ChatBedrock(
    model_id="anthropic.claude-3-5-sonnet-20241022-v2:0",
    region_name="us-east-1"
)

Step 5 — Chain Everything Together

# The chain — three components connected by pipe operators
chain = prompt_template | llm | parser

That is the entire chain. Three lines. Here is what each | does:

prompt_template
  Receives:  {"job_name": "...", "log_trace_signal": "..."}
  Produces:  ChatPromptValue (system + user messages formatted)
       |
      llm
  Receives:  ChatPromptValue
  Produces:  AIMessage (raw response with JSON string in .content)
       |
    parser
  Receives:  AIMessage
  Produces:  JobTriage (validated Pydantic object)

After the parser step, json.loads() failures, key errors, and unexpected fields are all handled automatically.

Step 6 — Invoke the Chain

def analyse_failed_job(job_name: str, trace: str) -> JobTriage:
    """
    Full pipeline: extract signals → invoke chain → return typed object.
    """
    # Signal extraction — replaces sending raw trace
    log_trace_signal = prepare_trace_input(trace, job_name)

    # Chain invocation — returns a JobTriage object, not a dict
    result: JobTriage = chain.invoke({
        "job_name":         job_name,
        "log_trace_signal": log_trace_signal
    })

    return result

The Full Updated main.py

from gitlab_client import get_failed_jobs, get_job_trace
from triage import analyse_failed_job
import os

def main():
    pipeline_id = os.environ["FAILED_PIPELINE_ID"]
    print(f"Analysing pipeline {pipeline_id}...")

    failed_jobs = get_failed_jobs(pipeline_id)
    if not failed_jobs:
        print("No failed jobs found.")
        return

    for job in failed_jobs:
        job_id   = str(job["id"])
        job_name = job["name"]

        trace  = get_job_trace(job_id)
        result = analyse_failed_job(job_name, trace)

        # result is now a JobTriage object — typed, validated
        print(f"\n--- {job_name} ---")
        print(f"Category:   {result.failure_category}")
        print(f"Reason:     {result.failed_reason}")
        print(f"Fix:        {result.suggested_fix}")
        print(f"Confidence: {result.confidence}")

if __name__ == "__main__":
    main()

What the Output Looks Like Now

Analysing pipeline 882341...

--- test:unit ---
Category:   dependency
Reason:     ModuleNotFoundError for 'fastavro' — not in requirements.txt
Fix:        Add fastavro>=1.7.0 to requirements.txt
Confidence: high

--- build:docker ---
Category:   config
Reason:     Base image python:3.11-alpine not found — tag changed upstream
Fix:        Update Dockerfile FROM to python:3.11-alpine3.18
Confidence: high

Same useful output as Part 1 — but now:

• Typed JobTriage objects instead of raw dicts

• result.confidence never throws a KeyError

• Literal["high", "medium", "low"] validation means the value is always one of three strings

• Token cost reduced by 70% on typical traces

• Parser handles any JSON formatting quirks Claude introduces

Understanding the LCEL Type Contract

One thing worth understanding clearly before moving forward: the | pipe operator in LCEL is not just aesthetic — it enforces type contracts at connection time.

Works:
  prompt_template | llm        → prompt outputs messages, llm accepts messages ✅
  llm | parser                 → llm outputs AIMessage, parser accepts AIMessage ✅

Fails at runtime:
  parser | llm                 → parser outputs Pydantic object, llm expects messages ❌
  prompt_template | parser     → prompt outputs messages, parser expects AIMessage ❌

LangChain does not enforce these contracts at definition time — it fails at invocation. This means: define your chain once and test it immediately with a real call. If the types are wrong you will know straight away.

The useful mental model is that each component is a function with a specific signature, and | is function composition. You would not write parser(prompt_template(input)) in raw Python — and LCEL is the clean way to express the same idea.

What We Have Now vs Part 1

What's Still Missing

With efficient input and structured output working, the system classifies failures accurately and cheaply. But every classification is based on general LLM knowledge — what Claude learned during training.

It does not know:

• Your organisation's internal packages and conventions

• Your team's known fix patterns for recurring failures

• Past successful fixes your team has applied to similar errors

In Part 3 we add this contextual knowledge using Retrieval-Augmented Generation (RAG) with AWS Bedrock Knowledge Base — so the LLM's analysis is grounded in your organisation's own history, not just the internet.

Every failed pipeline is a detective story buried in thousands of lines of output. Here's how we built the first version of an agent that reads them so you don't have to.

The Problem We Are Tired Of

It starts with a Slack message. "Pipeline failed." You open GitLab, find the failed job, scroll through hundreds of lines of output — Docker pulling layers, npm install noise, test setup logs — to find the three lines that actually matter. Then you fix it, push, wait, and do it again.

For a busy team running dozens of pipelines a day, this is not a minor inconvenience. It is a significant drain on engineering time. The worst part? Most failures repeat. The same dependency conflict. The same schema mismatch. The same environment variable that someone forgot to set. An engineer should not need to read the same error twice.

The idea is simple: if an LLM can understand a Python/Node error from a Stack Overflow post, it can surely understand a CI log. Let's find out

What We're Building in This Series

This is Part 1 of a series documenting a real project built over several months. We start with a single, humble LLM call and end with a production multi-agent system that automatically detects, classifies, and fixes CI/CD pipeline failures — creating merge requests with the corrected code before a human has to get involved.

The Tech Stack

Why Bedrock?

Bedrock allow IAM-based auth rather than managing API keys. Bedrock gives you access to Claude, Titan, and others through the same AWS credentials your CI runner already has. No new secrets to manage.

The First Working Version

The simplest possible thing that could work: fetch the trace of a failed GitLab job, send it to Claude with a prompt asking for a classification, and print the result. No framework. No orchestration. Just Python, boto3, and a prompt.

Step 1 — Fetch the Failed Job Trace from GitLab

GitLab exposes a /jobs/{id}/trace endpoint that returns the raw console output of any CI job. With a personal access token and the project ID, we can pull this programmatically.

import requests
import os

GITLAB_URL   = "https://gitlab.com"
GITLAB_TOKEN = os.environ["GITLAB_TOKEN"]  # personal access token
PROJECT_ID   = os.environ["CI_PROJECT_ID"]   # available in GitLab CI env

def get_failed_jobs(pipeline_id: str) -> list:
    """Return all failed jobs for a given pipeline."""
    headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
    url = f"{GITLAB_URL}/api/v4/projects/{PROJECT_ID}/pipelines/{pipeline_id}/jobs"

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    # Return only the jobs that actually failed
    return [job for job in response.json() if job["status"] == "failed"]


def get_job_trace(job_id: str) -> str:
    """Fetch the raw console trace for a specific job."""
    headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
    url = f"{GITLAB_URL}/api/v4/projects/{PROJECT_ID}/jobs/{job_id}/trace"

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    return response.text  # raw log output — can be 50K+ characters

Step 2 — Call the LLM With the Trace

AWS Bedrock exposes Claude via the bedrock-runtime boto3 client. You construct a message payload in Anthropic's format, call invoke_model(), and parse the response. That's it.

import boto3
import json

# bedrock-runtime — the client for invoking LLMs
# Note: this is different from "bedrock-agent-runtime" (Knowledge Base client)
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

MODEL_ID = "anthropic.claude-3-5-sonnet-20241022-v2:0"

def classify_failure(job_name: str, trace: str) -> dict:
    """
    Send a job trace to Claude and ask for a failure classification.
    Returns a dict with type, reason, suggested fix, and confidence.
    """

    prompt = f"""You are a DevOps engineer analysing a failed CI/CD job.

Job name: {job_name}

CI log:
{trace}

Analyse the failure and respond with a JSON object containing:
{{
  "failure_type": "one of: dependency | syntax | test | config | infra | flaky",
  "failed_reason": "one clear sentence explaining what failed and why",
  "suggested_fix": "one actionable sentence describing how to fix it",
  "confidence": "high | medium | low"
}}

Respond with valid JSON only. No preamble."""

    response = bedrock.invoke_model(
        modelId=MODEL_ID,
        body=json.dumps({
            "anthropic_version": "bedrock-2023-05-31",
            "max_tokens": 512,
            "messages": [{
                "role":    "user",
                "content": prompt
            }]
        })
    )

    # Parse the response body
    body    = json.loads(response["body"].read())
    raw_txt = body["content"][0]["text"]

    return json.loads(raw_txt)

Step 3 — Wire It Together

The entry point loops over every failed job in a pipeline, classifies each one, and prints a summary. At this stage we're running this as a downstream GitLab CI job that triggers when any pipeline in the group fails.

from gitlab_client import get_failed_jobs, get_job_trace
from triage import classify_failure
import os, json

def main():
    pipeline_id = os.environ["FAILED_PIPELINE_ID"]  # passed via trigger variables

    print(f"🔍 Analysing pipeline {pipeline_id}...")
    failed_jobs = get_failed_jobs(pipeline_id)

    if not failed_jobs:
        print("No failed jobs found.")
        return

    print(f"Found {len(failed_jobs)} failed job(s)\n")

    for job in failed_jobs:
        job_id   = job["id"]
        job_name = job["name"]

        print(f"--- Job: {job_name} ({job_id}) ---")

        trace  = get_job_trace(job_id)
        result = classify_failure(job_name, trace)

        print(f"Type:      {result['failure_type']}")
        print(f"Reason:    {result['failed_reason']}")
        print(f"Fix:       {result['suggested_fix']}")
        print(f"Confidence:{result['confidence']}\n")

if __name__ == "__main__":
    main()

What It Looks Like Running

$ python main.py

🔍 Analysing pipeline 882341...
Found 2 failed job(s)

--- Job: test:unit (job_id: 4421098) ---
Type:       dependency
Reason:     ModuleNotFoundError for 'fastavro' — package missing from requirements.txt
Fix:        Add fastavro to requirements.txt and re-run the pipeline
Confidence: high

--- Job: build:docker (job_id: 4421099) ---
Type:       config
Reason:     Base image 'python:3.11-alpine' not found — image tag changed upstream
Fix:        Update Dockerfile FROM to python:3.11-alpine3.18 or use python:3.11-slim
Confidence: high

Alternatively , you can publish the triage for a pipeline failure to a slack channel

It works. Two jobs, two accurate classifications, two actionable fixes — all from raw log output that an engineer would have spent 15 minutes reading. The entire execution takes about 8 seconds.

The Reality Check — What's Already Broken

The moment it works is also the moment you see every limitation clearly. Here's the honest assessment after running it against a real day of pipeline failures:

1. Problem 1 — Context Window

   Real CI traces are often 50,000–200,000 characters. We're sending the entire raw trace to Claude. This is expensive, sometimes hits limits, and buries the actual error in noise. We fix this in next part

2. Problem 2 — Unstructured Output

   We asked the LLM to return JSON — and it mostly does. But sometimes it adds a preamble. Sometimes the keys are slightly different json.loads() fails silently. The fix — proper structured output with Pydantic — comes in subsequent parts.

3. Problem 3 — No Org Context

   The LLM gives you generic internet fixes. It doesn't know your internal packages, your team's conventions, or your past fixes for the same errors. RAG comes in subsequent parts.

4. Problem 4 — One Job at a Time

   Multiple failed jobs in a pipeline produce multiple separate outputs with no consolidated view. Each is analysed in isolation so the LLM can't detect that three failures share one root cause. We redesign this in subsequent parts.