Why Knowing AI Is Not Enough to Build or Modify Scalable Systems
Executive Overview
In the current era of Generative AI and agentic coding tools, there is a pervasive narrative: natural language is the new programming language, and anyone with a well-crafted prompt can build or modify production software.
As practitioners maintaining distributed enterprise platforms, high-throughput streaming backbones, and mission-critical cloud pipelines, we know the reality is far more nuanced. While AI excels at localized algorithm generation, boilerplate synthesis, and regex construction, it possesses zero inherent understanding of your system's global architecture, runtime topology, or blast radius.
When an engineer relies solely on AI without understanding the system design, the failure modes are rarely syntax errors or compile-time failures—modern LLMs write syntactically clean code. Instead, the failure modes are architectural: silent regressions, unintended pipeline mutations, catastrophic state overwrites, and topological miswires.
This post walks through a real production engineering case study: a routine feature request to link monthly remediation tracking tickets to an automated executive dashboard. We examine how an ungrounded AI generated code that was syntactically functional but architecturally disastrous—and how senior architectural thinking, structured feedback loops, and mental models of system topology turned a 300-line potential outage into a clean, resilient 15-line integration.
The Feature: A Routine Dashboard Integration
In enterprise cloud platforms, visibility is everything. Whether monitoring daily infrastructure health checks, distributed streaming lags, or security patch compliance across dozens of microservices, teams rely on centralized dashboards (such as Atlassian Confluence, Datadog, or Grafana) updated by automated CI/CD pipelines.
In our platform, automated auditing workflows run scheduled scans across multiple cloud accounts, compiling telemetry and publishing structured reports. Each month, engineering teams maintain an active Master Tracking Story in Jira (with domain-specific subtasks assigned to individual engineers) to remediate any operational findings.
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE DESIRED INTEGRATION │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ [Scheduled CI/CD Audit] ──► [Jira API] (Fetch Active Monthly Master Story) │
│ │ │
│ ▼ │
│ [Confluence REST API v2] │
│ │ │
│ ▼ │
│ [Executive Dashboard: Render Dynamic Remediation Progress] │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
The feature requirement was conceptually straightforward:
- When the reporting pipeline executes, query the ticketing system for the active monthly Master Remediation Story.
- Dynamically calculate remediation progress based on completed subtasks.
- Display a clean, styled status badge at the top of the centralized executive dashboard without disrupting existing metrics.
A colleague was assigned this task. Lacking deep familiarity with the broader repository and CI/CD topology, they delegated the implementation entirely to an AI coding assistant.
What followed is a textbook example of why prompting is not engineering.
The Anatomy of Blind AI Generation: 5 Critical Failure Modes
When the resulting Pull Request arrived for review, the diff revealed that the AI had generated hundreds of lines of code. It had created new YAML workflows, added Python scripts, and modified existing pipelines.
On the surface, the Python code was well-formatted, typed, and included docstrings. But beneath that veneer of correctness lay five critical architectural blunders:
WHAT THE AI PRODUCED
│
┌───────────────────────────┼───────────────────────────┐
▼ ▼ ▼
[Accidental Collateral] [Topological Miswiring] [Destructive Overwrite]
Touched unrelated tools Hooked into wrong job Would erase 6 months
in the repository (Metrics vs Audits) of historical trend data
1. Accidental Collateral Damage (Context Window Bleed)
The PR contained modifications to an entirely unrelated workflow: a centralized deployment validation tool. The AI had altered input parameters from data_product to repo_url, breaking the CLI contract of an independent operational utility.
Why this happened: Modern AI coding tools frequently index the entire workspace. When given a vague prompt like "update the workflow to take a repository", the model latched onto the first workflow matching its lexical search, blindly modifying an unrelated tool that shared common variable names.
2. Topological Miswiring (Pipeline Disconnect)
Instead of integrating into the scheduled vulnerability reporting workflow, the AI chained the new step as a downstream dependent of a bi-weekly Jira metrics summary workflow.
To the AI, words like "metrics", "report", and "Jira" belonged to the same conceptual cluster. But in our runtime topology, those workflows operate on completely different schedules, under different IAM roles, and target completely different operational stakeholders.
3. The Destructive Overwrite Hazard
The AI authored a 280-line standalone Python script to handle Confluence publishing. To handle page updates, it included the following page assembly logic:
# What the AI generated:
def _assemble_page(existing_body: str, live_html: str, history_entry: str) -> str:
anchor_match = _ANCHOR_RE.search(existing_body)
if not anchor_match:
# First run fallback: completely replaces page body!
return live_html + "\n" + _HISTORY_ANCHOR + "\n" + history_entry
# Discards everything before the anchor!
history_body = existing_body[anchor_match.end():]
return live_html + "\n" + _HISTORY_ANCHOR + "\n" + history_entry + "\n" + history_body
The catastrophic flaw: The executive dashboard contained over 6 months of historical trend charts, severity distribution donuts, 30-day non-prod vs. prod comparisons, and team SLA tables. Because the script looked for a custom anchor that did not exist on the live page, running this script would have instantly wiped out the entire Confluence page, replacing months of rich executive telemetry with a single solitary Jira badge!
The AI knew how to call the Confluence REST API v2, but it had no concept of page composition versus page ownership.
4. Hierarchy Blindness: Hub vs. Spoke
Enterprise platforms frequently implement a Hub-and-Spoke reporting topology:
ENTERPRISE HUB-AND-SPOKE TOPOLOGY
┌──────────────────────────────┐
│ Master Aggregator (HUB) │ ──► [Executive Dashboard]
└──────────────┬───────────────┘ (Needs Master Jira Badge)
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
[Pipeline Spoke 1] [Pipeline Spoke 2] [Pipeline Spoke N]
(Aircraft Towing) (Flight Schedules) (Ground Operations)
- Spokes (~20 microservice pages): Generated per pipeline by a distributed runner.
- Hub (1 Master Executive Dashboard): Generated by an aggregator script compiling data across all spokes into high-level KPI trends.
When the teammate attempted a second pass with AI, the AI generated a cleaner macro builder, but injected it into the spoke page updater (update_confluence.py) instead of the master dashboard aggregator (aggregate_master.py).
The result? The Jira remediation badge would have been stamped across 20 individual microservice pages (where it made no sense), while the main executive dashboard—the exact page reviewed by leadership—remained completely blank.
Figure 2: The follow-up code review: while the AI successfully cleaned up the code and built the styled alert macro, it wired the component to individual pipeline spoke pages instead of the centralized Master Dashboard.
5. Silent Boundary Failures: Escaping and Contracts
The AI queried Jira issue summaries and descriptions, embedding them directly into Confluence XHTML storage strings:
# Unescaped XML injection generated by AI:
body = f"<tr><td><strong>Description</strong></td><td>{issue['description']}</td></tr>"
In enterprise environments, engineers frequently write code snippets, shell commands, or HTML tags inside Jira descriptions (e.g., <URL>, a & b, <lambda_arn>). Injected raw into Confluence's strict XML storage format, unescaped &, <, and > characters cause the Atlassian API to immediately throw a fatal 400 Bad Request: Error parsing XML.
Figure 1: The initial GitHub code review detailing the 6 critical architectural and boundary issues introduced by ungrounded AI generation.
Why Prompting Alone Cannot Solve System Design
These failure modes illustrate a fundamental truth about software engineering in the age of LLMs:
┌─────────────────────────────────────────────────────────────────────────────┐
│ THE ARCHITECTURAL GAP IN GENERATIVE AI │
├─────────────────────────────────────────────────────────────────────────────┤
│ WHAT AI CAN SEE: │
│ • The immediate file in active buffer │
│ • Lexical syntax and type signatures │
│ • Localized function contracts and boilerplate │
│ │
│ WHAT AI CANNOT SEE: │
│ • The runtime execution topology of your CI/CD runners │
│ • Blast radius across disparate cloud pipelines and IAM boundaries │
│ • Page ownership semantics (embedded widget vs. page owner) │
│ • Established architectural precedents and team conventions in the repo │
└─────────────────────────────────────────────────────────────────────────────┘
Without an engineer providing architectural grounding, AI operates like an eager junior developer with infinite typing speed but zero spatial awareness. It solves the localized prompt with the bluntest tool available, often introducing massive architectural drift.
To successfully leverage AI in scalable systems, engineers must provide three layers of context that cannot be derived from a prompt alone:
1. Topology & Hierarchy
You must understand whether a component is an orchestrator, a worker, a hub aggregator, or a spoke publisher. If you do not know which script owns the target presentation layer, AI will simply attach code to the first file with a matching name.
2. State & Mutability Semantics
Is the operation additive, idempotent, or destructive? When writing to data stores or wikis, does the caller own the lifecycle of the document, or is it a guest embedding a widget? An engineer must enforce boundaries that prevent whole-page overwrites.
3. Repository Precedents & Reusability
In any mature enterprise repository, someone has likely already solved an identical pattern. In our codebase, we had already implemented this exact Jira-linking pattern in our Daily Health Check workflow (health-check/confluence/html_builder.py). Reusing established, battle-tested patterns guarantees consistency and avoids duplicating code.
The Solution: Architectural Mentorship & Structured AI Guidance
Engineering leadership is not about rejecting AI or berating colleagues who lean on it. It is about mentoring engineers to become the architectural guide that AI requires.
Rather than rejecting the PR with vague criticism, we formulated a dual-layer review approach:
1. Preserving Dignity & Psychological Safety
When reviewing code that was clearly generated by AI, calling it out aggressively or saying "AI did this wrong" erodes trust and damages morale.
Instead, we framed the feedback around established repository architecture and reusability:
"We actually solved this exact problem very recently in our Daily Health Check workflow (PR #147). To keep our reporting pages consistent and prevent overwriting existing dashboards, let's align this with our existing pattern."
2. Delivering "AI-Actionable" Architectural Feedback
Because we knew the colleague would feed the review comments back into their AI coding assistant (Amazon Q / Copilot), we structured the review comments with unambiguous architectural boundaries:
- Revert Collateral Files: Explicitly identify files to
git checkout/ revert. - Define UI Composition: Define a 3-tier visual hierarchy for the Confluence page:
- 🔴 Red Alert Panel: Existing problem scope (
Action Required — N Critical Findings). - 🔵 Blue Info Panel: Overall inventory breakdown (
Total Findings: 38846). - 🟢 Light Green Tip Panel: Active remediation tracking (
Master Jira: OMS-597 | 70% Subtasks Done). - ⚠️ Red Warning Fallback: Clear actionable guidance if the monthly master ticket is missing.
- 🔴 Red Alert Panel: Existing problem scope (
- Specify the Exact Insertion Point: Pinpoint the aggregator script (
aggregate_master.py) and the exact line inhtml_builders.pybetween the blue severity chips and the 30-day trend charts. - Mandate Security & Parsing Invariants: Require
html.escape()on all external strings.
THE FINAL 3-TIER DASHBOARD ARCHITECTURE
┌────────────────────────────────────────────────────────────────────────┐
│ 🔴 Red Alert Panel: Action Required (2225 Critical Findings) │
├────────────────────────────────────────────────────────────────────────┤
│ 🔵 Blue Info Panel: Total Findings: 38846 [Critical | High | Medium] │
├────────────────────────────────────────────────────────────────────────┤
│ 🟢 Light Green Panel: Master Jira: [OMS-597] | 7 / 10 Done (70%) │
├────────────────────────────────────────────────────────────────────────┤
│ 📊 30-Day Historical Trend Charts & Distribution Donuts │
└────────────────────────────────────────────────────────────────────────┘
When the colleague passed this structured, architecturally bounded prompt to their AI tool, the result was night and day:
- The 280-line destructive script was deleted.
- The unintended workflow modifications disappeared.
- A clean, 15-line helper was added to
html_builders.py, flawlessly embedding the remediation status directly under the blue metrics box without touching the rest of the page.
Production Lessons for the GenAI Era
As AI coding assistants become ubiquitous across engineering teams, the definition of a great software engineer is shifting rapidly. Here are the core lessons every engineering organization should embrace:
| Traditional Focus | GenAI Era Reality |
|---|---|
| Memorizing syntax and standard libraries | Understanding system design, data flow, and contracts |
| Writing routine boilerplate from scratch | Reviewing AI output for blast radius and architectural drift |
| Evaluating code purely by unit tests | Evaluating code by topological fit and operational failure modes |
| Debugging compiler errors | Debugging context bleed and silent runtime regressions |
1. Code Generation is Cheap; Architectural Cohesion is Expensive
Writing code is no longer the bottleneck. The bottleneck is knowing where code belongs, how it interacts with adjacent microservices, and what assumptions it makes about shared state.
2. Treat AI as a Fast Intern, Not a Principal Architect
An intern can write brilliant helper functions, but you would never allow them to deploy to production without verifying that they didn't hook into the wrong CI/CD workflow or overwrite shared storage. Treat every line of AI-generated code with the same architectural scrutiny.
3. Review for Boundaries, Not Just Logic
When reviewing AI-assisted PRs, spend 80% of your time checking:
- Did this PR touch files outside its domain?
- Does this change preserve established repository conventions?
- What happens if the downstream API fails or returns unexpected characters?
- Does this code operate at the right layer of the system hierarchy?
Conclusion
Knowing AI is not enough to build, scale, or maintain enterprise systems.
AI can suggest code, but it cannot architect a platform. It can write an API call, but it cannot comprehend the operational blast radius of a damaged dashboard. The engineers who will thrive in this new landscape are not those who prompt the fastest, but those who possess the deep architectural intuition to verify, constrain, and guide AI toward resilient, scalable solutions.
Are you leveraging AI assistants in your CI/CD and cloud automation pipelines? How is your team adapting code review practices to catch architectural drift? Let's connect on LinkedIn or explore more notes across the blog.
You've reached the end. Explore more engineering notes.
More Engineering Notes