Network observability platform with automated NOC triage

A mid-size logistics operator running twelve regional hubs and more than four hundred warehouse access points found its network operations center overwhelmed by telemetry volume with almost no actionable correlation. The NOC processed more than three thousand four hundred events per week from SNMP traps, syslog floods, carrier emails, and ad-hoc phone calls from dock managers reporting 'the Wi‑Fi is slow.' Engineers carried tribal knowledge about which MPLS provider served which hub and which core switch failure would silence handheld scanners—but that knowledge lived in spreadsheets, not in monitoring systems. Mean time to resolve P2 network incidents averaged two point eight hours, with the majority spent identifying scope rather than executing fixes. Leadership asked Dippa to unify observability, reduce alert noise without hiding real outages, and automate first-line triage for event classes where safe runbooks already existed.

Over eleven weeks we deployed a network observability platform centered on Kentik ^[1] for flow and path analytics, Grafana ^[2] for operator dashboards, Prometheus for metric retention, and Ansible ^[3] for closed-loop auto-triage on an allowlisted playbook set. NetBox ^[4] supplied device roles, site hierarchy, and dependency context at ingest time so every alert arrived pre-enriched with business impact tags. After sixty days in production, alert volume paging on-call engineers dropped seventy-one percent, fifty-eight percent of L1-eligible events cleared through auto-triage without human touch, and P2 mean time to resolution improved forty-one percent. This case study walks through the operating constraints we respected, the telemetry model we built, intent-based alerting rules that replaced static thresholds, and the governance model that kept automation from becoming automation-driven outage.

Operating environment and pain points

The logistics operator's network spanned three distinct domains: MPLS-connected hub sites with datacenter-grade cores, warehouse campus fabrics with a mix of vendor switches and legacy Wi‑Fi controllers, and cloud-attached VPN aggregates for remote freight offices. Each domain exported telemetry through a different toolchain—carrier portals for circuit status, SNMP polling servers with inconsistent polling intervals, a legacy syslog server with no structured parsing, and synthetic ping tools whose results were not correlated with application impact. PagerDuty incidents often contained subject lines like 'LINK-DOWN on sw-core-07' with no indication that sw-core-07 served Hub 7's warehouse management system and forty dock scanners.

NOC L1 operators escalated aggressively because false negatives had previously resulted in discipline; the rational response to ambiguous alerts was to page senior engineers. Those engineers, in turn, spent the first thirty to forty minutes of every incident reconstructing topology from NetBox tabs, Slack threads, and memory. Capacity planning was similarly reactive: link utilization reports were quarterly spreadsheets exported from different tools that disagreed by five to twelve percent on the same interfaces. Finance challenged circuit upgrade requests because neither side could produce a single trusted data source. The program's executive sponsor defined success as a NOC that sees the network as one system—not six monitoring tools producing six conflicting narratives.

• No alert may page on-call without site name, device role, and dependent business service tags attached.
• Auto-triage may execute only runbooks explicitly approved by network engineering and security.
• All telemetry retained thirteen months for capacity planning, vendor disputes, and audit.
• Dashboards must be usable by NOC L1 without requiring CLI access or deep BGP knowledge.
• Maintenance windows must suppress flap-sensitive alerts without silencing genuine hard-down events.

Discovery and data model design

Discovery began with a two-week telemetry census: inventory every export path, measure event rates per source, and classify alert types by whether they had ever resulted in customer-visible impact. We found that sixty-two percent of weekly SNMP traps were interface up/down flaps on access ports connected to end-user devices—not infrastructure faults. Another eighteen percent were duplicate traps generated by overlapping polling engines re-alerting on the same condition. Only nine percent of events correlated with incidents logged in the past twelve months, yet all nine percent arrived with the same severity as noise.

We designed a canonical entity model: Site → Device (with role) → Interface → Service dependency. NetBox became the enrichment authority—device roles like hub-core, warehouse-access, or edge-router mapped to business services such as WMS, dock scanners, VOIP, or corporate office LAN. Kentik received flow exports and BGP session metadata; Prometheus scraped SNMP exporters with standardized scrape intervals; synthetic probes ran from twelve hub vantage points targeting critical application endpoints. Enrichment ran at ingest via a lightweight pipeline so Grafana and PagerDuty never displayed raw device names without context ^[1].

Dippa field documentation

Observability stack architecture

Kentik anchors the observability stack for flow-aware analysis: which hubs send traffic toward WMS APIs, which transit providers carry the majority of inter-hub replication, and where microbursts precede interface discards. We enabled BGP monitoring integration so session state changes include AS path and peer role—transit versus peering versus warehouse CE—without manual annotation. Prometheus retains high-resolution SNMP metrics with recording rules that pre-compute five-minute utilization averages to stabilize dashboard queries during morning shift change when thirty operators refresh panels simultaneously.

Grafana organizes dashboards by operator persona: L1 gets a traffic-light hub overview with deep links to device detail; L2/L3 gets BGP, MPLS, and Wi‑Fi controller panels; capacity planners get twelve-month trend boards with export to finance-friendly CSV. Dashboards query enriched labels exclusively—operators filter by site or service, not by memorized hostname prefixes. Synthetic probes validate HTTPS reachability to internal APIs and UDP latency to handheld scanner gateways; probe failures attach to the same service tags as SNMP events so incidents deduplicate correctly.

Telemetry sources unified

Six previously siloed telemetry sources feed the platform: SNMP from network devices and standardized exporters; NetFlow/IPFIX exported to Kentik; BGP session state from route servers and edge routers; synthetic probes from hub-based agents; structured syslog from cores and controllers funneled through a parsing layer; and carrier circuit status APIs polled every five minutes for known provider maintenance. Legacy warehouse Wi‑Fi controllers in two facilities still emit non-standard MIB traps—we deployed a parser shim that normalizes trap OIDs into the canonical event schema until hardware refresh in Q4 replaces those controllers.

Retention policies align with operational and legal requirements: thirteen months of metrics and flows online, with cold storage for forensic exports. Vendor disputes over circuit SLA credits now begin with Kentik-backed utilization graphs rather than anecdote. The enrichment pipeline adds roughly forty milliseconds median latency at ingest—acceptable given the goal is actionable alerts, not microsecond-level trading telemetry.

Intent-based alerting

Static thresholds caused most of the noise. A core uplink at eighty-five percent utilization might be normal for nightly batch replication; the same reading at 14:00 during picking hours indicates imminent impact. We replaced hundreds of static rules with intent-based policies encoding duration, maintenance context, device role, and dependency severity. BGP session down pages only if idle longer than ninety seconds, the peer is classified transit or hub-critical, and the device is not in a NetBox-flagged maintenance window. Access port flaps increment a counter but page only if linked to a known infrastructure device or if flap rate exceeds five events in ten minutes on uplink-class ports.

# Intent alert rule — BGP peer down (production excerpt)
name: bgp_peer_down_critical
description: Transit or hub-critical BGP idle >90s outside maintenance
match:
  metric: bgp.session.state
  labels:
    peer_role: [transit, hub_critical]
  value: idle
  duration: 90s
enrich:
  source: netbox
  attach:
    - site.name
    - device.role
    - dependent_services
    - last_config_backup_url
suppress_if:
  - netbox.maintenance_window: true
  - device.tags: [lab, decommissioning]
route:
  severity: P2
  pagerduty: network-oncall
  runbook: https://runbooks.internal/bgp-peer-down
auto_triage:
  allowed_if:
    - maintenance_window: false
    - peer_type: transit
  action: ansible/playbooks/check_transit_provider_status.yml
  timeout_seconds: 120
  on_success: resolve_with_note
  on_failure: escalate_l2

Intent rules are version-controlled in Git with pull-request review from network engineering—mirroring how software teams manage feature flags. A staging environment replays historical alert streams against rule changes so engineers can measure noise reduction before production promotion. During the first intent-rule rollout, replay analysis showed that a proposed warehouse Wi‑Fi rule would have suppressed two genuine controller failures; we adjusted duration thresholds before deploy. That replay discipline prevented the common failure mode of 'alerting projects' that trade pager fatigue for missed outages.

Flap suppression and deduplication

Flap suppression uses sliding windows keyed by device, interface, and alert type. Correlated events collapse into a single PagerDuty incident with a timeline of constituent traps—operators see 'ge1/0/24 flapped 6 times in 8 min' rather than six sequential pages. Deduplication crosses sources: if synthetic probes and SNMP agree an API endpoint is unreachable from Hub 3, one incident opens with both signals attached. Carrier maintenance emails ingest as informational events linked to affected sites; if SNMP remains clean during the window, no page fires.

• Sustained threshold violations required before page—transient spikes become dashboard annotations only.
• Maintenance windows sourced from NetBox; overrides expire automatically after scheduled end.
• P1 reserved for dual-uplink hub-down or multi-site correlated failures affecting WMS.
• L1-eligible auto-triage classes explicitly tagged; everything else escalates with full enrichment.

Dippa field documentation

Auto-triage with Ansible

Auto-triage executes only against an allowlisted set of playbooks reviewed by network engineering and security. Each playbook is read-only or low-risk by design: query transit provider status APIs, verify interface error counters, bounce access ports in err-disabled state on known-safe templates, or collect show commands and attach output to the incident. Destructive actions—BGP neighbor resets, core routing changes, firmware pushes—remain manual with L2 approval. Ansible Automation Platform runs playbooks in isolated execution environments with credentials vaulted and scoped per site; a playbook cannot target devices outside the incident's enriched site list.

When an eligible alert fires, the orchestrator invokes the mapped playbook, waits up to two minutes, and posts structured results back to PagerDuty as incident notes. Success criteria are explicit: for err-disabled access ports, oper-state up and error counters stable for sixty seconds; for provider checks, API returns no active outage matching circuit ID. If auto-triage succeeds, the incident resolves with a machine-generated summary L1 can audit. If it fails or times out, escalation to L2 includes playbook stdout and the last known-good config link from NetBox.

# Ansible playbook excerpt — access port err-disabled recovery
- name: Recover err-disabled access port (allowlisted roles only)
  hosts: localhost
  gather_facts: false
  vars:
    device: "{{ incident.device }}"
    interface: "{{ incident.interface }}"
  tasks:
    - name: Validate device role is warehouse-access
      ansible.builtin.assert:
        that:
          - incident.device_role == 'warehouse-access'
        fail_msg: Device not in auto-triage allowlist

    - name: Collect interface status
      ansible.netcommon.cli_command:
        command: show interfaces {{ interface }} status err-disabled
      register: if_status
      delegate_to: "{{ device }}"

    - name: Bounce interface if err-disabled
      ansible.netcommon.cli_config:
        config: |
          interface {{ interface }}
           shutdown
           no shutdown
      when: "'err-disabled' in if_status.stdout"
      delegate_to: "{{ device }}"

    - name: Post result to incident timeline
      uri:
        url: "{{ pagerduty_note_api }}"
        method: POST
        body_format: json
        body:
          note: "Auto-triage: {{ interface }} on {{ device }} recovered"

Governance reviews occur monthly: which playbooks fired, false positive rate, and any operator overrides. One playbook was retired after it cleared a port connected to a misconfigured industrial label printer that repeatedly err-disabled due to duplex mismatch—auto-triage restored the port but the underlying fault needed a physical repair ticket. We added a dependency check requiring stable link training before auto-resolve. Security audited credential scopes and confirmed playbooks could not exfiltrate config to external endpoints.

NetBox enrichment and dependency graph

NetBox holds site hierarchy, device roles, circuit IDs, and custom fields linking devices to business services. The enrichment service subscribes to NetBox webhook events and caches topology graphs refreshed every five minutes. When an alert references sw-hub7-core-01, enrichment attaches Hub 7, role hub-core, dependent services WMS and dock scanners, upstream MPLS circuit IDs, and the on-call runbook URL. Dependency edges are modeled conservatively: if a core switch fails, dependent access switches and Wi‑Fi controllers inherit impact tags without waiting for downstream traps—speeding scope assessment.

NetBox maintenance windows integrate with alerting suppression and auto-triage locks. During planned core upgrades, BGP intent rules downgrade to ticket-only notifications; auto-triage playbooks refuse to run against devices tagged maintenance. This prevents the embarrassing class of automated 'recovery' actions fighting controlled change windows—a failure mode seen in early automation pilots at other enterprises ^[4].

NOC workflow transformation

Before go-live, NOC shifts started with operators opening six browser tabs and a spreadsheet of hub-to-provider mappings. After cutover, L1 opens a single Grafana home board filtered to their shift's watched regions. Red tiles show enriched incidents with service impact counts—'Hub 7 · hub-core · WMS + scanners affected.' Operators follow runbook links embedded in PagerDuty, review auto-triage notes if present, and escalate with context instead of raw hostnames. Training included four supervised shifts and two game-day exercises simulating transit provider outages and warehouse Wi‑Fi controller failures.

We measured operator task time for the top ten alert types before and after enrichment. Median time to identify affected service dropped from nineteen minutes to under two minutes. Escalations to L2 now include attachment of recent Kentik flow views and Prometheus graphs auto-linked in the incident template—reducing back-and-forth requests for 'can you pull utilization on ge-0/0/1.' Two FTE previously dedicated to alert triage reallocated partially to capacity planning projects using the same Grafana trend boards finance trusts.

Game-day validation and rollout

Rollout proceeded in three phases: observe-only enrichment for two weeks (alerts enriched but legacy rules still paging), parallel intent rules with shadow comparison metrics, then full cutover with legacy polling decommissioned site by site. Game-day one simulated dual transit loss at Hub 4; intent rules opened a single P1 with correct service tags and auto-triage confirmed provider outage via API—no manual bounce attempts. Game-day two injected access-port err-disabled traps on lab devices mapped into production dashboards in read-only mode; auto-triage recovered ports and resolved incidents within ninety seconds.

Legacy SNMP servers decommissioned after shadow comparison showed less than zero point five percent divergence on critical metrics for fourteen consecutive days—meaning enriched alerts matched or outperformed legacy detection. Wi‑Fi controllers on the parser shim remained monitor-only for auto-triage until Q4 refresh; their alerts enrich and route correctly but skip automation pending standardized MIB support.

Results after sixty days

Sixty days post cutover, the metrics met or exceeded program targets. Alert volume reaching on-call engineers decreased seventy-one percent compared to the pre-program baseline week normalized for seasonal shipping volume. Fifty-eight percent of L1-eligible events cleared through Ansible auto-triage without human action; the remaining forty-two percent escalated with enrichment that L2 reported sufficient for immediate action in seventy-six percent of cases. P2 mean time to resolution improved forty-one percent—from two point eight hours to one point seven hours median.

• Seventy-one percent alert noise reduction with zero missed P1 outages in the observation window.
• Fifty-eight percent of L1-eligible events auto-cleared with audited playbook logs.
• Forty-one percent faster P2 MTTR; two FTE reallocated from triage to capacity planning.
• Six telemetry sources unified under one enrichment model with thirteen-month retention.
• Legacy Wi‑Fi controllers on parser shim until hardware refresh—auto-triage explicitly disabled.

Capacity planning and downstream benefits

Unified telemetry unlocked capacity workflows previously blocked by data disputes. Planners use Kentik twelve-month trends to justify MPLS upgrades at Hub 9 and defer expensive augmentation at Hub 2 where flow analysis showed peering optimization sufficed. Finance approved two circuit changes in one quarter—the fastest approval cycle the network team had seen—because dashboards exported consistent numbers across SNMP and flow sources. Vendor SLA credit requests now attach automated reports, recovering twelve thousand dollars in the first quarter from documented provider outages that previously lacked evidentiary graphs.

Security operations requested read-only Grafana access to hub-edge dashboards for correlating firewall deny spikes with WAN events—a collaboration that emerged because the NOC stopped hoarding fragmented tools. The SOC now links DDoS scrubbing triggers to Kentik flow anomalies without opening separate carrier portals.

Risks, limitations, and roadmap

Residual risks are documented and owned. Two warehouse Wi‑Fi controller models remain on custom MIB translation with monitor-only automation status until Q4 hardware refresh. Auto-remediation for BGP neighbor bounce is limited to lab-approved scenarios—not production—after a tabletop exercise showed provider-side asymmetry could worsen conditions if mistimed. Playbook catalog growth is gated: new automation requires replay validation against ninety days of historical alerts and sign-off from network and security engineering.

Phase two roadmap items include expanding auto-triage to certificate expiry on management interfaces, integrating change-management tickets so intent rules auto-correlate with approved maintenance, and ML-assisted anomaly detection on flow baselines—explicitly secondary to rule-based intent alerting because the NOC prioritized explainability over black-box scores ^[5]. The operator extended Dippa's engagement to migrate remaining remote office VPN telemetry into the same platform, treating this architecture as the long-term operations backbone rather than a NOC-only science project.

Lessons for network operations teams

Three lessons transfer to similar environments. First, enrichment beats aggregation: collecting more traps into one mailbox without site and service context does not reduce MTTR—it merely centralizes noise. NetBox (or any CMDB with accurate roles) must be in the critical path at ingest. Second, auto-triage requires governance as rigorous as production change control; allowlists, replay testing, and monthly audits prevent automation from becoming a denial-of-service against your own engineers. Third, design dashboards for L1 first; architect-grade BGP visualizations are useful but not if operators cannot navigate from red tile to runbook in three clicks.

Teams evaluating Kentik, Grafana, and Ansible together should sequence work as we did: unify the data model and enrichment, deploy intent alerting with replay validation, then add closed-loop automation only for well-understood event classes. Skipping straight to Ansible playbooks on raw SNMP traps recreates the same trust problems as manual triage—just faster and less visible.

Integration with change management and ITSM

Alerts do not exist in isolation from change windows. We integrated the ITSM tool so approved change tickets automatically create NetBox maintenance windows with matching time bounds and device lists. Intent rules read those windows at alert evaluation time—engineers no longer manually suppress paging before every planned core upgrade. Post-change, synthetic probes and SNMP baselines compare against pre-change captures; unexpected deltas open a linked incident even if no trap fired yet. That closed loop shortened verification after maintenance from an average of forty-five minutes of manual CLI checks to twelve minutes of dashboard review.

Incident records now store Kentik deep links and Grafana panel snapshots as permanent attachments, improving post-incident review quality. Root-cause categories tagged in PagerDuty feed monthly reports to network leadership: provider outages, config change, hardware failure, and unknown. The unknown bucket shrank from twenty-eight percent of P2 incidents pre-program to nine percent after enrichment—evidence that better context accelerates classification, not just clearance speed.

Conclusion

This program showed that a logistics NOC drowning in disconnected telemetry can recover operational headroom without hiring proportional headcount. Kentik and Prometheus supply the observability depth; Grafana makes it legible; NetBox enrichment makes it actionable; Ansible auto-triage handles repetitive clearance safely within guardrails. Seventy-one percent alert noise reduction and forty-one percent faster P2 resolution were not magic thresholds—they emerged from intent rules tuned with historical replay, conservative automation scope, and operator workflows redesigned around enriched incidents rather than raw traps. For network leaders facing similar pain, the imperative is to invest in context before automation: once every page tells you what broke and who cares, the path to trustworthy auto-triage becomes clear.