Thousands of Agents? Stop SSHing! 🤫 (Mastering OpAMP)

Aug 25, 2026

—

Hello! Today, I’d like to talk about a thorny problem that any developer, SRE, or DevOps engineer operating large-scale systems has likely encountered: the management of numerous telemetry agents.

When you had 10 servers, it was fine. But what happens when that number grows to 100 or 1000? You might have to connect to hundreds of servers just to change one configuration file, and a version update could mean pulling an all-nighter. 😱

Let’s explore a standard protocol that can solve this terrible situation: OpAMP (Open Agent Management Protocol). We’ll delve into how OpAMP makes all of this possible and uncover its core principles in an easy-to-understand and detailed way!

🤔 The Root of the Problem: Why is Agent Management Difficult?

Traditional methods have several limitations.

Manual changes after SSH connection: This is the most primitive method, highly prone to human error, and becomes nearly impossible as scale increases.
Configuration management tools like Ansible/Puppet: These are excellent tools, but they focus on periodic state synchronization rather than real-time bidirectional communication. It’s cumbersome when you want to change settings “right now.”
Periodic HTTP Polling by agents: This method involves agents constantly asking the server, “Do you have anything for me to do?” It lacks real-time capability and generates unnecessary network traffic.

These methods share a common problem: the difficulty of achieving continuous, real-time, bidirectional communication between agents and the management server. OpAMP precisely addresses this point.

✨ The Magic of OpAMP: “Persistent Bidirectional Communication Channel”

The core of OpAMP can be summarized in one sentence.

“The agent (client) initiates a connection to the server and continuously maintains that connection.”

Why is this important?

Client-Initiated Connection 🏠

Communication always originates from the agent towards the server.
Thanks to this, even agents hidden behind firewalls or NAT can connect to the server without issues. The server doesn’t need to know the agent’s IP or access it directly. It’s like a person inside a house opening the door and stepping outside.

Using WebSocket 🤝

Once a connection is established, it is continuously maintained via WebSocket.
This ‘tunnel’ allows the server to send real-time commands to the agent whenever needed. Instead of constantly asking, “Do you have anything for me to do?”, the server can directly say, “Do this!”

🚀 So, What Can You Do with OpAMP? (4 Core Features)

Through this intelligent communication channel, OpAMP provides the following powerful features.

1. Agent Identification and Status Reporting (Agent → Server) 🙋‍♂️

Agents don’t just connect; they periodically report their status to the server.

“Who am I?”: Upon connection, the agent informs the server of its unique ID, version, execution environment (OS, etc.), and currently applied configuration.
“Am I okay right now?”: Subsequently, it periodically reports its health status (CPU/Memory usage, etc.) and whether any errors have occurred.

This allows administrators to grasp the status of thousands of agents at a glance from a central dashboard.

2. Remote Configuration Management (Server → Agent) 📝

This is the most powerful feature! Administrators no longer need to connect to each server individually.

“Do it this way from now on!”: The server can push new configurations to agents via the OpAMP channel. (e.g., “Change the data collection endpoint from A to B,” “Increase the log level to debug”)
“Understood, I’ve changed it!”: After applying the settings, the agent reports the success status and the changed configuration details back to the server.

This allows for consistent configuration changes across all agents with just a few clicks, and the results can be verified instantly.

3. Agent Package Management (Server → Agent) 📦

Agent updates are now also possible remotely.

“A new version is out, update!”: The server sends commands to agents to upgrade to a specific version or downgrade to a previous version if issues arise.
“Update complete!”: The agent downloads and performs the update according to the instructions, then reports the result back to the server.

4. Credential Management (Server → Agent) 🔑

Sensitive information like API keys or certificates no longer needs to be hardcoded into configuration files.

“Use this secret key”: The server securely transmits credentials like API keys, needed to send telemetry data to the backend, via the OpAMP channel.
“Understood!”: The agent receives and uses this information in memory, reducing the risk of sensitive data being exposed in files. Furthermore, keys can be easily rotated centrally if they expire or are compromised.

🗺️ Overview of the Entire Process

Let’s take a look at the overall context of how this entire process unfolds.

Start 🚀: The agent starts and attempts to establish a WebSocket connection to the OpAMP server.
Introduction 🙋‍♂️: If the connection is successful, the agent reports its status (version, current configuration, etc.) to the server.
Command Issuance 📜: The server checks the agent’s status and, if changes are needed, responds with new configurations or update commands.
Command Execution ✅: The agent executes the commands received from the server. (e.g., configuration changes, package updates)
Result Reporting 📬: The agent reports the command execution result (success/failure) and its updated status back to the server.
Continuous Communication 🔄: This process repeats periodically as long as the connection is maintained, ensuring the agent is always under central control.

👀 What Does Actual Communication Data Look Like?

OpAMP typically uses Protocol Buffers, but for better understanding, let’s represent it in JSON format.

1. Message from Agent reporting status to Server (Agent → Server)

// AgentToServer.proto (virtual)
{
  "instance_uid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "agent_description": {
    "identifying_attributes": [
      { "key": "service.name", "value": "my-app-agent" },
      { "key": "host.name", "value": "prod-server-01" }
    ],
    "non_identifying_attributes": [
      { "key": "agent.version", "value": "1.2.0" },
      { "key": "os.type", "value": "linux" }
    ]
  },
  "status_report": {
    "uptime_seconds": 3600,
    "health": {
      "healthy": true,
      "last_error": ""
    },
    "effective_config": {
      "config_map": {
        "receivers": { "otlp": { "protocols": { "grpc": { "endpoint": "collector:4317" } } } },
        "exporters": { "logging": { "loglevel": "info" } }
      }
    }
  }
}

2. Message from Server commanding Agent to change settings (Server → Agent)

// ServerToAgent.proto (virtual)
{
  "instance_uid": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "command": {
    "type": "SET_REMOTE_CONFIG",
    "remote_config": {
      "config": {
        "config_map": {
          "receivers": { "otlp": { "protocols": { "grpc": { "endpoint": "collector:4317" } } } },
          "exporters": { "logging": { "loglevel": "debug" } } // Change loglevel to debug!
        }
      },
      "config_hash": "sha256:abcdef123456..."
    }
  }
}

By exchanging structured data as shown above, the agent applies the contents of remote_config to its own configuration and reports the results back to the server.

Conclusion

OpAMP is revolutionizing large-scale telemetry agent management by standardizing a WebSocket-based persistent bidirectional communication channel. Break free from the pain of manual operations and manage your entire agent fleet efficiently and securely from a central location!

OpAMP is being actively adopted in modern open-source projects like OpenTelemetry Collector and is highly likely to become the standard management method for large-scale distributed system monitoring in the future. Why not apply it to your system right now? 😉