An AI agent is a program that executes actions chosen by a language model using actual tools, continuing its work. In this context, a safety mechanism should not merely display “do not do that” messages, but must stop unauthorized requests at the execution boundary. A log indicating a stop is different from evidence that a request never reached the target server.
If you’ve looked into ARTEX’s operational structure and the role of harnesses and guardrails, the following questions can be tested on a small scale: Do forbidden requests truly not go out? Does a failed attempt still consume a call budget? Does the next request not proceed after an interruption?
This time, I built a local HTTP server and a single tool using only Python’s standard library and verified it directly. This is a deterministic test of the tool execution boundary, without using external APIs or actual LLMs. It does not imply that ARTEX as a whole or LLM automated tests have succeeded. The observation date is October 10, 2026, and the execution environment is macOS with Python 3.9.6.
1. First, Define What “Pass” Means
The server only accepts requests on a temporary port at 127.0.0.1. Instead of business data, it uses fixed responses and an in-memory reception counter. The server increments the counter when it actually reads a request, allowing verification of arrival independently of the status returned by the tool.
Initially, we verify that one normal request arrives and receives an HTTP 200 response. This is because a program that blocks even normal requests might show 0 forbidden requests. After passing this normal control test, we inject the failures listed below.
| Test | Injected Condition | Pass Judgment |
| Forbidden Request | Non-existent ID and URL-like input | All rejected, server reception count 0 |
| Redirect | Allowed path responds with 302 to /forbidden | Only first request received, destination 0 |
| Failure/Retry | Repeated 503 responses, retry budget 2 | Received twice, third transmission blocked |
| Communication Failure | Connection closed without response after receiving request | Budget 1 exhausted, next transmission blocked |
| Request Timeout | Response withheld, request limit 0.25 seconds | Connection cleanup, next call blocked |
| Overall Timeout | Request limit 2 seconds, overall supervision limit 0.35 seconds | Execution task terminated, next call blocked |
| Explicit Cancellation | Cancel execution after confirming server reception | Cancellation complete, additional reception 0 |
The budget here refers to the number of allowed call attempts to initiate a transmission. Inputs rejected by policy do not initiate a transmission attempt and thus do not consume this budget. Failed attempts after initiating a request are not refunded. Cost, tokens, or concurrency are not the budgets measured here.
2. Do Not Give the Tool URL Selection Authority
The tool’s input is not a URL, but case IDs like OK, FAIL, SLOW_CANCEL. A list within the code converts these IDs into fixed paths, and the connection target is always the practice server’s 127.0.0.1 and a temporary port. There is no path to change the host or port via input values.
CASES = {
"OK": "/ok", "FAIL": "/fail", "DROP": "/drop",
"REDIRECT": "/redirect", "SLOW_REQUEST": "/slow-request",
"SLOW_GLOBAL": "/slow-global", "SLOW_CANCEL": "/slow-cancel",
}This choice does not mean that a general-purpose URL tool has been safely completed. It merely removes input freedom not required for this practice. OWASP also describes excessive functionality, permissions, and autonomy as risk factors, recommending providing narrower functionality than tools that accept arbitrary commands or URLs. OWASP LLM06:2025 Excessive Agency
Redirects are also subject to the same boundary. If the client automatically follows the Location header in a response after only checking the initial URL, it could navigate to a destination not initially allowed. The small HTTP client in this practice returns 3xx status as redirect_blocked and does not make a second request. Indeed, /redirect was received once, and /forbidden was received 0 times.
This list represents internal program control. It is not an operating system isolation mechanism that prevents other code from directly opening sockets or calling other tools. To connect to a real agent, it is necessary to separately verify that all execution paths pass through this control.
3. Consume Budget Before Failure
If the budget is calculated based on successful attempts, a continuously failing tool might never reach its limit. This implementation reserves an attempt before transmission and does not refund the deducted value even if an error or timeout occurs. Below is the actual call boundary from the attached practice file.
async def call(self, case_id):
if self.stopped is not None:
return "stopped"
if case_id not in CASES:
return "forbidden"
if self.spent >= self.budget:
return "budget_exhausted"
self.spent += 1 # Deducted before I/O, no refund even on failure
try:
return await asyncio.wait_for(
self.exchange(CASES[case_id]), self.request_timeout)
except asyncio.TimeoutError:
self.stop("request_timeout")
return "request_timeout"
except asyncio.CancelledError:
self.stop("cancelled")
raise
except (OSError, asyncio.IncompleteReadError,
asyncio.LimitOverrunError):
return "transport_error"When attempting to call the case returning 503 three times, the result was http_503 → http_503 → budget_exhausted. The server received it twice, and the third request was not made. The case of disconnecting without a response also exhausted budget 1 with a single failure. This result aligns not only with failure records but also with the number of receptions and attempts used.
The practice does not include hidden retries from an HTTP library. The `plan()` function, responsible for call repetition, re-invokes the same tool after a failure, undergoing the same budget check each time. The author’s recommendation is to identify the layer where retries occur in a real system and link them to the same attempt budget. If SDK internal retries and higher-level execution loop retries operate independently, the actual number of transmissions cannot be determined solely from the number of higher-level calls.
4. Separate Request Limits and Overall Supervision Limits
The time spent waiting for a single request’s response and the total time for an entire plan to execute are different limits. If a new limit starts with each request, the total time for a series of requests can become long. This implementation bundles connection, transmission, and response reading into a single coroutine to apply a request limit, and a separate supervisor function waits for the entire plan.
In the request limit test, the server withheld its response. With a 0.25-second limit, the call ended in approximately 0.253 seconds with a request_timeout, and the next call to the same tool was stopped. This is the chosen policy for this practice to stop the operation after a request timeout. It is not a specification that all services must stop the entire operation on every timeout.
In the overall supervision test, the request limit was increased to 2 seconds, and the supervision limit was set to 0.35 seconds. It terminated with a supervisor_timeout in approximately 0.352 seconds, before the request limit. The supervisor function first records the stopped state, then cancels the execution task, and waits until it terminates.
tool.stop("supervisor_timeout")
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await taskCalling cancel() alone does not determine that execution has ended. Inside the tool, CancelledError is re-raised, and connections are closed in a finally block. The supervisor returns after that cleanup is complete. Python documentation also explains that tasks can swallow cancellation exceptions, and wait_for() might take longer than the specified time while waiting for cancellation to complete. The measurements above are observational results and do not guarantee absolute termination times. Python Task Cancellation and Timeouts
This code sequentially calls tools within a single event loop. It did not test parallel workers, synchronous code blocking execution, or libraries that ignore cancellations. In such environments, additional verification of concurrent budget consumption for shared budgets and the boundaries of already initiated requests would be necessary.
5. Cancellation Does Not Revert Already Received Requests
The explicit cancellation test first waits for the event that a request has arrived at the server. Then, it records a stopped state to prevent new calls and cancels the execution task. It verifies whether the task’s cancelled() is true, whether the next normal call is rejected, and whether the reception count does not increase further.
These three conditions passed. However, when the pending processing on the server was released, the server’s synthetic effect counter still increased by 1 even after client cancellation. The same effect remained in the request timeout and overall timeout tests. The effect was only an increase in a number in memory, and no files or external resources were changed.
Therefore, the “stop” verified here means ending the client’s execution and not transmitting subsequent tool calls. It does not guarantee that the target server has canceled or reverted already received work. Nor is it a test that proves the entire race condition where requests initiated before the cancellation point arrive later.
If dealing with actual modification operations, the author’s recommendation is to separately design server-side job cancellation support, idempotent keys to prevent duplicate effects, and result inquiry with compensatory actions if necessary. These functionalities were not implemented or verified in this practice.
When implementing tools with shell commands, the boundaries also expand. subprocess.run(timeout=…) describes the behavior of terminating and waiting for child processes it started upon timeout. This alone cannot be interpreted as all descendant processes or the effects of already transmitted requests disappearing. Furthermore, Popen.communicate(timeout=…) does not terminate child processes solely on timeout, requiring separate cleanup. This practice did not create subprocesses and did not test process tree termination. Python subprocess official documentation
6. Reproduce with a Single Small File and Read Results
Save the attached agent_safety_lab.py in an empty practice directory and run it. It requires Python 3.9 or higher and an execution environment capable of opening local sockets; no package installation, accounts, or API keys are needed. There are no options to input URLs or target servers.
python3 --version
python3 agent_safety_lab.py --output results.jsonThe actual output saved this time is as follows:
PASS allowed_positive_control
PASS forbidden_no_reception
PASS redirect_not_followed
PASS http_failures_retries_spend_budget
PASS transport_failure_spends_budget
PASS request_timeout_stops_new_calls
PASS supervisor_timeout_drains_execution
PASS cancel_blocks_new_calls_without_rollback
PASS cleanup
checks=9 passed=9 total_received=8Including normal control and cleanup, all 9 tests passed, and the total server reception count was 8. After termination, 0 pending tasks, 0 server processing tasks, 0 open client connections, and the listening socket closure were confirmed. The practice created 0 subprocesses and deleted the temporary working directory used for execution. The manuscript, code, and measurement JSON were preserved for review.
If execution is blocked, first check the location of the error. If local port binding is refused, the server has not started, and this is not counted as a success for the blocking test. The first run also had the execution sandbox blocking the binding, and the above results were obtained after re-running within the approved local practice execution scope. Security settings were not changed, nor was an external server substituted. Measurement times may vary in different environments, and if a FAIL or exception occurs, the results should be preserved and the cause investigated.
When verifying a real agent next time, instead of simply adopting the pass results from this file, the same judgment criteria must be applied to all tool, retry, and interruption paths of that agent. To explain at which boundary the safety mechanism operated, it is necessary to simultaneously verify rejection results, actual reception counts, budget exhaustion, and execution termination.
Suggested Category: General IT/AI
Suggested Tags: AI Agent, Guardrail, Python, Tool Execution, Call Budget, Timeout, Cancellation
Leave a Reply