A retry can create a second job after an HTTP 500
SabaTechnical writing sample by Saba. Experiment and source checks: November 4, 2026. This is a controlled local example using synthetic data, with no customer incident or commercial outcome claimed.
An HTTP 500 does not establish that no job was created. In this experiment, the server saved a job, returned an error, and then created a second job when the client repeated the request. Reusing an operation key prevented the second effect. The response statuses alone could not distinguish those two outcomes: both sequences were 500, then 201.
That distinction matters for an agent deciding what to do after a tool reports an error. Repeating the command may look like recovery while changing the underlying system again. A useful check needs to observe the intended effect as well as the response.
Build a failure whose timing is known
I used Python 3.14.7 and its standard library to create a small job service. Each experiment started a fresh HTTP server on 127.0.0.1, using a port selected by the operating system. The client disabled proxy handling and made real HTTP requests to that server. There was no external service, account or paid API involved.
The endpoint accepted POST /jobs with one task value. A new job received an integer ID and was appended to an in-memory list. The first newly created job in each experiment deliberately received a 500 response after it had been stored. Later new jobs received 201 responses. A separate GET /jobs returned the list for inspection.
The error response was delivered successfully. This fixture does not reproduce a dropped connection, a client timeout, a database transaction failure or an unexplained production outage. Its purpose is to fix the order of events: store the effect, send an error, then observe what another attempt does.
The complete companion source appears at the end of this page. Save the two named code blocks in one directory, then run:
python -B -m unittest -v test_demo python -B demo.py
The first command runs the checks. The second prints the requests' statuses, response bodies and rows observed through GET. Keeping those observations separate makes the result easier to assess than a single success flag.
Compare effects, not just responses
A separate run of the experiment produced these results. Each case starts with a fresh store.
- One request without a key: response statuses 500; jobs observed afterward: 1.
- Two identical requests without keys: response statuses 500, 201; jobs observed afterward: 2.
- Two identical requests with the same key: response statuses 500, 201; jobs observed afterward: 1.
- Two identical requests with different keys: response statuses 500, 201; jobs observed afterward: 2.
- Same key with a changed task value: response statuses 500, 409; jobs observed afterward: 1.
The first case establishes the premise. The response reported an error, while the separate state query found job 1. The operation had already changed this server's state.
In the unkeyed retry, the second request created job 2. Its task string matched job 1, but the server had no identifier connecting the two requests to one intended operation. The 201 response described a successful second creation. It did not repair or remove the first job.
The stable-key retry also returned 201 on the second attempt, but its body named job 1 and GET showed one row. An observer watching only the statuses would miss the difference between duplicate creation and reuse of an existing result.
Give the operation an identity
The keyed version used an Idempotency-Key header. Under a lock, the server checked whether it had already seen that key. For a new key, it stored the task value and the job result alongside the effect. A later request using the same key and task value received the existing job instead of creating another.
The lock serialized access to this one process's shared state. It did not make the state durable. The stored result was the nominal successful job result, even when the first outward response was the deliberately injected 500. That is the contract chosen for this example.
Generating a fresh key for every attempt defeated that contract. The different-key control produced two jobs, just as the unkeyed control did. The useful identity belongs to the intended operation across its attempts. A header that changes on each retry cannot provide that connection.
The changed-task control tested the opposite mistake. Reusing operation-1 for a different task returned 409 and left the original row intact. This prevents the example from silently treating a changed request as a successful replay. Its comparison covers the single task string sent by this client; it is not a general request-fingerprinting implementation.
Identical task text can also represent two legitimate jobs. Treating the text itself as the operation's identity would collapse those intentions. The caller needs to distinguish another attempt at one job from an intentional request for a second job.
Read the endpoint's actual retry contract
HTTP idempotence concerns the intended server effect of repeated identical requests. RFC 9110 cautions against automatically retrying non-idempotent requests unless the client knows their semantics permit it or can determine that the original request was not applied. A generic error response alone does not establish that the original request was not applied. RFC 9110, section 9.2.2.
An idempotency key also does not determine which response a retry will receive. Stripe's API v1 documentation says it saves the first executed request's status and body for a key, including a 500, and replays that response. Its documentation describes different recovery behavior for API v2. Those are provider-specific contracts. Stripe's idempotent-request documentation.
This local example instead remembers the created job and returns it on a matching retry. A client written around that behavior could misinterpret a provider that keeps returning the original error. The comparison here is documentation-based; the experiment made no Stripe requests or payment transactions.
Before implementing a retry, identify what the service remembers, how long it remembers it, what happens when parameters change, and how to inspect the original operation. A key alone does not answer those questions. Nor does a successful retry response establish how many effects occurred.
Check the fixture as well as the lesson
The initial five integration checks covered the outcomes listed above using real HTTP calls. Review also found a cleanup gap: client setup happened after the server thread started but before the cleanup block. If setup raised an exception, the server could remain running.
A regression test injected an opener-construction failure while capturing a real server and thread. It failed before the correction because the thread was still alive. After setup moved inside the cleanup protection, the test confirmed that the thread stopped and the port could be rebound. The final separate run passed all six checks.
That regression uses fault injection for the setup failure. It does not replace the five HTTP behavior tests with mocked successes. The code attempts shutdown and socket closure in cleanup, joins the serving thread with a two-second timeout, and reports a thread that remains alive. This verifies the exercised paths, rather than proving every possible startup or operating-system failure is handled.
The service remains an in-memory, single-process teaching fixture. It does not test restart recovery, durable deduplication, multiple workers, key expiry or complete input validation. Python's documentation also explicitly excludes http.server from production recommendations. Python 3.14 HTTP-server documentation.
For an agent performing a write, record the intended operation's identifier and the state that would demonstrate completion before running it. After an error, inspect that state through the service's supported mechanism. If a retry is appropriate, follow that endpoint's contract and preserve the operation identity. Keep the response and the observed effect in the record, especially when they disagree.
Companion source: demo.py
Save the following block as demo.py.
import json
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
def run_case(operation_keys, tasks=None):
operation_keys = list(operation_keys)
if tasks is None:
tasks = ["demo"] * len(operation_keys)
else:
tasks = list(tasks)
if len(operation_keys) != len(tasks):
raise ValueError("operation_keys and tasks must have equal lengths")
rows = []
key_cache = {}
state_lock = threading.Lock()
first_effect = True
class Handler(BaseHTTPRequestHandler):
def send_json(self, status, value):
encoded = json.dumps(value).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Content-Length", str(len(encoded)))
self.end_headers()
self.wfile.write(encoded)
def do_POST(self):
nonlocal first_effect
if self.path != "/jobs":
self.send_json(404, {"error": "not found"})
return
length = int(self.headers.get("Content-Length", "0"))
payload = json.loads(self.rfile.read(length).decode("utf-8"))
task = payload["task"]
key = self.headers.get("Idempotency-Key")
with state_lock:
if key is not None and key in key_cache:
cached_task, job = key_cache[key]
if task != cached_task:
status, body = 409, {"error": "key reused with different payload"}
else:
status, body = 201, job
else:
job = {"id": len(rows) + 1, "task": task}
rows.append(job)
if key is not None:
key_cache[key] = (task, job)
if first_effect:
first_effect = False
status, body = 500, {"error": "simulated acknowledgement failure"}
else:
status, body = 201, job
self.send_json(status, body)
def do_GET(self):
if self.path != "/jobs":
self.send_json(404, {"error": "not found"})
return
with state_lock:
snapshot = list(rows)
self.send_json(200, snapshot)
def log_message(self, format, *args):
return
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server.daemon_threads = True
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
base_url = "http://127.0.0.1:{}".format(server.server_address[1])
statuses = []
bodies = []
for key, task in zip(operation_keys, tasks):
payload = json.dumps({"task": str(task)}).encode("utf-8")
headers = {"Content-Type": "application/json"}
if key is not None:
headers["Idempotency-Key"] = str(key)
request = urllib.request.Request(base_url + "/jobs", data=payload, headers=headers, method="POST")
try:
with opener.open(request, timeout=2) as response:
status = response.status
body = response.read()
except urllib.error.HTTPError as error:
status = error.code
try:
body = error.read()
finally:
error.close()
statuses.append(status)
bodies.append(json.loads(body.decode("utf-8")))
request = urllib.request.Request(base_url + "/jobs", method="GET")
with opener.open(request, timeout=2) as response:
observed_rows = json.loads(response.read().decode("utf-8"))
return {"statuses": statuses, "bodies": bodies, "rows": observed_rows}
finally:
try:
server.shutdown()
finally:
try:
server.server_close()
finally:
thread.join(timeout=2)
if thread.is_alive():
raise RuntimeError("server thread did not stop within 2 seconds")
def demo_cases():
return {
"first_failure": run_case([None]),
"unkeyed_retry": run_case([None, None]),
"same_key_retry": run_case(["operation-1", "operation-1"]),
"new_key_retry": run_case(["operation-1", "operation-2"]),
"changed_payload": run_case(["operation-1", "operation-1"], ["demo", "changed"]),
}
if __name__ == "__main__":
print(json.dumps(demo_cases(), indent=2, sort_keys=True))
Companion source: test_demo.py
Save the following block as test_demo.py.
import socket
import threading
import unittest
from unittest.mock import patch
from demo import run_case
import demo
class RetryStateHTTPTests(unittest.TestCase):
def test_first_unkeyed_effect_returns_error_but_is_stored(self):
result = run_case([None])
self.assertEqual(result["statuses"], [500])
self.assertEqual(result["rows"], [{"id": 1, "task": "demo"}])
def test_each_unkeyed_retry_creates_another_effect(self):
result = run_case([None, None])
self.assertEqual(result["statuses"], [500, 201])
self.assertEqual([row["id"] for row in result["rows"]], [1, 2])
def test_same_key_retry_returns_existing_effect(self):
result = run_case(["operation-1", "operation-1"])
self.assertEqual(result["statuses"], [500, 201])
self.assertEqual(result["rows"], [{"id": 1, "task": "demo"}])
self.assertEqual(result["bodies"][1], {"id": 1, "task": "demo"})
def test_new_key_creates_another_effect(self):
result = run_case(["operation-1", "operation-2"])
self.assertEqual(result["statuses"], [500, 201])
self.assertEqual([row["id"] for row in result["rows"]], [1, 2])
def test_same_key_with_changed_payload_conflicts_without_effect(self):
result = run_case(["operation-1", "operation-1"], ["demo", "changed"])
self.assertEqual(result["statuses"], [500, 409])
self.assertEqual(result["rows"], [{"id": 1, "task": "demo"}])
def test_setup_failure_closes_server_and_stops_serve_thread(self):
real_server_type = demo.ThreadingHTTPServer
real_thread_type = threading.Thread
captured_servers = []
captured_threads = []
def capture_server(*args, **kwargs):
server = real_server_type(*args, **kwargs)
captured_servers.append(server)
return server
def capture_thread(*args, **kwargs):
thread = real_thread_type(*args, **kwargs)
captured_threads.append(thread)
return thread
try:
with patch.object(demo, "ThreadingHTTPServer", side_effect=capture_server), \
patch.object(demo.threading, "Thread", side_effect=capture_thread), \
patch.object(demo.urllib.request, "build_opener", side_effect=RuntimeError("opener setup failed")):
with self.assertRaisesRegex(RuntimeError, "opener setup failed"):
run_case(["operation-1"])
self.assertEqual(len(captured_servers), 1)
self.assertEqual(len(captured_threads), 1)
server = captured_servers[0]
thread = captured_threads[0]
self.assertFalse(thread.is_alive(), "serve thread is still alive")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:
probe.bind(server.server_address)
finally:
for server in captured_servers:
try:
if captured_threads and captured_threads[0].is_alive():
server.shutdown()
server.server_close()
finally:
if captured_threads and captured_threads[0].is_alive():
captured_threads[0].join(timeout=2)
if __name__ == "__main__":
unittest.main()