We replaced Twilio Studio with a visual IVR — here's why the simulator matters

Two designs exist for proving an IVR before it takes a real call, and they buy different things. Twilio Studio's documented test path is a real call from a listed test number that runs the latest draft revision, so the test and production share one execution path. A pure-data simulator walks the stored tree without telephony, which is instant and free and structurally not the runtime. Neither is strictly better. What matters is knowing exactly which class of bug your test cannot see, and this post names four of ours.

What "testing an IVR" means in each of the two designs

An IVR is a tree of nodes and transitions. Testing it means answering one of two very different questions, and the tools split along that line.

The first question is "does the routing logic go where I think it goes?" Press 2 at the main menu, land on billing, press 1 there, reach the transfer. That is a graph-traversal question, answerable from the stored data alone, without a phone.

The second is "does a real carrier, executing the document my server produces, behave the way that traversal predicts?" That is a different question, and the data cannot answer it, because between the tree and the caller sits a rendering step, a network, and another company's parser.

Most builders answer one of these well. Studio answers the second by making the test a real call. A simulator answers the first by removing the call. The failure mode of each is the question it did not answer.

Twilio Studio's documented test path is a real call to a draft

Worth stating precisely because the folklore around it is wrong in both directions.

Twilio describes Studio as a low-code visual builder for creating and managing communication workflows, assembled by dragging widgets onto a canvas. Its user guide documents testing this way: you can test a draft of a Flow without publishing changes, and when you use one of the numbers entered in the Test Users section to contact the Twilio phone number associated with that Flow, the Flow runs through the latest draft rather than the latest published version. When you are finished testing you click Publish to make the changes accessible to everyone.

Read what that architecture guarantees. The test call and the production call go through the same execution machinery — the only difference is which revision is selected. There is no second implementation to drift, because there is no second implementation. That is a real property and it is the strongest thing about the design.

Read what it costs. Every test consumes a real call on a real number from a real handset, so the loop is minutes rather than seconds. You cannot test forty input permutations before lunch. You cannot assert on a branch in a unit test. And a test user has to be a phone number somebody holds, which makes automated regression testing of the routing graph awkward at exactly the point where the tree gets large enough to need it.

A pure-data simulator is instant, free, and structurally not the runtime

The other design separates the two questions and answers the cheap one cheaply.

In Autocloz, POST /voice/ivr/trees/{tree_id}/simulate takes a list of input digits and walks the tree from its entry node, following DTMF transitions and falling back to a no_match or timeout action when no digit matches. It returns a trace — one entry per node visited, carrying the node id, its kind, its label, the action chosen with its trigger and action types, and the next node — plus a terminated_reason drawn from a small vocabulary: hangup_node, end_call, no_match_no_fallback, input_exhausted, missing_node, or max_steps when a 50-step loop guard fires.

That is genuinely useful. It runs in milliseconds, needs no phone number, costs nothing per run, is scoped to the caller's own workspace, and can be asserted on in a test. For the first question — does the graph go where I think — it is strictly better than a phone call.

It is also, unavoidably, a second implementation of the traversal. The live runtime lives in a different module, issues its own queries, and renders TeXML rather than returning a trace. Two implementations of one concept is a drift class, and the honest thing to do is not to claim they are the same code but to enumerate where they differ and defend the gaps with tests.

Four places a walker and an emitter can disagree, from our own code

These are real divergences in Autocloz's current implementation, not hypotheticals. Each one is a bug class a green simulation cannot rule out.

1. The status filter. The live runtime loads a tree only when status = 'active' and archived_at IS NULL. The simulator requires only that the tree is not archived. So a tree parked in draft or paused simulates perfectly and answers a real call with "Service unavailable." followed by a hangup. This is the divergence with the worst blast radius and it gets its own section below.

2. The dispatch axis is not the same everywhere. The simulator branches on the chosen action's action_type. The runtime's node renderer branches on the node's kind — a hangup kind emits an optional spoken line and , a transfer kind resolves a DID and emits , message and voicemail kinds speak or play and hang up, and anything else is rendered as a menu with a . The runtime's digit-advance leg then branches on action_type again. A node whose kind and whose actions imply different things will walk one way in the simulator and another on the wire.

3. Transfer targets are resolved in one and ignored in the other. The runtime looks up the DID for a transfer action, scoped to the tree's own organisation so a stale action row cannot route a call to another tenant's number, and emits when the lookup returns nothing. The simulator never consults the DID table at all — it records the action and continues the walk. A transfer node pointing at a deleted or cross-tenant number therefore simulates as a successful transfer and hangs up on a live caller.

4. The loop guard exists in only one of them. The simulator stops after 50 steps and reports max_steps. The runtime cannot count, because each carrier callback is an independent HTTP request that carries the position in its URL and nothing else. A cyclic tree loops until the carrier gives up. The simulator will tell you the cycle exists — which is exactly the sort of thing it is good for — but only if you happen to feed it enough digits to enter the cycle.

There is a fifth worth mentioning because it is the least obvious. If a tree has no entry node, the simulator refuses with a 409 and an actionable message telling you to set one. The runtime falls back to the first node returned by an unordered query. So the one tree state the simulator declines to model is the state where production behaviour is arbitrary.

The status filter is the divergence with the worst blast radius

Pulling one of the four out, because it is the one that reaches customers.

The status filter is a good design. Being able to take a tree out of service by flipping a field, and have the carrier receive a polite spoken refusal rather than an unpredictable partial walk, means an operator can edit a live menu mid-shift without coordinating a change to the carrier's configuration. Half-deployed trees are a real category of outage and this removes it.

The problem is only that the simulator does not model it, which means a perfectly green simulation run says nothing at all about whether that tree will answer. An operator who builds a tree, simulates it happily, points the carrier at it and never flips it to active has done everything the tool showed them and gets a dead number.

The fix is not to relax the runtime filter — that would reintroduce the half-deploy outage. It is that the simulator should report the status it found, and the interface should make an inactive tree visible at the moment you simulate it. Until it does, the operating procedure carries the gap: check status before you test, and treat a green trace on a draft tree as unproven.

The proof is an assertion on the emitted XML, not on the trace

This is the discipline that makes the whole arrangement safe, and it generalises well beyond IVRs.

A simulator trace is an assertion about a model. The thing the carrier executes is a document. So the test that actually protects a live call is one that hits the runtime URL and asserts on the rendered output — that the response is application/xml, that a menu node emits a whose action attribute is the absolute URL of the next leg, that a paused tree emits the service-unavailable response, that a transfer with an unresolvable DID emits rather than an empty .

Where those assertions land in reporting is worth wiring up too: a menu that hangs up on the first keypress shows in call reporting as a wave of very short inbound calls long before anyone complains.

That is testable end to end without carrier credentials, which is the point. The TeXML the runtime produces is deterministic given the tree, so a test can post digits to the advance leg and assert the exact XML that comes back. A live carrier call only adds the network round trip, and that leg is the carrier's to attest.

The general rule the four divergences above are an instance of: a structural assertion is a supplement, never the proof. A test that checks the simulator's trace is checking that the walker agrees with itself. Only an assertion on the artefact the other system consumes can catch a divergence between two implementations, and the same logic applies to any UI change verified in a browser rather than by reading the source. That principle is worth applying to the sequence layer too, where the IVR menu builder sits alongside the rest of the call configuration.

Escaping is not optional, and here is the test that pins it

A small mechanical section, because this is the cheapest possible production incident.

Every dynamic value rendered into TeXML gets XML-escaped — ampersand, angle brackets, quote — and that includes the operator's spoken prompt. Skipping it does not create a security hole worth worrying about; nobody is injecting hostile XML into their own greeting. It creates a document the carrier's parser rejects, during a live call, with a failure that surfaces nowhere near the field that caused it.

The guard is a test that feeds a node the prompt A & B "d" and asserts the rendered document contains A & B <c> rather than the raw string. That test costs nothing and pins a whole class of "the menu stopped working after someone edited the greeting" reports.

The audio itself is registered once and reused across trees, which is why it lives in a shared audio library rather than being attached to a node — a greeting re-recorded in one place should not need editing in nine.

The same discipline applies to the audio a menu plays. The carrier fetches a URL with no authorisation header, so it cannot use the ordinary authenticated file route. Autocloz mints a short-lived signed URL whose token binds both the organisation and the specific audio id, purpose-salted so it cannot be confused with a tracking or reply token, with a one-hour lifetime because a live call fetches it immediately. A token minted for one file cannot fetch another, a tampered token is refused with a 403, and the row is loaded using the org id inside the token rather than one supplied by the request.

What TeXML actually is, and what the carrier does with it

Worth a short section because the mental model explains most of the failure modes above.

Telnyx describes TeXML as an XML-based data structure used to control calls, containing sequential instructions executed from top to bottom, and its translator interprets the verb and noun vocabulary from existing XML call-instruction files written for another provider. The flow is: the carrier receives a call, requests your configured URL, and executes whatever verbs come back.

The important consequence is that there is no session. A element carries an action attribute — an absolute URL — and a numDigits and a timeout. When the caller presses a digit, the carrier posts those digits to that URL as form data, and your server has to reconstruct where it was purely from the path. Autocloz encodes it as /voice/ivr/runtime/telnyx/{tree_id}/{node_id}, so the tree and the current node are in the URL and the server holds nothing between turns.

That is also why both the entry callback and every mid-call URL have to be derived from the same public base. If the entry leg advertises one host prefix and the gather leg advertises another, a reverse proxy that routes only one of them produces a menu that greets the caller and then dies on the first keypress — and the failure appears only mid-call. Autocloz derives both from a single configured public base and refuses to place an outbound IVR call at all when that base is unset, returning an explicit error saying the answer webhook would resolve to localhost and the call would connect with no menu, rather than dialling and failing silently. The digit signalling underneath all of this is standardised in RFC 4733, "RTP Payload for DTMF Digits, Telephony Tones, and Telephony Signals", published December 2006, which obsoletes the older RFC 2833.

Autocloz's free plan covers 5 users and 10 mailboxes with the call channel included, so an inbound menu and a simulated walk of it are both available before anyone pays — start free and point a carrier at a tree you have already traced.

What a simulator cannot tell you, and what Autocloz does not do

The honest boundaries, because a post about a testing tool that oversells the tool has defeated itself.

A simulator cannot tell you what a caller hears. It has no audio, no text-to-speech, no timing, and no sense of whether a nine-second greeting exhausts a caller's patience before the options arrive. It cannot tell you whether the timeout of five seconds is long enough for a real person holding a phone in a noisy room, and that is frequently the actual defect in a menu that tests perfectly.

It cannot tell you anything about the carrier leg: whether digits are being signalled in a format the carrier passes through, whether the audio file is reachable and in a codec the carrier accepts, or whether a transfer to an external number completes. Those need a real call.

And Autocloz specifically. The simulator and the live runtime are two implementations, with the four divergences named above — that is the accurate description, and any claim that a builder's preview and its runtime are literally the same code path is worth checking rather than believing. Voicemail recording, complex transfers and SIP REFER bridging are explicitly out of scope for this runtime, because they need carrier-specific testing a development environment cannot provide. Outbound IVR runs on Telnyx only: it requires both a REST outbound-dial API and a programmable answer webhook, and among the supported carriers only Telnyx has both — FreJun's IVR builder is dashboard-configured and inbound-only, and DIDLogic is SIP with no carrier-side webhook — so a campaign configured for outbound IVR on another provider is refused with an explicit reason rather than quietly downgraded. The provider trade-offs that follow from that are set out in what cloud telephony actually is, the dial modes each carrier can run are compared in auto dialler against power and predictive dialling, and the carrier connection itself is documented on the Telnyx integration page.