Opens in a new tab
Explainer

How GANs can vary a synthetic telemetry test

In brief

Give a timeout detector varied queue data while keeping the overlapping retries it must find under your control.

5 min read

Is this the right starting point?
Before you read
Basic machine-learning training, gradients and telemetry concepts; simple timeline arithmetic supports the constructed timeout example.
What you will take away
Explain GAN training and its limits, and design synthetic variation around a controlled timeout-overlap detection fixture.
Sources
Woodcut printmaking scene compares repeated waveform prints with a varied reference sheet under a magnifying glass.
On this page4 sections

Suppose you are testing a detector for cascading timeouts. A generated queue graph gives the test some variety, but there is a harder question behind it. Does the data show a retry reaching the downstream service while the original call is still running? A plausible graph can make the test look realistic without supplying the event you need it to detect.

A generative adversarial network, or GAN, can give a controlled test new surroundings. Use the generator for variation and construct the failure separately, so you can tell what the test actually covers.

How the generator learns what looks familiar

For this queue test, the generator turns random noise into a window of queue-depth values. In an illustrative training batch, suppose observed queues drain gradually after a burst, but a generated window drops straight to zero. The discriminator sees both kinds of window and learns to tell them apart. Training checks its judgments against each window's known origin. That abrupt drop is a possible clue. Only the discriminator's internal numerical settings, or parameters, change during this step.

Then hold those parameters fixed while updating the generator. Suppose the discriminator would find a gentler drain more convincing. Working backward through both networks shows how small changes to the generator's parameters would affect that judgment. These rates of change provide the gradient that guides the generator's update. Training changes its settings, not the saved window's numbers. Its next run can produce different values. The two kinds of update alternate.

GAN training alternates two learning steps
Step one updates the discriminator using observed and generated examples while the generator is fixed. Step two fixes the discriminator and uses its gradient to update the generator. Solid arrows carry examples; the dashed arrow carries a learning signal.
Update the discriminator on observed and generated examples. Then hold it fixed while its gradient guides a generator update.

Goodfellow and colleagues describe this process in Generative Adversarial Nets. The ideal is for the generator to reproduce the full mix of patterns in the training data, leaving the discriminator unable to tell them apart. That equilibrium is a theoretical result. In practice, updates can oscillate and the gradient reaching the generator can weaken. Our generator might produce convincing bursts while losing gradual buildups and slow drains present in its training data. That loss of variety is mode collapse. Near-chance discriminator accuracy could reflect better generation or a discriminator that learned little. Inspect the patterns actually covered; that score cannot establish service health.

Give the generator data in the form you need

For this proposed test, represent each minute of queue depth as sixty values in order, one per second. The generator produces an array of the same shape. If training used normalized values, convert back to the original units before checking the result. Queue depth must remain a valid count, and changes must make sense alongside the arrivals and completions you include. A plausible count can still fall faster than the work leaves.

That is why this test needs numbers behind the graph. Training an image generator on queue graphs does not, by itself, ensure its counts agree with arrivals and completions. Generating log text would add another problem: choosing discrete tokens, such as words or word-pieces, makes it difficult to pass the usual discriminator gradient back to the generator. SeqGAN addresses this problem for token sequences.

To request a burst rather than a slow drain, conditional generation gives both networks a workload label. With suitable labeled queue data, the generator receives random noise plus “burst.” The discriminator sees observed burst windows and generated windows, each accompanied by that label. It learns to distinguish their origins within that condition, giving the generator feedback toward the requested kind of window. After training, supply “burst” with fresh noise to request another such window. A slow drain might look perfectly ordinary and still miss the burst you asked for. Inspect the values before using that window.

Write down the calls the detector has to find

Use one simulation clock for this hypothetical failure. A calls B, and B calls C. When A times out, B leaves its outstanding call to C running. That is the behavior that allows the retries to stack up.

Elapsed timeConstructed eventWhat remains active
0 msA calls B. B starts C1 at C.C1 remains outstanding.
500 msA times out and retries. B starts C2.C1 and C2 overlap.
1,000 msA times out again. The next retry starts C3.All three calls overlap.
1,200 msC1 completes after A's first attempt timed out.The later calls are still outstanding.

Keep a shared identity for the logical request, separate IDs for the attempts, their start and end times and the links that identify retries. C1 and C2 overlap for 700 milliseconds. C3 joins the overlap at 1,000 milliseconds. A matching histogram cannot show which calls belonged together. Add a comparison case where the original downstream call ends before the retry begins, so this particular overlap is absent.

Now use an inspected generated window as the background. Try the same constructed calls during a burst, a gradual buildup or a slow drain. Preserve the IDs and timing links and recompute the expected label after overlaying the events. If a transformation moves C1's end before C2 starts, you have a different case. Run each background without the injected overlap too, to see whether the detector would respond to a burst that lacks the targeted relationship. This is a test design, not a reported model or detector run.

Save how you made each sample: the training population, requested condition, time scale and constraints you checked. Mark the cases as synthetic and keep them out of incident statistics. Quiet weekday training data cannot show that your test covers holiday peaks. Before sharing the results, check permission and leakage. GAN-Leaks studies membership inference, in which an attacker tries to tell whether a record appeared in training. A synthetic label does not establish that the sample is safe to share.

Keep control of the failure you understand

A simulator can generate a queue that grows because arrivals exceed a fixed completion rate. It gives you direct control of the cause. A GAN might add useful variation in the background, if its outputs survive the checks and the extra work is justified. For anomaly scoring, compare that added complexity with a variational autoencoder, a conventional model or a fixed rule.

If the generated windows make a useful pipeline or scale test, keep that result. To claim that the detector catches the timeout sequence, look for the expected difference between the paired cases. The meaningful evidence is that it finds the still-running call when the retry starts, and stays quiet on the same background when that overlap is absent.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

Keep reading

Where to go next