Wandering Adventure Party

@x0 hardware design already involves various forms of fuzzing, but the amount of state involved is so large that no amount of testing can be truly exhaustive

@vfrmedia @perpetuum_mobile if you have some free time this is a good deep dive: https://cseweb.ucsd.edu/classes/fa14/cse240A-a/pdf/04/Gonzalez_Processor_Microarchitecture_2010_Claypool.pdf

While it doesn't cover some of the most recent advancement it captures 90% of what you need to know.

If you have a lot of free time and want to dive deeper there's this: https://www.agner.org/optimize/microarchitecture.pdf

@mdione yes, it's very complex, but motherboard firmware has a mechanism to load the new microcode right as the CPU is bootstrapped. That is even before the CPU is capable of accessing DRAM. All the rest of the UEFI machinery runs after that. Note that this early bootstrap mechanisms usually involves a separate bootstrap CPU, usually an embedded microcontroller whose task is to get the main x86 core up and running.

@arclight on the other hand watch out for memory errors. Those can crop up much sooner than CPU problems due to circuit degradation: https://fosstodon.org/@gabrielesvelto/112407741329145666

@arclight timing degradation should not be visible outside of the highest-spec desktop CPUs which are really pushing the envelope even when they're new. Embedded systems and even mid-range desktop CPUs will never fail because of it. What might become visible is increased power consumption over time though.

@tehstu yes, absolutely. I've encountered several bugs in AMD CPUs, not many on ARM just yet, but our ARM user-base is very small compared to x86, so it's just less likely for us to stumble upon them. Plus we have some machinery that can detect some hardware bugs automatically but it doesn't work on ARM just yet.

@dubiousblur glad you liked it!

@KimSJ ah yes, very good point. It's been a while since my days in hardware land and I had forgotten about it.

@AndresFreundTec admittedly we get a lot more after a new microarchitecture launches, and then they go down as microcode updates get rolled out. If Microsoft hadn't started shipping microcode updates with their OS updates we'd be swamped.

@AndresFreundTec I've been in charge of Firefox stability for ten years now and some of my early work to detect hardware issues dates back then. In pre-2020 years we could get a 2-3 bugs per year, usually across different CPUs. Now we get dozens, it's really on another level.

@gsuberland thanks, I was playing a bit fast and loose with the terminology. As I was writing these toots I reminded myself that entire books have been written just to model transistor behavior and propagation delay, and my very crude wording would probably give their authors a heart attack.

Bonus end-of-thread post: when you encounter these bugs try to cut the hardware designers some slack. They work on increasingly complex stuff, with increasingly pressing deadlines and under upper management who rarely understands what they're doing. Put the blame for these bugs where it's due: on executives that haven't allocated enough time, people and resources to make a quality product.

All in all modern CPUs are beasts of tremendous complexity and bugs have become inevitable. I wish the industry would be spending more resources addressing them, improving design and testing before CPUs ship to users, but alas most of the tech sector seems more keen on playing with unreliable statistical toys rather than ensuring that the hardware users pay good money for works correctly. 31/31

And remember there's a lot of variables involved: timing broadly depends on transistor sizing and wire resistance. Higher voltages improve transistor performance but increase power dissipation and thus temperature. Temperature increases resistance which decreases propagation speed in wires. It's a delicate dance to keep a dynamic equilibrium of optimal power consumption, adequate performance and reliability. 30/31

When CPUs ship their most performance critical circuits are supposed to come with a certain timing slack that will compensate for this effect. Over time this timing slack gets smaller. If a CPU is already operating near the edge, aging might cut this slack all the way down to zero, causing the core to fail consistently. 29/31

You might also remember that Raptor Lake CPU problems get worse over time. That's because circuits degrade, and applying the wrong voltage can cause them to degrade faster. Circuit degradation is a research field of its own, but its effects are broadly the same: resistance in wires go up, capacity of trench capacitors go down, etc… and the combined effect of these changes is that circuits get slower and need more voltage to operate at the same frequency. 28/31

I can't be sure that this is exactly what's happening on Raptor Lake CPUs, it's just a theory. But a modern CPU core has millions upon millions of these types of circuits, and a timing issue in any of them can lead to these kinds of problems. And that's without saying that voltage delivery across a core is an exquisitely analog problem, with voltage fluctuations that might be caused by all sorts of events: instructions being executed, temperature, etc... 27/31

This is a circuit with two sets of 8 wires that go into it, plus one wire to select which inputs will go to the output, and a single set of 8 wires going out. Depending on the value of the select signal you'll get one or the other set of inputs. Guess what happens if the select signal arrives too late, for example right after the end of the clock cycle? You get the wrong set of bits in the output. 26/31

In Raptor Lake's case a very common pattern that me and others have noticed is that sometimes the wrong 8-bit value is delivered. This happens when reading 8-bit registers such as AH or AL, which are just slices of larger integer registers, and don't have dedicated physical storage. The operation that pulls out the higher or lower 8 bits of the last 16 bits of a regular register is usually done via a multiplexer or MUX. 25/31

The speed at which signals propagate in circuits is proportional to how much voltage is being applied. In older CPUs this voltage was fixed, but in modern ones it changes thousands of times per second to save power. Providing just as little voltage needed for a certain clock frequency can dramatically reduce power consumption, but providing too little voltage may cause a signal to arrive late, or the wrong signal to reach the pipeline register, causing in turn a cascade of failures. 24/31