Building WOPR: A 7x4090 AI Server

• Mainboard / CPU / RAM
• Risers
• Power and Cooling
• Frame
• Bill of Materials
• FAQ

Multiple Power Supplies

This part appears simple, but it's worth discussing.

With around 3.5KW worth of power dissipation, there's no easy way around using multiple power supplies with something like this. When I built my machine I just went for it with four 1000W power supplies, and it works. But I've read that this isn't ideal, and that horrible things could happen when using multiple power supplies with one system in a naive way. So I'll share what I did, what some others have done, but once again consider yourself warned.

Note: George Hotz has pointed out that the 12VDC pins on the PCIe slot can / should be disconnected to possibly mitigate potential issues. Again, this is uncharted territory and without knowing a lot about specific power supplies it's hard to know what the risk may or may not be.

What I Used (You Can Do Better)

The first thing we have to look at is the power requirements of the system. You can get the general idea by looking at the Thermal Design Power (TDP) of the individual components. It roughly breaks down as follows:

• Seven 4090s: 7 * 450W = 3150W
• AMD Epyc 7763: 280W

There are other things drawing power in this system: maybe 10W for the SSD, probably similar for the retimer board, a handful of watts for the NIC, etc. But those are basically rounding errors in comparison to the CPU and GPUs.

What we're looking at is a total of say, 3.5KW for the whole system. But since we're running multiple power supplies you have to divvy it up.

I chose to go with four 1000W Corsair power supplies, with one power supply per pair of GPUs for the first 6 GPUs, then the 4th power supply running the 7th GPU as well as the motherboard, CPU, etc. That gives me ample headroom on each power supply (2 x 450W = 900W per GPU supply) and they're all the same which can make things like having a spare replacement easier.

The specific model of power supply I chose is the Corsair RM1000x Shift, mainly because that's what I could get 4 of at MicroCenter combined with Corsair's good reputation as a power supply manufacturer. It's a good power supply, but only has an 80 Gold efficiency rating. You can find something nicer, or consider the server power supply setup below.

This power supply is a little odd in that it has the cabling on the side instead of the front, but this worked well for cable management.

While I chose consumer power supplies for the sake of expediency, there's another option to discuss and that's server power supplies.

Server Power Supplies

In a lot of ways, server power supplies are ideal for something like this. They're efficient, they are designed for high duty cycles, and so on. They're built for the job.

But server power supplies have one big drawback which is cabling. Most are hot-swap, which means that instead of having a bunch of cabling from the supply they have a funny edge connector that's meant to slide into its mate inside a server. It looks almost like a PCIe card connector. To use one, you have to get a compatible breakout board so we can connect our gear.

Thankfully there's a kit which is ideal for this, which comes from Parallel Miner. Yeah, the crypto guys strike again. But it comes recommended from my buddy @drivelinekyle who is part of a little Twitter GC we have for this stuff. He's had good luck with both their customer service and product, and I have no reason to doubt him.

The kit is the Remote Managed Lite-On 4100/4400w 110-240v Mining Power Supply Kit Gold Rated 92%. In addition to providing 4x1100W server power supplies it also includes the breakout board as well as a remote management board that lets you turn the PSUs on and off over Ethernet. And best of all it's $240.

If I had it to do again, that's the route I'd take for sure. Just saying.

Wall Power

Your situation is sure to be different than mine in where your machine will be physically, but I can offer some thoughts on how to get power to something like this.

Call an Electrician

First off, you should recognize if you don't already that 3.5KW is not an insignificant amount of power - it's in the same territory as a residential clothes dryer. The ones with the huge plugs - maybe you get where I'm going with this. If you aren't qualified and have knowledge of electrical codes, you should absolutely consult an electrician. Considering how much you've spent on this equipment (not to mention your home or office where this is going), the cost of getting a professional involved isn't that much. Make the call.

You will find people talking about running machines like this off extension cords from different circuits in their house, etc. Please don't do that. I'm less offended by the idea of hijacking an existing dryer outlet, but one still has to consider breaker and wiring capacities, GFCI requirements, and other factors.

Supply Voltage

Some thoughts on voltages. Depending on where you live, you may only be looking at 230VAC (Europe, etc) but where I'm at as well as some other places normal office or residential wall power is 115VAC (+/- transmission loss which is why you find these voltages listed anywhere from 110-120VAC and 220-240VAC. It's all the same thing within those ranges.)

If you find yourself with the option, I would highly recommend thinking about running everything off 230VAC. Most switching computer power supplies are actually multi-voltage so they can be sold around the world, and should handle either but be sure to check yours especially if you run server power supplies. By running off 230VAC vs 115VAC, you reduce by half the amount of current flowing through the wiring for a given power draw. That means the wiring can be smaller (less $) and everything in general will be more efficient.

To give some imprecise numbers, at 115VAC and 4000W you will pull about 40A total (+/- efficiency) and at 230V you'll have around 20A. You should check the manual or nameplate on your power supplies to get the specific max rated current.

230VAC does carry more risk to you as a human so when selecting plugs and receptacles be sure to find ones which are rated for 230VAC and the correct current, and use them properly.

My setup

For me, this was going in my basement far from the main panel. For convenience, I had the electrician install a 100A subpanel right by the machines. Then, they dropped two 30A 230VAC circuits, each with 4 plugs sharing that 30A circuit. That lets me plug in a server per circuit, and everything is well within the rating of said circuit leaving me some headroom if I want more later on.

While I was at it, I had them drop some convenience 115VAC outlets for my other gear, which was a long time coming as I was getting close to maxing out the common outlet circuit in the basement.

I forgot to ask while they were in the panel, but I should have had them install a surge suppressor as well. You can get these really cheap and they cover the whole panel.

Cooling

Firstly, you should recognize that 3.5KW of power consumption = 3.5KW of power dissipation. As a point of reference, a 1000 square feet of residential space in a 4-season climate requires about 10 KW of heating capacity. Yeah. At full tilt, this thing on its own could handle 1/3 of a typical apartment's heating requirement. If you put this in your bedroom, in the summer, it's going to get warm in there.

As mentioned previously, I chose to put mine in the basement. It stays pretty cool down there, and while I notice it's a bit warmer than it used to be I haven't had to do anything special. I've not run it during the summer yet but I think the basement walls are enough of a heatsink that it won't be much different. Also, at least some of the house's AC-cooled air is falling down to the basement so that should help as well.

If it comes down to it, I've physically located my machines by an HVAC unit so I can pull a duct off for local cooling if required. That would necessarily mean taking away from other parts of the house so it may not be ideal. But I really don't think I'll have to take that measure with the basement staying as cool as it does.

If you don't have a basement handy, you'll have to get creative with air conditioning. In general terms, your AC will pull about as much power as the unit does. So an unconditioned space with a little portable AC or window unit may not cut it if you're running full-tilt 24/7. Things to think about.