On our search for a valuable solution we came across the XHELMER, a computer cluster that works with regular PC components inside an IKEA Helmer cabinet running OS X, which again was inspired by the original Helmer Cluster. Though the XHELMER site looks pretty interesting at first it hasn’t been updated in a while and never has gotten any further than the initial state. Eventually we had to came up with our own design and choice of components.
bunnyburg is a 6 node renderfarm, each node run by an Intel i7 quad core with 3.4 GHz and 16 GByte of ram, summing up to 81.6 GHz of renderpower or 0.5 TFLOPS.
Each node has a Cinebench score of 6.78, which is about 80% of the power of a current MacPro 8-core with 2.4 GHz but costing only a fragment of a MacPro. The entire bunnyburg renderfarm for about 3.750 EUR hardly tops the price of a current MacPro 8-core.
Another advantage is the power consumption, which should also be taken into consideration when building a renderfarm. bunnyburg consumes 240W when waiting for frames to be rendered and goes up to as high as 680W when running at full speed – which still is about the amount of a medium tuned games machine, but hosting much more power and memory.
bunnyburg is controlled via Pipeline, our inhouse rendermanagement software, which is also available outside brave rabbit. All nodes are equipped with Maya, VRay and After Effects. The administration is done via screen sharing if necessary and any shutdown or restart can be initiated through the manager software.
Pipeline also supports Shake and Nuke rendering and the free license is available with two clients. Feel free to contact us via eMail for information and licensing.
Before building the farm a couple of tasks had to be addressed:
1. The choice of the right components to use
2. How to fit everything into one small Helmer compartment
3. How to modify the Helmer cabinet.
Even though I have been on the PC side for a long time and never buying a machine off the shelf but rather building one from various computer parts, working on a Mac for some years now, the world of PC components can be very overwhelming, rendering one absolutely clueless.
Fortunately the selection can be somewhat narrowed down because the goal is not to build a regular PC but a good functioning render node, that is small and powerful while running OS X, leaving only the need to select the most basic components.
The possibility to have the mac platform running on generic PC hardware is a great advantage, especially in the light of how much longer the “Pro” line of Apple will remain existent, but also comes at the price that it’s not always too easy to make both worlds join. Unsupported or ‘exotic’ hardware can cause incompatibilities or instability of the system and patches are not existent for everything. The result is that hardware components need to be found to meet all the needed requirements while still being compatible with the OS X platform.
A great ressource page for getting started and finding the way through is tonymacx86. Especially the hardware recommendations can help alot to find the right parts.
For the motherboard there are actually not too many choices out there. Since the room in a single Helmer compartment is very limited, even though it’s about ATX compatible (the width the compartments fit in), the ITX form factor is the best choice to also allow for some air flow. Gigabyte builds excellent boards that are also well tested and supported when it comes down to CustoMacs.
The H61 chipset is maybe not the best choice for some people but since the CPUs aren’t overclocked it should do it’s duty.
The i7 2600k is currently a pretty solid, fast and not too expensive CPU. Even though this K-version allows for overclocking the main reason for choosing this CPU is the integrated HD3000 graphics that will do fine for administration needs and is supported since the introduction of OS X 10.7. No additional graphics card will be needed when using this CPU, saving extra costs and power consumption.
The CPU is cooled via the standard cooler that comes with the boxed version. The main choice for using this cooler is the lack of space. It’s not the quietest cooler but still performs fine.
The last concern was the power supply. Again the available space had to be considered. The regular ATX power supplies actually do fit but leave no room for the fan that is usually mounted on top. Some few power supplies still have the fan mounted in the back but these might not the best choice as far as fan noise is concerned. The solution is a small and less powerful (but still powerful enough) supply that takes up less space and guarantees the needed air flow even with a top mounted fan. For picking the right sized power supply I first tried to research the average power consumption for the chosen components. The result was 74 Watt during idle time and 158 Watt when working under full load. So, a 350 Watt power supply seemed to be a good combination of performance and size. The actual consumption comes down to 38 Watt during idle and 114 Watt for full load rendering.
Since hardware decision and room availability were complementing each other it became pretty clear how the components would have to be laid out for each node.
In order to guarantee enough air flow an opening had to be made in the rear bottom of each compartment underneath the fan and power supply. Also the ordering of fan and power supply alternates between the nodes to maximize the air flow using the bottom opening as efficient as possible.
For additional ventilation the label slot in front of the drawer is cut at the bottom and bend backwards to allow for some air to enter the node from the front. The opening is covered with an expanded aluminium grill from the inside to protect the components that are right behind.
The front panel features an illuminated power switch in red and a blue LED for the hard drive activity.
The rear panel of each compartment had to be modified the most to allow for the mounting of the power supply and the back fan. Cutting the pieces is very elaborate because of the pre-build mounting rims that leave very little room to handle the panels and also the cuts made leave not much left. A professional stamp would have made the building process a lot easier…
Before building the entire bunnyburg we decided to start with a template first, to see how the components would work, if everything would be compatible with the CustoMac requirements and how to make it all fit in one Helmer compartment.
It turned out that almost everything worked the way it was presumed. There were only two minor modifications that were made for the remaining five nodes.
The first version (bunny1) is equipped with an Intel i7 2600 since overclocking isn’t an issue and hence the K-version of the processor wasn’t needed. But it turned out that the integrated graphics is only the HD2000 which is not yet supported by neither 10.6.8 nor 10.7.2 thus taking away the ability to connect a monitor. This only affects when OS X is loaded up with all updates; for setting up the bios and installing the OS the internal graphics works fine. This ususally doesn’t matter for a render node which is administrated through the network anyway but for the other nodes a possible graphics support under OS X 10.7.2 through the Intel i7 2600k with the HD3000 seemed to be a reasonable investment. Especially because the price difference between a 2600 and 2600k is not that much.
The other modification regarded the hard drive. For the first version we went for a regular 3.5 inch hard drive but found that even though all parts fit nicely some room could be optimized and so we decided for a smaller 2.5 inch hard drive. Since each render node has only the operating system and the needed render software resident on the drive the capacity could be as low as possible.
A single bunnyburg node is hardly louder during idle than the average Mac Pro which I respect as the most quiet computer I ever owned. Even though no large fans could be utilized due to the space limitations and while using a regular boxed cooler that comes with the cpu, which is probably the most noisy component in the rig. Of course the fan speed is increased during rendering resulting in a higher noise level. Here more room and larger cooling components would of course help to quieten the build. Still, it’s within a bearable range, especially when taking into consideration that there are six of them all in all and they have to sum up in some sort of way. And there are servers that produce more noise with a single machine than bunnyburg with six nodes.
The space limitation is showing as well in the generated temperatures, even though they don’t reach a critical level. Having all nodes rendering with 100% cpu load each node shows a temperature level of about 75°C to 80°C tops. The Intel i7 has a regular heat limit of 100°C. And 75°C and maybe a little more can be usual when having the cpu working under full load.
Setting up the system to work with the regular Mac OS was a daunting task even though there are quite a few good tutorials online. These can be used as a good starting point but as with all installations that are not only plug and play the right set of information has to be chosen. There are actually a couple of installation tutorials that about use the same hardware but differ in some points during the setup of the CustoMac.
Once it’s clear what tools are needed and how they are used the setup isn’t so complicated after all.
An essential thing that needs to be defined is that regardless of which components are used and even if the current build is some sort of Mac Mini the
option in MultiBeast leads to OS X running the hardware in a low profile mode where only 50% of the cpu power is used but with a visual 100% readout, resulting in very slow renders. In order to gain the full performance of the farm the
option can be used without a problem.
The current setup is running without any tested audio capabilities as these aren’t necessary for the render farm nodes. The same goes with the graphics as we are currently running OS X 10.6.8 in the machines and neither the i7 2600 nor the i7 2600k are capable of displaying anything with this setup. As soon as we switch over to Lion I have run some tests and update the information provided here.