Menu
Search
Electrical Design of Sun's Datacenter In Santa Clara, California
Electrical Design of Sun's Datacenter In Santa Clara, California (photo credit: datacenterjournal.com)

Electrical Design Tour Of a Datacenter

This technical article provides a top-to-bottom overview of Sun Microsystems’s Santa Clara, California datacenter from electrical design aspect. Sun’s electrical design tour of Santa Clara datacenter is done through a series of single line diagrams.

These diagrams illustrate the various components and how they are connected without the detail of showing each wire for each phase – hence the term single line diagram (SLD).

Let’s start with the following topics //

  1. Electrical Yard
  2. Backup Generators
  3. Battery-Based UPS
  4. Pod Power Distribution
  5. Metering and Power Management

1. Electrical Yard

Sun’s Santa Clara project is one of the largest datacenter consolidation efforts in their recent history. It allowed them to construct an entirely new electrical service yard to supply power to new datacenters and buildings.

This electrical yard receives two 12 kV power feeds from Silicon Valley Power, each of which allows us to draw 4.5 MVA of continuous power. To ensure that Sun has the power to grow over time, they have negotiated with the local utility to provide up to 24 MW of additional power to the electrical yard through additional 4.5 MVA feeders.

A set of medium voltage switchgear allows the additional feeds to be incorporated into the electrical yard without disrupting existing service. The medium voltage switchgear feeds three similar sets of 480V switchgear through step-down transformers, one of which is illustrated in Figure 1.

Each of three sets of 480V switchgear can support a doubling of power by adding a second transformer and opening a tie breaker
Figure 1 – Each of three sets of 480V switchgear can support a doubling of power by adding a second transformer and opening a tie breaker (click to expand single line diagram)

The switchgear supplies 480V power to office buildings and datacenters on the Santa Clara campus. The switchgear includes breakers for existing spaces and for future expansion, with conduit runs already installed to the future spaces.

Sun accommodates expansion at this level of the distribution hierarchy by supporting each of the three sets of switchgear with a single transformer. Each set of switchgear is configured with two halves that are connected with a tie breaker that is currently closed.

When the time comes to double the power supply, a second transformer will be connected to a new utility power feed, and the tie breaker will be opened. Now each half of the switchgear has a 3 MW continuous operating capacity.

Visual Perspective Of Datacenter

A photo of the Santa Clara switch yard gives a visual perspective (Figure 2 below).

In the foreground we can see three rows of similar switchgear. From right to left is the 12kV switching upstream of the transformer. Next is the transformer, followed by the 480V switchgear. Note the empty space at the end of two of the rows to accommodate an additional transformer and switchgear. One already has been installed.

Behind the 480V switchgear is the medium voltage, 12kV switchgear that accommodates existing and future utility feeds. Also visible in the photo are two diesel backup generators.

A view of the Santa Clara electrical yard shows the 12 kV to 480V transformers and switchgear in the foreground
Figure 2 – A view of the Santa Clara electrical yard shows the 12 kV to 480V transformers and switchgear in the foreground (click to expand image)

Go back to Go back to Design Topics ↑


2. Backup Generators

Because the majority of Sun’s datacenters are for research and development efforts, the pods in them are designed to Tier 1–2 specifications. For 20 percent of the load, Sun deploys UPS to support more critical operations.

As operational risks increase, Sun deploys generators behind the UPS to provide continuous power.

Figure 3 shows a portion of the single line diagram showing backup generator configuration. Two backup generators are currently installed, with space, conduit, and breaker positions for a third. This provides 2 MVA of power today, with 3 MVA when the third generator is installed.

The generators provide power to a set of paralleling switchgear that combines the generator outputs. A 1000 kW load bank is installed to provide a full load when testing the generators. Regular 100 percent load testing of generators is important to make sure the generators are operating correctly and always ready to accept load.

Sun's Santa Clara electrical yard includes three generators that provide backup power to UPS
Figure 3 – Sun’s Santa Clara electrical yard includes three generators that provide backup power to UPS (click to expand single line diagram)

The paralleling switchgear currently feeds one 800 kVA UPS with provisions for future growth.

In the event of a failure, the feed is switched through automatic transfer switches (ATS). Two automatic transfer switches (ATS) switch the power feed for the UPS from utility power to generator power in the event of a power failure. A third ATS supplies power to keep the cooling system operating.

Go back to Go back to Design Topics ↑


3. Battery-Based UPS

Sun’s Santa Clara site is equipped to accommodate a fourth UPS to bring the total to 3.2 MVA. The site uses three 800 kVA UPS capable of providing up to 2.4 MVA of uninterruptible power. Sun chose APC Symmetra MW units for the Santa Clara site for their modularity and efficiency.

The APC UPS is unique in that it enables the UPS system to grow in 200 kW increments.

At the Santa Clara site, two of the three UPS are sized with 800 kW frames with only 400 kW of power modules installed, right sizing the UPS to Sun’s day one needs. As their loads grow, Sun will add additional power modules that grow the UPS to its full 800 kW capacity with only minimal construction in the UPS room to match battery capacity to UPS capacity.

An 800 kVA UPS supplies power to three transformers, each of which supplies 208/120V uninterruptible power to a set of busways
Figure 4 – An 800 kVA UPS supplies power to three transformers, each of which supplies 208/120V uninterruptible power to a set of busways (click to expand single line diagram)

The UPS illustrated in Figure 4 takes its input power from the automatic transfer switch illustrated in Figure 15. Its output feeds a distribution panel that connects to three transformers that step down the 480/277V power to the 208/120V power that feeds the busways.

One transformer is illustrated in Figure 4. It supplies power to a distribution panel that feeds a set of 225A and 400A busways inside the datacenter spaces.

Go back to Go back to Design Topics ↑


4. Pod Power Distribution

For the majority of pods that only require Tier 1 service, a single 480V feed from the switchgear supports multiple transformers that bring the voltage down to the 208/120V supplied to pods. The example shown in Figure 5 illustrates pairs of busway jumpered together.

Power is distributed to non-critical loads through transformers that feed 208/120V power to overhead busways
Figure 5 – Power is distributed to non-critical loads through transformers that feed 208/120V power to overhead busways (click to expand single line diagram)

Go back to Go back to Design Topics ↑


5. Metering and Power Management

One of the most important aspects of datacenter electrical design is an integrated metering and energy management system. These systems are critical because they provide the ability to evaluate how power is being used in the datacenter, power quality or disturbance events, how efficiently the datacenter is operating, and what impact any energy-saving measures have on overall efficiency.

As we know, the power usage effectiveness (PUE) is the ratio of total facility power to the IT equipment load.

In order to obtain this ratio, and to be able to observe energy consumption of the various datacenter subsystems, appropriate metering capabilities must be placed at strategic locations cascaded through the electrical system.

Metering Locations

In Sun’s Santa Clara datacenters, there is metering and power management in place at the following locations:

  • The utility switchgear
  • All major switchgear
  • All UPS, ATS and generators
  • Each overhead busway
  • Each rack is equipped with a rack-mounted power distribution unit (rPDU) that monitors overall power consumption and also provides remote on/off per-outlet control. These rPDUs can be monitored over the network, providing fine-grained data on IT equipment loads.

Go back to Go back to Design Topics ↑

Reference // Electrical Design Of Energy Efficient Datacenters – Michael Ryan, Brett Rucker, Dean Nelson, Petr Vlasaty, Ramesh KV, Serena DeVito, and Brian Day Sun Global Lab and Datacenter Design Services

About Author //

author-pic

Edvard Csanyi

Edvard - Electrical engineer, programmer and founder of EEP. Highly specialized for design of LV high power busbar trunking (<6300A) in power substations, buildings and industry fascilities. Designing of LV/MV switchgears.Professional in AutoCAD programming and web-design.Present on

2 Comments


  1. David L Lewis
    Jul 01, 2016

    Very good article, well presented.

Leave a Comment

Tell us what you're thinking... we care about your opinion!

Time limit is exhausted. Please reload CAPTCHA.

Get PDF