Content
Evolving Toward the “Green” Data Center
by Jaroslav Rajić, CCNP/CCSI, Instructor at NIL Data Communications Ltd
“Green” Data Center Evolution Drivers
Evolving toward more energy-efficient consumption has been an increasing business trend in the last few years. By itself, reducing energy consumption provides benefits: it translates directly to lower operating expenses and less environmental pollution.
“Reducing [one’s] carbon footprint” is fashionable terminology based on scientific reality. When fossil fuels are burned to convert them to energy, carbon is released into the atmosphere as a byproduct; by using less energy, the Data Center decreases its carbon “footprint.”
Building a new “green” Data Center is a matter of choosing the right equipment, with power consumption in mind. When the next generation of any equipment is engineered, it tends to be designed with a more optimal per-port power consumption — a rule that applies to everything from line cards and modules to routers and security appliances.
One way of building a “green” Data Center is to choose more modern standalone equipment; another is to use a smaller number of more advanced pieces of equipment. To reach the same level of service as multiple boxes can offer, virtualization techniques are used.
Virtualization is applied on network equipment and on the server/services side. The goal is to achieve better utilization of hardware, maintaining the power consumption of only one set of equipment — or two, when high availability is required. (Consider 20 passengers riding a bus together, rather than driving 20 cars, to get to the same destination.)
Figure 1 shows some 85% reduction in power requirements, plus savings on rack space, reduced cabling, etc. The example shows the amount of power required for a segregated Data Center with several firewalling and application load-balancing devices, compared to functionally the same setup using virtualization on the Cisco Firewall Services Module (FWSM) and on the Cisco Application Control Engine (ACE).
Figure 1
Power consumption of a set of devices compared against a smaller number of virtualized devices [source: Cisco]
Figure 2 lists general requirements of a modern Data Center. The main goals are agility, resiliency and cost-effectiveness. Highly utilized, pooled, and standardized resources are most efficient.
Figure 2
Properties of a modern Data Center [source: Cisco]
Data Center Requirements Overview
The main requirements of a Data Center are related to providing adequate power, cooling capacities, and supporting services (security, and fire-hazard handling). The majority of power consumption goes for powering the equipment. However, every two or three watts of power delivery requires one watt for cooling (depending on the performance and quality of the cooling system). Physical security and lighting carry about 10% of the total power consumption burden.
In Figure 3, electricity bears the biggest stake in the Data Center’s total cost of ownership (TCO), and Figure 4 shows where most of that energy is consumed.
Figure 3
Data Center TCO distribution [source: APC]
Figure 4
Data Center energy consumption [source: APC]
Server Virtualization to Decrease Energy Consumption
A typical workload percentage of a particular network server’s uptime is 5–15% of its total capacity. However, even though the server is idle 85% of the time, it still consumes energy. Combining several services into one physical server solves this problem, as effective utilization of the hardware is increased with regard to data processing.
Virtualization tools support running several standalone servers virtually in a single physical server host. The VMware ESX hypervisor is a current example of such a solution (although this concept is not new – it has been around since the days of mainframes).
From a hardware point of view, blade servers are a better solution in terms of economics and energy consumption; they offer lower total power consumption versus 1 rack-unit [RU] servers, and lower total heat dissipation compared to 1 RU servers. However, if you’re using processor-intensive applications on diskless servers, your power requirements may vary a lot over time.
In terms of power, the needs of a full 42 RU server rack have risen to 15 kW, but the majority of Data Centers can accommodate only approximately 4 kW per rack. Figure 5 shows typical power consumption for Data Centers, with indications of typical power consumption for both 1 RU servers and blade servers.
Figure 5
Typical power consumption of a Data Center [source: Cisco]
Server virtualization is a whole topic on its own, and it is beyond the scope of this article.
Network Equipment Virtualization
Virtualization of equipment is the next step to increasing equipment usage efficiency, and thereby to decreasing the number of idle processor cycles. Switches have been supporting virtualization of LAN networks into Virtual Local Area Networks (VLANs) for a long time. VLANs have numerous uses now and are a widely adopted virtualization technique.
Network devices such as Cisco Adaptive Security Appliance (ASA) and FWSM, Cisco ACE, etc. support virtualization by allowing multiple so-called “contexts” running on the same appliance/module, thus reducing the number of modules needed for a particular functionality, and decreasing overall energy consumption. This approach is similar to server virtualization.
In the Cisco Catalyst 6500 family, equipment virtualization by means of the Virtual Switching System (VSS) is focused more on high availability and path selection than on equipment efficiency.
In storage networking, the MDS family of products supports virtualization by employing Virtual Storage Area Networks (VSANs) and features such as N_Port ID Virtualization (NPIV) and N_Port Virtualization (NPV).
The Cisco Nexus 7000 family supports Virtual Device Context (VDC) functionality, which enables you to divide a single Nexus switch into several virtual contexts, acting as up to four logically separated (and virtualized) switches. The VDC can help you in some cases to tailor your network to suit your requirements (for example, regulatory requirements, etc.) and still maintain heat dissipation and power consumption limited to a single physical device.
The Unified Fabric may also be regarded as a “green” efficiency measure, in which two types of cabling – for networking and storage – are replaced with a single cable of one type. Another option is equipment that serves as a consolidation point for both Ethernet and Fibre Channel traffic, such as the Cisco Nexus 5000 line, requiring only a single box instead of a Catalyst switch and an MDS switch, and thus reducing the number of devices that consume power and generate heat.
On the server side, Unified Fabric means that fewer NICs need to be powered. Another benefit that is not directly related to power consumption is that a much smaller number of cables interconnect the network and the servers.
DID YOU KNOW?
On a Cisco Catalyst 6500 switch, you can get chassis temperature information by using the show environment temperature and show environment cooling commands.
To minimize heat generation in the proximity of the access switches with many clients acquiring power over Ethernet (PoE), use inline power injectors in the patch panels instead of using the switch itself as the power source. This approach limits stress on the power supplies.
The left graphic in Figure 6 displays the per-port power consumption for the new 10 Gb Ethernet line cards, and the right graphic illustrates the per-port power consumption efficiency of the new Gigabit Ethernet line cards (all WS-X67XX series).
DID YOU KNOW?
Switch line cards have evolved to use much less power per port than in previous-generation line cards. The Catalyst 6500 WS-X6716-10G line card uses about half the power per port of an older WS-X6704-10G line card!
Figure 6
Typical per-port power consumption efficiency of various Catalyst 6500 10-Gigabit and Gigabit Ethernet line cards [source: Cisco]
Efficiency Metrics and Tools
To measure how efficient a Data Center is in terms of power and cooling, some metrics had to be invented to make comparisons and goal-tracking possible. Two of them are used most widely: Data Center Infrastructure Efficiency (DCIE), and Power Usage Effectiveness (PUE).
Some special-purpose metrics are composites, using up to 11 different DC parameters such as fire safety, etc., but are used only in special cases when precise metering is required for accurate comparisons.
Data Center Infrastructure Efficiency (DCIE)
Data Center Infrastructure Efficiency is a metric used to determine the energy efficiency of a Data Center. The metric is expressed as a percentage and is calculated by dividing IT equipment power by total power available at the facility.
Power Usage Effectiveness is the reciprocal value of DCIE. A typical PUE value is around 2.5, which means that for every 2.5 watts of power, one watt is consumed to power IT equipment. Using power-optimized equipment and “green” Data Center best practices, the target value of PUE should be at around 1.6.
PUE is defined as follows:
PUE = Total Facility Power / IT Equipment Power
and its reciprocal, DCIE, is defined as follows:
DCIE = 1 / PUE = IT Equipment Power / Total Facility Power × 100%
Cisco Power Calculator
The Cisco Power Calculator (available at http://tools.cisco.com/cpc/launch.jsp) is a web-based utility that you can use to determine the power and cooling needs of Cisco equipment. This tool allows you to configure the desired equipment setup and indicate the power consumption. You can use it for existing equipment or for equipment that you intend to purchase.
Data Center Room Layouts
There are two basic ways to implement a Data Center room.
Starting from the beginning and with correct budgeting, raised flooring offers a neat and clean Data Center, with cabling and cooling conduits to be implemented under the floor.
A central cooling unit can be used to cool the air, which is then distributed evenly through the room from under the floor.
An issue requiring special care is the mass load on the floor, because loaded racks can be very heavy: a fully loaded 60x60 cm rack can weigh up to 1 ton, and floors are regularly built to withstand some 500 kg per sq m. Figure 7 shows an example of a Data Center built with a raised floor.
Figure 7
Typical raised-floor Data Center room layout [source: Cisco]
Another approach is to position the racks on the floor and guide the cables overhead, implementing cooling based on the cooling requirements of the equipment. This approach is more practical to deploy and offers benefits of its own: since the cabling is overhead, adding more cables (if needed) is very practical.
Cooling cannot be implemented from under the floor, so a cooling unit is positioned somewhere in the Data Center room. You can identify the “hot spots” in your DC, placing additional cooling in the vicinity of server blade enclosures, etc. Uniform temperature is not obtained within the whole DC room, but isn’t really necessary. Other racks can be laid out in such a way that hot air is collected from a common space in the room. Here, positioning depends on airflow through the equipment in the racks: either front-to-back or side-to-side. Such a Data Center is displayed in Figure 8.
Figure 8
Typical “flat” Data Center room layout [source: Cisco]
Airflow Modeling and Raised Floors
The cooling of legacy Data Centers was designed to cool the DC room to a uniform temperature. However, server racks — and especially blade server housings — provide very concentrated hot spots, so now the focus is to direct cooling to this equipment and not to focus as much on overall room temperature. Some energy savings can be derived from this approach.
Extremely hot spots in the room require airflow modeling, where airflow is directed toward those hot areas in the DC. The easiest way to achieve a directed airflow effect is by having raised floors, and another approach is to add modular cooling units to the parts of the Data Center where additional cooling is needed. Proper placement of additional cooling is determined either by point measurements or by mathematical modeling.
When using underfloor cooling ducts, additional cold air can be supplied by using perforated tiles. Additional efficiency for perforated tiles can be achieved by insulating the cabling holes very well, in order not to lose air there and minimize energy loss.
Water-Cooled Racks
Using water as a medium to transfer heat is a rather new approach, considering the cooling systems that use air, and has the advantage that liquids provide a better medium to transfer heat. (The automotive industry has long taken advantage of this fact.) The main disadvantages are additional complexity, leading to increased costs.
Savings in floor space can be achieved by using water cooling, as more servers can be loaded into a single rack, which is cooled more efficiently, and less heat dissipates into the room. Another option is to have completely sealed racks that cool only the inside of the rack, with no impact to the room.
Sealed racks can be used in smaller system rooms (for example, smaller Data Centers at remote locations) and provide for quick add-on of cooled rack space, without upgrading the original cooling system. However, you should treat this option only as an interim or temporary solution when upgrading your existing Data Center for better energy efficiency.
Figure 9
An example of a liquid-cooled rack [source: Coolherm]
Getting Consulting Support
Cisco offers several initiatives designed to create a more efficient and “green” Data Center. The high-level Cisco Efficiency Assurance Program (EAP) includes services for planning and learning, as well as reference designs.
Through the help of the Cisco Efficiency Assessment Service, you can upgrade your Data Center to be energy-efficient and yet powerful. These services are carried jointly by Cisco Advanced Services and selected Cisco partners.
Summary
Several technologies, metrics, theory foundations, mathematical models and good practices are available to enable you to make your existing Data Center more energy-efficient, or to build a state-of-the-art “green” Data Center that is future-proof and will actually save you money because of the modern designs used.
No two Data Centers are equal in requirements and possible optimizations. You may come across hardware that cannot be replaced, or you may need to carry that legacy inefficiency for some time, until the next upgrade or business case change.
This article was meant as an overview of topics that are becoming more and more important as energy prices have risen worldwide in the last two or three years.
Glossary
ACE – Application Control Engine
ASA – Advanced Security Appliance
DCIE – Data Center Infrastructure Efficiency
EAP – Cisco Efficiency Assurance Program
FWSM – Firewall Services Module
NPIV – N_Port ID Virtualization
NPV – N_Port Virtualization
PUE – Power Usage Effectiveness
TCO – Total Cost of Ownership
VDC – Virtual Device Context
VLAN – Virtual Local Area Network
VSAN – Virtual Storage Area Network