Ubiquiti Switches and Routers in Extreme Heat and Humidity
Abstract
Shaw IT are regularly deploying outdoor networking cabinets in Thailand, often custom built for client sites. We use a variety of equipment from a variety of manufacturers. We experiment with consumer grade gear as well as enterprise and carrier grade equipment.
The purpose of this exercise is to examine the viability of Ubiquiti switches and routers in extreme ambient temperature environments, including high humidity and high airborne particulate matter. Daytime ambient temperatures are usually in the high 30s and often reach the mid 40s. Case temperatures can reach much higher values.
Adding to the difficulties facing our deployments are smoke, 100 percent humidity, insect and water ingress. During the dry season the north of Thailand suffers extreme smoke pollution from forest fires. For several months of the year the PM2.5 AQI index is in excess of 200 and for several weeks hits 800 – 1000 (off the scale, the worst readings in the world). Further, rainfall can be so heavy in Thailand that in microburst conditions water can be sucked into case fans. Cases must be designed with this in mind.
We have found that while in 'normal' rackmount conditions, Ubiquiti networking equipment in general is quite reliable. However, when we push things beyond their design environment things become very unreliable. This is not a surprise of course but moving to equipment specially ruggedised for extreme environments can get prohibitively expensive for many of our clients so in this article we describe our prototype outdoor cabinet designed for the above extremes, using less costly Ubiquiti switches and fibre optics.
Our Prototype HEPA Case
By far, the most significant risk of failure in Ubiquiti switches we find to be internal switched mode power supplies rather than the main board. Where the switch is powered by DC, we have found that the first thing to go is often the reverse polarity or over-voltage protection diode on the DC input. In fact we found one diode that had become so hot during it's failure that the solder had melted and it simply fell off, (which actually allowed the switch to recover and continue operating). We can often repair these issues in our lab and redeploy, but the site visits and downtime events are undesirable of course.
Our test site is a hotel in Thailand. In order to reduce overall cost and maintenance load, we constructed a prototype cabinet for an ES-12F switch with a mixture of SFP modules including 1G BiDi and 10G types connecting to 10 downstream buildings. The steel cabinet is in direct sunlight with the case exterior becoming far too hot to touch. 10G SFP modules run particularly hot. While this is expected, they add significant heat to an already very hot case, easily exceeding 70 degrees C.
We chose to build a positive pressure forced air system rather than negative pressure. We find that as with our other cabinets utilising positive pressure, sufficient airflow eliminates ant infestation problems. We are not sure why but postulate that the high airflow coming out through the cracks interferes with the ant's chemoreceptors and discourages them from entering. In any case, in all deployments we have completely eliminated infestation problems with high positive pressure airflow.
Protecting the Fans
We have found that fans clog very quickly due to the smoke, the dust of the dry season, and the airborne particles of the wet season (pollen, seeds etc). There are also several weeks per year when migrating termites shed their wings, which cause enormous damage when sucked into our fans. The only solution is HEPA rated filtration. This creates additional maintenance but we have found that if the filters are sufficiently large, changes can be as infrequent as three year intervals, although we recommend that filters are renewed annually. The advantages of going fully HEPA rated far outweigh the disadvantages and vastly reduce on-site visits and cabinet failures.
Plastic pipes are used as downward facing air intakes in front of the HEPA filters. This prevents water ingress even in the most extreme rainfall and microburst conditions. During the 2 year test period the case survived at least 2 serious and sustained microbursts with high winds, with no water ingress at all.
We use a minimum of two centrifugal fans for redundancy. We always choose dual ball bearing or maglev fans for their long life and resilience in high temperatures. These fans also have a high static pressure able to pull large amounts of air through filters. In our experimental prototype case we placed numerous temperature sensors around the cabinet and equipment to both monitor temperatures throughout the year and to shut down the entire system in the case of over-temperature.
We set a hard limit of 45 degrees for the case temperature, at which point a relay cuts off power completely (except for the fans which keep running) and power is automatically restored when the case temperature drops to 40 degrees. We chose 45 degrees not only to protect the networking equipment but also to protect the lead acid batteries employed for power backup.
Upon deployment this automatic cut-off and recovery was tested by artificially blocking the air intakes.
Three 40mm holes were cut into the ES-12F, one directly above the CPU, the other two covering the 12 SFP ports. We considered running these three fans at low speed as we were concerned about static build up but found that adequate grounding eliminated this risk. As noise is not an issue here, they were set to full speed. Note that this switch is powered with 24 volts DC, but the 220 volt power cable is left in place for additional ground protection (only the CPC is connected).
Monitoring these SFPs revealed a huge drop in temperature reported by the SFPs themselves. In all conditions, the SFPs, even the 10G modules, ran very cool at just a few degrees above ambient. This represents a ~50% drop in operating temperature, broadly speaking.
The prototype was operated for 2 years and then it's condition examined. Air circulation was sufficient to keep the case at an even temperature, a few degrees above ambient. There were no failures. No evidence of dust, smoke deposits or insects on any surfaces or fan blades was evident.
The prototype has now been refined and developed into a standard model and we deploy it at numerous customer sites. Ubiquiti switches and routers are suitable for deployment in these harsh environments, provided the protections detailed above are adhered to. With high flow of filtered air, failure rates are no different to 'normal' deployments.