Welcome to Security Sessions, a regular feature focused on security-related issues, policies and technologies. During the last couple of years I've had the opportunity to be involved with several generating plants going through upgrades and expansions and work with the plant, corporate and vendor personnel responsible for handling these various efforts. One facility was upgrading their automation systems; another was upgrading and expanding their security systems; and the third was replacing their antiquated telephone system. The people involved had years of experience with the systems and technologies at their respective plants, but in each of these three cases the plant personnel had limited or no experience with the state-of-the-art in any of the three areas being updated (i.e., plant automation, telecommunications and security). This lack of awareness led to misconceptions and security presumptions that were dangerously wrong. - Tim.
William T. (Tim) Shaw
PhD, CISSP
Many years ago when the first of these plants was built, the telephone system they installed was a typical analog PBX system that provided voice-grade phone lines to all of the offices and various other locations on the site. It also supported the connection of FAX machines, dial in/out modems and could even support some dedicated, point-to-point circuits within the plant for interconnecting external systems and devices. Although the PBX had been updated a few times, it was still basically an analog telephone system. Their plan was to replace the old phone system with a modern VoIP (Voice over IP) telephone system. This in and of itself was not an issue. But they were thinking about the new phone system as if it were still an old analog PBX, presuming that it would continue to utilize dedicated wiring, when, in fact, the existing plant-wide LAN was actually going to be used for the new phone system.
In discussing the new digital VoIP phone system with the vendor – and with the plant telecom folks overseeing the upgrade – the picture became clearer. That is, the existing Ethernet switch network in the plant would now be expanded to reach every point where a phone was needed and/or currently existed, including some located in obscure locations along the periphery of plant grounds. And, as one might expect, that same LAN network supported the desktop PCs as well as business and various engineering servers, isolated from the automation systems only by an “internal” firewall. Also, the PBX server – which turned out to be a full-blown Microsoft Windows server – would sit on that same LAN and have a T3 circuit connection from the phone company to interface with the public telephone system.
But, in order to provide remote administration and management of the PBX and phone system, the vendor planned to separate part of that T3 bandwidth and have the phone company route it onto the Internet. This would allow the vendor to remotely manage and support the system (how convenient!) and establish a direct Internet connection pathway to all of the plant PCs and the plant-wide LAN, totally bypassing the carefully established “external” firewall that corporate IT had installed to isolate the plant’s networks. (Oops!)
The problem was that the vendor and the plant personnel were dealing in mutual-mystification. The vendor just figured that the plant people understood how a modern digital PBX worked and the plant people just thought it was a cool new phone system with lots of fancy features. Fortunately, the problem was discovered in time to make changes to the plan and preserve the necessary electronic security perimeter.
In another plant, where an automation upgrade was planned, many of the plant personnel carried two-way radios even though many of these personnel also carried cell phones – you know, the ones that act like a walkie-talkie – and used them as often as they used the twoway radios – or frequently in place of those radios. The automation vendor in the project was trying to offer the plant the ‘latest and greatest’ technology as well as expanding their “$cope” of work as far as possible. As part of that offer, the vendor suggested establishing a WiFi “umbrella” over the plant site to allow for the use of wireless devices/instrumentation and for their in-plant communications. This would have entailed placing wireless WiFi repeaters and access points around the plant, connected to the plant LAN at various points.
The vendor promised to upgrade plant personnel with cell phones that could make use of the WiFi, eventually replacing the twoway radios. The plant personnel didn’t immediately understand the security implications posed by cell phones that use the public cellular infrastructure versus those that seemed to work the same way, but actually make use of the private wired and wireless plantwide networks. In particular, they didn’t understand that the WiFi infrastructure would offer an attack portal into the plant networks, whereas the public cellular system did not. (Fortunately, the vendor did finally raise this point and offered to add various kinds of wireless security to plug the potential security breach – which I might remind you, they were implicitly going to create – in the electronic security perimeter!)
The third plant had a mix of security technologies. The plant’s security systems – which were added after the plant was built – were a combination of analog and digital technologies. A Closed Circuit Television (CCTV) system connected by dedicated coaxial cables allowed remote monitoring of various entrance points and critical plant areas. And analog video tape recorders maintained a compressed recording of the camera inputs. A key-card access control system had also been installed, along with card readers and door controllers scattered around the plant and connected by serial communication links to a central configuration and monitoring computer.
The security vendor was proposing to upgrade the plant to use “WebCam” technology so that the existing plant-wide LAN could be used rather than running new, separate cables. Moreover, that same LAN was going to be used to connect the replacement access control units and associated RFID tag readers and intrusion sensors. This, in turn, allowed all of the information to be routed to a plant security office, where PCs would be used to display and record the video and manage the access controls.
The server for the access control system was also going to be given a second Ethernet interface so that it could connect to the corporate WAN. Notably, this approach would allow their corporate HR department to remotely administer the access rights and personnel enrollment. Of course, this design also introduced unacceptable cyber vulnerabilities. For example, it created a ‘bridge’ between the plant LAN and the corporate WAN so that an attacker who penetrated the corporate network had an unprotected path onto the plant LAN that could potentially compromise or disable both the alarm/access control system and the video surveillance system. Again, this oversight was caught in time and corrected.
In all three of these examples it became obvious that the people involved, though perhaps well meaning and well intentioned, lacked a sufficiently current awareness of how the various technologies they planned to employ had converged and changed over time. Worse yet, they might have even been inadvertently aided by support from the corporate IT folks. In some cases, these IT personnel might not have had the requisite telecommunications expertise or experience with modern security systems technologies, but they would have been much more likely to have spotted the security issues.
The plant personnel – due to budget cuts and spending limits – were not being allowed to take continuing education courses or attend technical conferences; thus, their knowledge was outdated. This, more than anything else, contributed to the lack of awareness of the vulnerabilities they were poised to inadvertently introduce. Unfortunately, these are industrywide issues and are not limited to just the three organizations used here as examples. There is certainly a case to be made for how some organizations are being “penny wise and pound foolish” when they skimp on continuing education of their engineering staff – but that will be the subject matter for a future session. – Tim
About the Author
William T. “Tim” Shaw (PhD, CISSP) has been active in industrial automation for more than 30 years and is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems. Tim has contributed to several other books and is a prolific writer and presenter on a range of technical topics. He is currently a senior security consultant for Securi- Con, an information security solutions firm, based in Alexandria, Virginia. Tim has been directly involved in the development of several DCS and SCADA system products and regularly teaches courses for ISA (International Society of Automation) on various topics. Inquiries or comments about this column may be directed to Tim at Tim@electricenergyonline.com.
