Simulators are great training tools. It sure beats flying 777s around for your annual pilot recert. Gaming technology has become so good along with many other technologies, that operators of process plants and machinery should be well trained to respond appropriately to any emergency.
Georgia Institute of Technology sent this information about an advancement in simulation for operator training. Good stuff.
A simulator that comes complete with a virtual explosion could help the operators of chemical processing plants – and other industrial facilities – learn to detect attacks by hackers bent on causing mayhem. The simulator will also help students and researchers understand better the security issues of industrial control systems.
This flow chart shows data flows within a simulated chemical processing facility.
Facilities such as electric power networks, manufacturing operations and water purification plants are among the potential targets for malicious actors because they use programmable logic controllers (PLCs) to open and close valves, redirect electricity flows and manage large pieces of machinery. Efforts are underway to secure these facilities, and helping operators become more skilled at detecting potential attacks is a key part of improving security.
Screen captures show a simulated explosion in a chemical processing plant precipitated by a cyberattack on the system.
“The goal is to give operators, researchers and students experience with attacking systems, detecting attacks and also seeing the consequences of manipulating the physical processes in these systems,” said Raheem Beyah, the Motorola Foundation Professor in the School of Electrical and Computer Engineering at the Georgia Institute of Technology. “This system allows operators to learn what kinds of things will happen. Our goal is to make sure the good guys get this experience so they can respond appropriately.”
Details of the simulator were presented August 8 at Black Hat USA 2018, and August 13 at the 2018 USENIX Workshop on Advances in Security Education. The simulator was developed in part by Atlanta security startup company Fortiphyd Logic, and supported by the Georgia Research Alliance.
The simulated chemical processing plant, known as the Graphical Realism Framework for Industrial Control Simulations (GRFICS), allows users to play the roles of both attackers and defenders – with separate views provided. The attackers might take control of valves in the plant to build up pressure in a reaction vessel to cause an explosion. The defenders have to watch for signs of attack and make sure security systems remain operational.
Screen capture shows a chemical processing plant in which critical parameters are rising due to false process data and control commands injected by an attacker.
Of great concern is the “man-in-the-middle” attack in which a bad actor breaks into the facility’s control system – and also takes control of the sensors and instruments that provide feedback to the operators. By gaining control of sensors and valve position indicators, the attacker could send false readings that would reassure the operators – while the damage proceeded.
“The pressure and reactant levels could be made to seem normal to the operators, while the pressure is building toward a dangerous point,” Beyah said. Though the readings may appear normal, however, a knowledgeable operator might still detect clues that the system has been attacked. “The more the operators know the process, the harder it will be to fool them,” he said.
The GRFICS system was built using an existing chemical processing plant simulator, as well as a 3D video gaming engine running on Linux virtual machines. At its heart is the software that runs PLCs, which can be changed out to represent different types of controllers appropriate to a range of facilities. The human-machine interface can also be altered as needed to show a realistic operator control panel monitoring reaction parameters and valve controller positions.
“This is a complete virtual network, so you can set up your own entry detection rules and play on the defensive side to see whether or not your defenses are detecting the attacks,” said David Formby, a Georgia Tech postdoctoral researcher who has launched Fortiphyd Logic with Beyah to develop industrial control security products. “We provide access to simulated physical systems that allow students and operators to repeatedly study different parameters and scenarios.”
GRFICS is currently available as an open source, free download for use by classes or individuals. It runs on a laptop, but because of heavy use of graphics, requires considerable processing power and memory. An online version is planned, and future versions will simulate the electric power grid, water and wastewater treatment facilities, manufacturing facilities and other users of PLCs.
Formby hopes GRFICS will expand the number of people who have experience with the security of industrial control systems.
“We want to open this space up to more people,” he said. “It’s very difficult now to find people who have the right experience. We haven’t seen many attacks on these systems yet, but that’s not because they are secure. The barrier for people who want to work in the cyber-physical security space is high right now, and we want to lower that.”
Beyah and Formby have been working for several years to increase awareness of the vulnerabilities inherent in industrial control systems. While the community still has more to do, Beyah is encouraged.
“Several years ago, we talked to a lot of process control engineers as part of the NSF’s I-Corps program,” he said. “It was clear that for many of these folks then, security was not a major concern. But we’ve seen changes, and lots of people are now taking system security seriously.”
Looking for the source of innovation in manufacturing technology. Not only am I planning for direction in 2018, I’m in conversations about where lies the excitement.
OK, so it’s been two months I’ve been digesting some thoughts. In my meager defense, November and December were very busy and hectic months for me. Still lots going on in January as I gear up for the year.
Last November, I quoted Seth Godin:
Like Mary Shelley
When she wrote Frankenstein, it changed everything. A different style of writing. A different kind of writer. And the use of technology in ways that no one expected and that left a mark.
Henry Ford did that. One car and one process after another, for decades. Companies wanted to be the Ford of _____. Progress makes more progress easier. Momentum builds. But Ford couldn’t make the streak last. The momentum gets easier, but the risks feel bigger too.
Google was like that. Changing the way we used mail and documents and the internet itself. Companies wanted to be the Google of _____. And Apple was like that, twice with personal computers, then with the phone. And, as often happens with public companies, they both got greedy.
Tesla is still like that. They’re the new Ford. Using technology in a conceptual, relentless, and profound fashion to remake industries and expectations, again and again. Take a breakthrough, add a posture, apply it again and again. PS Audio is like that in stereos, and perhaps you could be like that… The Mary Shelley of ____.
So I asked on Twitter “Who will be the Mary Shelley of automation?
I’m sitting in a soccer referee certification clinic when I glance at the phone. Twitter notifications are piling up.
Andy Robinson (@Archestranaut) got fired up and started this tweet storm:
Gary… why do you have to get me fired up on a chilly November morning! I’m not sure we have any.. at least at any scale. And the more I’ve pondered this more the more I consider the role or culpability of the customer. Buyers of automation at any scale tend to be 1/
incredibly conservative. If they are ok with technology that isn’t much more than a minor evolution of the existing then we aren’t going to get anywhere. Recently I devoured Clayton Christensen’s The Innovator’s Dilemma. I keep trying to figure out how a small player 2/
with disruptive tech can move our industry. There are pockets and potential but ultimately if there isn’t enough uptake by customers willing to take a risk then we don’t move forward. Considering all this I “think” I have figured out one potential causal factor. 3/
If you look at where the fastest innovation is happening it’s in software. Is the majority of the innovation coming from vendors or asset owners. it’s asset owners. Amazon, Netflix, AirBnB, etc. are all doing amazing things and taking risks writing new code for their systems4/
Having been an asset owner and vendor I can tell you for a fact I was way more willing to take risks when I was the owner. As a vendor I want to deliver a solution to spec with minimal risk. Fundamentally product companies are doing the same thing. Just good enough with 5/
minimum risk to supply chain, warranty repairs, reliable field operations etc. Even platforms like Kubernetes that appear unaffiliated were developed by asset owners like Google, taking risks and pushing the boundaries. The Exxon work with open automation “has” this 6/
potential but I don’t know if the willpower up and down the chain and left and right with partners is going to be there. It takes incredible willpower to take risks and accept that there will be blow back and consequences in the form of loss of political capital and failure. 7/
So maybe it all boils down to the fact that until we as an industry find a place where failure is acceptable and even celebrated on a small scale we will continue to innovate at a speed somewhere between typewriters and vacuum cleaners. 8/
is it any wonder we have such a hard time attracting young talent? Pay is good and challenges to solve real problems are there. But looking 20 years out we are still doing same things, just a new operating system, faster Ethernet, and new style of button bar on the HMI /endrant
He asks some good questions and provides some interesting insights.
I’ve had positions with companies at different points of the supply chain. He makes sense with the observation that the asset owners may be the most innovative. My time in product development with consumer goods manufacturers taught me such lessons as:
- Fear of keeping ahead of the competition
- Relentless concentration on the customer
- Not just cost, but best value of components going into the product
- Explaining what we were doing in simple, yet provocative terms
Today? I’m seeing some product companies acquiring talent with new ideas. Some are bringing innovative outlooks to companies who find it very hard to take a risk for all the reasons Andy brings up. The gamble is whether the big company can actually bring out the product—and then integrate it with existing products to bring something really innovative to market. They of course have the funds to market the ideas from the small groups.
Next step, do the innovative people from the small company just get integrated into the bureaucracy? Often there is the one great idea. It gets integrated and then that’s the end. The innovators wait out their contract and then go out and innovate again. I’ve seen it play out many times in my career as observer.
Often the other source of big company innovation bubbles up from customers. An engineer is trying to solve a problem. Needs something new from a supplier. Goes to the supplier and asks for an innovation.
I’d look for innovation from asset owners, universities, small groups of innovative engineers and business thinkers. They live in the world of innovating to stay ahead of the competition or just the world of ideas.
I’m reading Walter Isaacson’s biography “Leonardo” right after his one on Einstein. He offers insights on what to personality to look for if you want to develop an innovative culture in your workforce. Wrote about that recently here.
Walter Issacson has done deep research and written biographies of several men you could call geniuses. Benjamin Franklin, Leonardo Da Vinci (next on my reading list), Steve Jobs, Albert Einstein. I just finished the Einstein book.
Reflecting on his career while speaking at a conference I attended, Issacson said that they all shared certain characteristics–they were rebellious, they didn’t quite fit in with their contemporaries, and they could bring in ideas from numerous sources. Think of this in terms of building a great workforce.
I was probably 10 or 11 when I first read a biography of Franklin. Even then I was impressed by his wide-ranging curiosity. He seemed to learn something about everything. Yet, he grew up poor and didn’t have the perks of wealth.
The Einstein book was enjoyable, if long. My wife said she had some trouble getting through it. I can believe it. All the stories about his wives and family troubles were hard to get through. But the detailed discussion about the developments in physics–ah, suburb. OK, so maybe she liked the family stuff and I preferred the physics.
By the way, it’s not true that he failed arithmetic as a boy. But his genius was not in math. He had friends who helped out on the math side of the theories.
Einstein didn’t accept all the common knowledge about physics of the day. As he pondered the influential experiments and thoughts of the late 19th century, he performed thought experiments. That is, he used his imagination.
In fact, one saying attributed to him concerns the importance of imagination over rote learning.
He said later in life that one doesn’t attend college to learn facts. You go to college to learn to think.
I was no doubt influenced by that statement many years ago when I formulated my description of an educated person (note that it says nothing about degrees)–you learn how to learn, you learn how to think clearly, you learn how to express yourself.
In fact, while I respect the tenacity of those who have advanced degrees, I got that out of my system early. The university shut down the program I was in. After getting accepted at a couple of other universities, I looked at the curriculum and decided to study what I wanted. I’ve always viewed a degree as a certificate that entitles the bearer entry into a club.
In that respect, I’ve always sided with the greater Marx philosopher–Groucho. “I don’t want to join any club that would want people like me as a member.”
Workforce tip: You’re going to look for minimum capability, of course. Don’t hire a programmer who thinks Java is something you drink. However, look for people who are curious, who challenge things as they are, who can balance individual work with team work. Cultivate a workplace where ideas (not personalities) form the foundation of lively and challenging discussions.
Rockwell has had a strong training program for many years. I took my first week-long class in 1991 or 1992. Altogether I have taken about six classes—controls, PLCs, drives, motor control centers, software. I know how intense the training can be.
Last week I posted a podcast of thoughts from Rockwell Automation’s annual series of events held the week prior to Thanksgiving. Now I’m in Spain at yet another conference and trying to get caught up on posts before I start a flurry of posts from here.
So first—training, diversity, and education.
When the company showed off some graduates of its new Academy of Advanced Manufacturing and they talked about the intensity of the three month program, memories came back.
ManpowerGroup and Rockwell Automation celebrated the first military veterans to graduate from the Academy of Advanced Manufacturing and secure high-paying jobs in the rapidly-evolving manufacturing industry.
The 12-week program launched in August combines classroom learning with hands-on laboratory experience. Veterans are trained in Rockwell Automation’s state-of-the art facility in Mayfield Heights, Ohio for in-demand jobs in advanced manufacturing. All of the graduates have job offers and more than half have multiple job offers that significantly increase — some graduates even doubling — their previous salaries.
“This program felt like it was made just for me,” says Travis Tolbert, U.S. Navy veteran and academy graduate. “It focused on controls and automation, which is something I’ve always wanted to do, but was never able to do until now. The academy helped me take my military skills and understand how I could make them relevant for jobs outside of the Navy.”
“In recognition of Veterans Day, on behalf of Rockwell and ManpowerGroup, we thank all our veterans for their service,” said Blake Moret, CEO of Rockwell Automation. “We are honored to recognize our first military veterans to graduate the Academy of Advanced Manufacturing. We’ve seen their unique combination of core work and tech-savvy skills evolve to successfully position them for careers in the industry. We’re confident this program will help solve a challenge critical to the growth of advanced manufacturing.”
If the accomplishments and future prospects of these veterans didn’t bring a tear or two, you had to have no feelings.
Rockwell Automation has been announced as a 2017 Catalyst Award winner. The Catalyst Award honors innovative organizational approaches that address the recruitment, development and advancement of women and have led to proven, measurable results.
“We are thrilled to receive this recognition from Catalyst for our Culture of Inclusion journey, demonstrating our commitment to our employees, customers and community,” said Moret. “Our people are the foundation of our company’s success, and so we must create an environment where employees can and want to do their best work every day.”
The Culture of Inclusion journey began in 2007 with senior leaders renewing their commitment to diversity, inclusion and engagement. This was in response to employee data showing that women and people of color at the company had lower retention rates than white men, and there were gaps in the levels of representation for key demographics. A driving force of this strategy is the knowledge that in order to effect sustainable change, the dominant group—in this case, white men—must be aware of the impact of their privilege, be engaged, and partner with women and underrepresented groups in a meaningful way.
Results: Between 2008 and 2016, women’s representation in the U.S. increased from 11.9% to 23.5% among vice presidents, from 14.7% to 23.2% among directors, and from 19.3% to 24.3% at the middle-manager level. At the most senior leadership levels, women’s representation doubled, increasing from 11.1% to 25.0% among the CEO’s direct reports and from 11.1% to 20.0% on the board of directors. In addition, the Rockwell Automation voluntary turnover is well below the benchmark average for women.
On the Automation Fair show floor, Jay Flores, Rockwell Automation global STEM ambassador, led me on a tour of the FIRST Robotics area and explained how Rockwell is continuing its commitment to the program.
It announced a $12M, four-year commitment to FIRST—For Inspiration and Recognition of Science and Technology—founded to inspire young people’s interest and participation in science and technology.
Over the past 10 years, Rockwell Automation has provided more than $15M of broad-based support to address the critical need to fill science, technology, engineering and math (STEM) jobs that drive innovation. Many of these jobs go unfilled because of both the lack of awareness of the kinds of high-tech jobs available and the lack of skills to qualify for today’s needs.
“Through our technology and people, we are helping to inspire the next generation of innovators to fill the talent pipeline for our customers and for our company,” said Moret. “Our strategic partnership with FIRST helps us increase our reach and visibility to STEM students around the world.”
In addition to being a global sponsor of the FIRST LEGO League program and sole sponsor of the FIRST Robotics Competition (FRC) Rockwell Automation Innovation in Control Award, nearly 200 Rockwell Automation employees around the world donate their time for the FIRST programs, and more than 300 employees volunteer for the organization in other capacities. The company also donates products integral to FIRST program games and scoring. These product donations are specifically used for the FIRST Robotics Competition playing fields and scoring systems, and they are included within the parts kits teams use to build their robots.
“This generous, multiyear commitment from Rockwell Automation will allow us to focus on the strategic aspects of our partnership while continuing to help scale our programs and expose students to a broader range of industry-leading products and applications,” said Donald E. Bossi, president, FIRST. “The company has a long, rich history of supporting FIRST.”
Companies are adopting Lean manufacturing with increasing frequency. And that is a good thing. A Lean culture is people-friendly, not to mention profit friendly. And thus the story of a GE Brilliant Factory award winning plant.
GE has around 400 manufacturing locations. It has had a contest to find the “most brilliant of the Brilliant Factory” plants in its system. I had the opportunity to interview Rob McKeel, CEO of GE Automation & Controls, whose plant in Charlottesville, VA was one of the 17 chosen from 400+.
Manufacturing Day was last Friday, but we need to continue to promote the importance of manufacturing and production throughout the year so that we can attract our fair share of the best and brightest young people into our industry.
McKeel told me the theme is digitizing Lean manufacturing. The plants are using the advantages of GE’s tools. Different plants chose different problems to tackle. The A&C factory in Charlottesville, VA was chosen as one of the “Most Brilliant of the Brilliant Factories” by meeting its goal to reduce cycles—lean out inventory turns.
The biggest challenge was changing the culture to really become Lean. The worker at the line really owns the results in Lean. Everyone around them has the function of supporting the line worker. On Gemba walks, the line leader presents the situation for that line and then asks for help. Help is given immediately.
Here’s a video that GE created about its Brilliant Factory in Charlottesville.
The second thing is to apply technology. Some technologies used included robots, augmented reality, and visualization to provide data in real time.
“We have a very different plant from 25 years ago—mostly due to tapping the energy of the people,” stated McKeel.
I asked how they went about transforming culture. He told me that first the plant manager went to Toyota to study the Toyota Production System. He took the “big” course. But everyone needs to understand. So then he had some team members took Lean training at Toyota. Then, walking the talk, showing the changes they wanted to effect. The first teams learned to react to worker problems quickly. That action and trust led to other questions. Main value is that the worker comes first, management and other team members support the worker.
Sounds to me like they used a basic method of creating trust. Without trust, you’ll never have a successful Lean implementation.
McKeel said, “We don’t have a single unproductive moment for the worker.”
A&C was awarded the GE Brilliant Factory of the Year for its leadership, people and manufacturing excellence. While four inventory turns per year has long been standard in the industry, the Charlottesville BF is pacing for 50 inventory turns in 2017 on its model product line.
Part of the media / analyst program at Emerson Global User Exchange 2017 was a tour of a manufacturing facility and state-of-the-art (or beyond) training center. The Shakopee, MN facility includes final assembly of pressure sensors, product design (which we didn’t see), and an Interactive Plant Environment training center. The latter itself is a $10 million investment. This is one of two (Charlotte, NC being the other) while a third is planned in Houston.
The Interactive Plant Environment training center includes a classroom and a production facility. The facility includes tanks and pipes, valves, sensors and instrumentation, water and air. No steam or corrosive chemicals, of course. It helps customers and students simulate real life process conditions through hands-on learning in a safe environment. The IPE boasts a breadth of Emerson products where students can increase skills and knowledge through real-life scenario-based labs. Students are taught an aspect of instrumentation and then given a work order. They don hardhat, safety glasses, steel-toed shoes and enter the “plant” to perform the work—whether it be trouble shooting or calibration or whatever.
Students have the opportunity to better understand best practices and troubleshooting techniques from the mentorship of certified Emerson instructors. It is as if they are immersed in a typical plant environment (minus smells and mud) where they can replicate the most common, as well as unexpected, operational scenarios.
This is a great example of forward thinking in the training field. It is also impressive that Emerson continues to make these investments. Emerson alone among its competitors at this time is showing momentum and growth.
The first thing we saw past the lobby was a Collaboration Center. Looking like a high-tech conference room, this Center enables customers to learn to manage remote operations and interact with experts located anywhere in the world. There is one display for video conferencing. Another digital wall includes capability to display a variety of information that people in the room can interact with. The displays may include weather maps with maps of facilities. Or perhaps a “heat map” of wireless installations. This should be a great productivity booster.
Production facility is an excellent example of Lean Manufacturing. We saw an excellent Kanban system as well as many other examples of the visual factory, 5S, and more. I just love seeing the spreading adoption of lean. It’s great for workers, as well as, great for the bottom line.