by Gary Mintchell | Aug 30, 2021 | Capital Projects, Leadership, Workforce
Working on the factory floor early in my career taught me how much typical manufacturing workers know and care about the company’s products. Consultants came from time to time, studied, rearranged, left. Not much useful happened. But the individual guys (in those days) on the line knew more about what was going on than most of the supervisors and all of management.
Therefore, an opportunity to talk with Paul Vragel, Founder and President of 4aBetterBusiness in Evanston, IL to discuss his experiences as a project engineer and integrator was too good to pass up. After all, the values he learned and still implements include these:
- Listen to people
- Engage employees
- Respect
- Ask everyone to look for problems with no fault issued
- Assume employees have needed knowledge
Vragel told me, “My initial education and experience is in Naval Architecture and Marine Engineering – that is, ship design and construction. Building a ship involves building a hotel, a restaurant, a huge warehouse and a power plant, putting them all together, putting a propeller on it and sending it out on the ocean where there are no service stations. Ship design and construction is essentially a demanding, large-scale systems engineering project.”
After graduation from Webb Institute of Naval Architecture, he worked at Newport News Shipbuilding. After a year, 2 prior graduates of Webb Institute, working for Amoco Corporation, hired him, at the age of 22, to manage ship construction programs in Spain. “A couple weeks after I was hired, I was on a plane to Spain with my instruction set being, essentially, ‘figure out what you’re supposed to do, and do that’ ”.
After a year, one of the earlier ships built in the series came in to Lisbon for its guarantee drydocking and inspection. When we opened one of the crankshaft bearings of the 30,000 hp main diesel engine, we saw the bearing material, which was supposed to be in the bearing, was lying on the crankshaft journal, in pieces.
Talk about a complex situation—the ship was built by a company controlled by the Spanish Government. They were the holder of the guarantee. The engine was built by a different company, also controlled by the Spanish Government. Amoco had a contract with the shipyard, not the engine builder. And the engine was built under license from a company in Denmark.
Vragel was there as an observer for the new construction department. The ship was under the control of the operations department. I had no authority and no staff reporting to me. “I had no technical knowledge of poured metal bearings in high-powered diesel engines, I didn’t speak Portuguese or Spanish, I was 23 years old, and the instruction from my boss was very simple: ‘Fix It!’ To add to the urgency of the ship being out of service, the shipyard in Lisbon, where the ship was located, was charging $30,000/day (about $250,000 in today’s dollars), just for being there.”
Vragel went to the engine builder in Spain who said, “We don’t think we have a problem – we think the Danes have a problem. They designed the engine, we just built it according to their instructions.”
Figuring that getting the Danish engineers down to Spain for a meeting wouldn’t be productive, he decided the only thing to do was to go into the plant and talk to the people who made the bearings. One problem – they only spoke Spanish, and he only spoke English. But there are lots of ways to communicate if you really want to. “I observed what they were doing, pointed, asked a lot of questions – they learned a little English, I learned a little Spanish – and we sketched out how the bearings were made.”
After a couple of days, he thought he had figured out the cause of the problem, but “I had the good sense to shut up. While our communication had become pretty good, I was sure that there were other parts of the process they knew about that we hadn’t touched on that might be part of the problem or solution. If I just told them what I thought, everything would stop there without awareness of those elements and we wouldn’t get an effective solution. But if I could work with them through the process so they saw the issues, the employees would bring those additional elements to the table. We would have a full understanding of the system, the employees would be part of the solution. In this way, employees would have ownership in the results.”
“And that’s exactly what happened. With a little more effort we found and fixed the causes of the problem (which was causing porosity in the bearing).”
I had no authority, no technical expertise, no staff, I was 23 years old, I didn’t speak Portuguese or Spanish, and in a few days, working cross language and cross culture in an overseas plant I had never seen in a technology in which I had no experience, we together achieved a solution that permanently raised their manufacturing capability – that they owned.
This key formative experience led to the beliefs on which 4aBetter Business was founded:
- We believe that employees are the world’s experts at knowing what they actually do every day – their local systems
- We believe that 90% of the issues in a company are embedded in the way these local systems work and work together
This lesson applies to 22-year-olds and 52-year-olds alike. Sometimes we get so wrapped up in our own ideas that we overlook an obvious source of great expertise.
by Gary Mintchell | Aug 27, 2021 | Automation, Security
This reminds me of other technologies I’ve seen transition from few users to industry standard seemingly overnight. This latest survey from Nozomi Networks and SANS Institute finds industrial organizations are leveraging the cloud as they mature cybersecurity defenses and prioritize control system reliability. However, threats remain high and are growing in severity. In response, a growing majority of organizations have significantly matured their security postures since the last SANS OT/ICS survey in 2019. From the report: In spite of the progress, almost half (48%) don’t know whether their organizations had been compromised. The Nozomi Networks-sponsored survey echoes Nozomi Networks’ own experiences with customers worldwide.
“It’s concerning to see that nearly half of this year’s survey respondents don’t know if they’ve been attacked when visibility and detection solutions are readily available to provide that awareness,” said Nozomi Networks Co-founder and CPO Andrea Carcano. “Threats may be increasing in severity, but new technologies and frameworks for defeating them are available and the survey found that more organizations are proactively using them. Still, there’s work to be done. We encourage others to adopt a post-breach mindset pre-breach and strengthen their security and operational resiliency before an attack.”
Cyber threats to OT environments continue to rise and threat severity is at an all-time high.
- Most respondents (69.8%) rated the risk to their OT environment as high or severe (up from 51.2% in 2019).
- Ransomware and financially motivated cybercrimes topped the list of threat vectors (54.2%) followed by nation-state sponsored cyberattacks (43.1%). Unprotected devices and things added to the network came in third (cited by 31.3% of survey respondents).
- Of the 15% of survey respondents who indicated they had experienced a breach in the last 12 months, a concerning 18.4% said the engineering workstation was an initial infection vector.
- Nearly half of all respondents (48%) did not know whether their organizations had been compromised and only 12% were confident that they hadn’t had an incident.
- In general, external connections are the dominant access vector (49%) with remote access services identified as the most prevalent reported initial access vector for incidents (36.7%).
This year’s survey found most organizations are taking ICS threats seriously and making solid progress in maturing their security postures to address them. Over the last two years organizations have improved monitoring and threat intelligence capabilities. They are moving away from traditional indicator-based defense capabilities and moving toward threat hunting and hypothesis-based security models. They’re also focusing on data loss prevention.
- 47% say their control system security budget increased over the past two years.
- Almost 70% have a monitoring program in place for OT security.
- 51% say they are now detecting compromises within the first 24 hours of an incident. The majority say they move from detection to containment within 6 to 24 hours.
- 9% have conducted a security audit of their OT/control systems or networks in the past year and almost a third (29.5%) have now implemented a continual assessment program.
- 50% say they have vendor-provided ICS-specific threat intelligence feeds and there is less reliance (36%) on IT threat intelligence providers.
- OT SOC adoption is up by a sharp 11% from 2019 to 2021, re-emphasizing the focus away from traditional indicator-based defense capabilities and more toward a threat hunting and hypothesis-based security model.
- Data loss prevention technologies also saw a sharp increase in deployment (11%).
- As process reliability becomes a top concern, 34% say they’re implementing zero-trust principles and an additional 31% say they plan to.
ICS is Getting Cloudy
Adoption of cloud-native technologies and services transformed the IT industry. This year’s survey found similar impacts are also beginning to be felt in the OT environment.
- 1% of all survey respondents indicate they are using some cloud-based services for OT/ICS systems.
- Almost all (91%) are using cloud technologies to directly support ICS operations (combining remote monitoring configuration and analysis; cloud services supporting OT; and remote control/logic).
- All respondents using cloud technologies are using cloud services for at least one type of cybersecurity function (company NOC/SOC, business continuity and MSSP support).
- Respondents consider cloud assets relatively secure, with only 13% of responses classifying them as risky.
To learn more about the latest trends in OT/ICS cybersecurity:
• Download A SANS 2021 Survey: OT/ICS Cybersecurity
The SANS Institute was established in 1989 as a cooperative research and education organization. SANS is the most trusted and, by far, the largest provider of training and certification to professionals at governments and commercial institutions world-wide. Renowned SANS instructors teach over 50 different courses at more than 200 live cyber security training events as well as online. GIAC, an affiliate of the SANS Institute, validates employee qualifications via 30 hands-on, technical certifications in information security. The SANS Technology Institute, a regionally accredited independent subsidiary, offers master’s degrees in cyber security. SANS offers a myriad of free resources to the InfoSec community including consensus projects, research reports, and newsletters; it also operates the Internet’s early warning system–the Internet Storm Center. At the heart of SANS are the many security practitioners, representing varied global organizations from corporations to universities, working together to help the entire information security community. (www.SANS.org)
• Here are a few responses to questions about the report:
1 What were the most surprising things you found in the report?
Chris Grove – Technology Evangelist – Nozomi Networks
Positive: It was a pleasant surprise to see that a large group of respondents (40.1%) have embraced cloud-base services. It’s a trend that Nozomi Networks has seen in the field and one that we have responded to with our own cloud-based security offerings. As Industrial and critical infrastructure organizations embrace IoT and converge their OT and IT efforts, they must be able to protect thousands of devices quickly and cost-effectively from threats in real-time and ensure ongoing operational resilience. Cloud-based technologies make that possible. It’s also encouraging to see the majority are confident in the security of their cloud assets. We believe ICS organizations will continue to adopt cloud technologies and the adoption of cloud-base security solutions will grow significantly over the next few years.
Negative: It’s alarming to see that detection and response is still a significant issue for organizations. In fact, the problem seems to have grown since the previous survey (48% of survey participants did not know whether they’d had an incident vs. 42% in 2019). Solutions are available to address this problem and adopting them should be a top priority.
Mark Bristow – Author – A SANS 2021 Survey: OT/ICS Cybersecurity
I found three things particularly striking in the report results.
● The level of adoption of cloud technologies for operational outcomes was striking. Two years ago, cloud adoption was not being seriously discussed and now 49% are using it.
● Incident visibility and confidence is not high. 48% of respondents could not attest that they didn’t have an incident. A further 90% of these incidents had some level of operational impact.
● 18% of incidents involved the engineering workstation. This is a critical piece of equipment and having this involved in so many incidents is troubling.
2 What are three things you think ICS operators need to focus on moving forward to protect themselves?
Chris Grove, Nozomi Networks: Considering Ransomware is such a pervasive issue; it might be a first concern for many operators. Starting off with some tabletop exercises, operators would be able to identify areas where improvements can be made. Typically, one area that gets highlighted is the need for a systematic risk assessment that details likely points of entry and identifies ways to harden the target. Sometimes this is in the form of patching, network segmentation, policies, procedures, etc. In almost all cases, increased visibility makes everything easier to manage. From having a detailed asset inventory, to monitoring network traffic patterns, to inspecting traffic for attacks or operational anomalies…. visibility is a crucial component of successfully defending operations. Finally, the third and final thing that operators should consider is Consequence Reduction. As part of a post-Breach mindset, operators should consider the fact that eventually the attackers will breach the perimeter, and one should be prepared for that day. How do we limit the blast radius of the attack? How do we hold them at bay, and subsequently eradicate them from the system? How do we carefully maintain, safely shutdown, or restore operations potentially affected by the breach? These are tough questions to be asked before that day comes.
Mark Bristow, SANS:
· It’s great that we now have monitoring programs in place, but we are still mostly looking at the IT aspects of our OT environments. We need to be correlating our IT and OT security telemetry as well as process data to truly understand potential impacts to safety and operations.
· Focus on fundamentals. Too many respondents do not have a formal program for asset identification and inventory. Without this foundational step, further security investments may be invalid or misplaced.
· Ransomware is a huge risk, but it’s not one that is specifically targeting ICS. A malicious actor who is specifically targeting your ICS environment will not be as blunt or noisy as ransomware is, and we are struggling to defend against ransomware.
by Gary Mintchell | Aug 25, 2021 | Automation, Internet of Things, Networking, Operations Management, Security
Before the Industrial Internet Consortium changed its name (Industry IoT Consortium) I had two news items from it. The first is a Networking Framework publication and the second a definition for trustworthiness in cyber-physical systems. They both appear to be worthwhile additions to the state of the art.
IIC Defines Trustworthiness for Cyber-Physical Systems
The IIC has published IIoT Trustworthiness Framework Foundations. This foundational document explains the key concepts and benefits of trustworthiness in context, relating it to the real-world supply chain and offering model approaches. Trustworthiness is essential to government and commercial organizations with cyber-physical systems impacting the safety and well-being of people and the environment. These systems include industrial control systems and almost all systems that use digital technology to sense or affect the environment.
“Trustworthiness, and confidence in that trustworthiness, are an essential aspect of cyber-physical systems,” said Marcellus Buchheit, President & CEO, Wibu-Systems USA, a Co-Chair of the IIC Trustworthiness Task Group and one of the authors of the document. “Inattention to trustworthiness can lead to loss of human life, long-term environmental impacts, interruption of critical infrastructure, or other consequences such as disclosure of sensitive data, destruction of equipment, economic loss, and reputation damage,” continued Buchheit.
The IIoT Trustworthiness Framework Foundations document defines trustworthiness as a combination of security, safety, reliability, resilience, and privacy and the tradeoffs made among them in the face of environmental disturbances, human errors, system faults, and attacks. Ultimately, trustworthiness depends on the strategic intent and motivation of an organization, particularly its top management, to create and operate systems that inspire trust by partners, customers, and other stakeholders, including the community.
“Trustworthiness is the degree of confidence one has that a system performs as expected. It requires an understanding of the system, including interactions and emergent properties,” said Frederick Hirsch, Strategy Consultant, Upham Security, Co-Chair of the IIC Trustworthiness Task Group, and one of the authors of the foundational document. “In the digital world, trust and trustworthiness are achieved by understanding and addressing concerns related to the trustworthiness characteristics appropriately for the context of the entire system. Providing evidence of this can give others confidence.”
IIoT stakeholders will make different decisions and tradeoffs depending on the nature and or industry of the system. “Concerns in a factory are not the same as those for a hospital operating room,” said Bob Martin, Senior Principal Engineer, Cyber Solutions Innovation Center, The MITRE Corporation, Co-Chair of the IIC Trustworthiness Task Group, one of the authors of the document. “Designers must understand the many considerations involved in defining the appropriate trustworthiness implementation, including the supply chain, assembly, operation, and maintenance of a system.”
The IIoT Trustworthiness Framework Foundations document builds on the Industrial Internet of Things Security Framework (IISF). It is part of the IIC’s Industrial Internet Reference Architecture (IIRA), which provides an architectural framework of Industrial IoT Systems.
You can find IIoT Trustworthiness Framework Foundations and a list of IIC members who contributed to it here. Watch a short overview video. Register for the webinar, Ensuring Trustworthy Industrial Systems on September 1, 2021 at noon PST or 7:00 pm PST.
IIC Publishes IIoT Networking Framework
The IIC announced the Industrial Internet of Things Networking Framework (IINF) publication. The framework guides IIoT stakeholders on designing and developing the appropriate networking solutions to enable industrial IoT (IIoT) applications and stimulate industrial digital transformation. It details the requirements, technologies, standards, and solutions for networking that support diverse applications and deployments across a broad range of IIoT sectors and vertical industries.
“An underlying network is the foundation of any IIoT solution. It includes technologies at the network layer and below as well as related capabilities for management and security,” said David Lou, Co-chair, IIC Networking Task Group, Chief Researcher, Huawei Technologies, and one of the primary authors of the framework. “An underlying network enables the exchange of data and control and forms the basis of digital transformation across industries.”
The framework serves as a guideline and toolbox for IIoT networking solution stakeholders who design, develop, deploy, or operate the solutions and end-users in many industries trying to network their assets or products.
“IIoT applications span a range of industrial sectors as well as business, usage, deployment, and performance perspectives,” said Jan Holler, Co-chair IIC Networking Task Group, Research Fellow, Ericsson, and one of the primary authors of the framework. “The IINF helps organizations sort through numerous networking technologies to ensure interoperability across industry sectors. It answers the fundamental question, ‘How do I design, deploy, and operate a successful networking solution for my industrial IoT applications?'”
The IINF includes use cases from several industrial sectors, including smart factories, mining, oil & gas, and smart grid, to illustrate the diversity of networking considerations. Networking technologies and standards are covered in-depth to help organizations address their concerns and technical requirements. Finally, the IINF includes best practices for IT architectural blueprints.
by Gary Mintchell | Aug 24, 2021 | Asset Performance Management, Operations Management, Workforce
An old friend and several acquaintances found themselves adrift when a magazine closed. All being entrepreneurial, they started a website and newsletter—RAM Review (Reliability, Availability, Maintenance). Old friend Jane Alexander is the editor. Not meaning she’s old, just that we’ve known each other for many years.
I met Bob Williamson 10 or 12 years ago mostly around discussions of ISO 55000 on asset management. He wrote the lead essay for a recent email newsletter on workforce. Now, I have to admit that the only part of manufacturing I never worked in was maintenance and reliability. I did work with skilled trades when I was a sales engineer, though. I considered them geniuses for the way they could fix things. One of the points of Bob’s essay is taking care of things before they break and need help.
The main workforce discussion in media concerns remote or hybrid work. Many engineering roles can be performed remotely. Many roles within manufacturing and production must be performed on site. With the current and projected future labor shortage, I like his closing paragraph except for the put down on current operators. I knew plenty who cared for their machine or process. Of course, many didn’t. Most likely a management failure. But cross-training people to be at least to some degree both competent operators and first-line RAM people seems to me to be a winning strategy. I’ve reprinted most of Bob’s essay below. You can read it on their website.
For many manufacturers, returning to traditional ways of work simply will not be an option. Something must change if they are to attract, hire, and retain a capable workforce. Therefore, I believe technology and desperately willing top-management teams will also help alter work cultures on factory floors. Respondents to the Manufacturing Alliance/Aon survey suggested offering “flexible working hours, compressed work weeks, split shifts, shift swapping, and part-time positions.” Use of such enticements with plant-floor workforces would look very different than use among the carpet dwellers in front offices.
We have another option, of course: Technology can automate our manufacturing processes, and much of it is far more affordable than it was a decade ago. In fact, given the rising cost of labor over the past decade, with increasing healthcare-cost burdens and skills shortages, many businesses have already automated some of their labor-intensive processes. The times we are in call for—make that scream for—large-scale automation. Yet, while process automation can be easier for large, deep-pocketed companies than for the smalls, it’s still a huge challenge.
There are four big hurdles to be overcome when automating manufacturing processes: availability, installation, sustainable reliability, and work-culture change. And remember, skills and labor shortages are widespread in these post-pandemic times. Moreover, despite the supply chain’s efforts to heal and keep up, manufacturers of automation technologies aren’t immune to the production-barrier ills that others face these days.
To repeat: RAM professionals are on manufacturing’s front line. Skill shortages may be affecting our ranks, but there are recruiting and training efforts underway in many companies to remedy the situation. In addition, we have technologies for carrying out data collection, analysis, and problem-solving somewhat remotely. However, the boots-on-the-ground parts of reliability and maintenance will not be virtual or remote.
So, consider this option: Recruit and train displaced production workers to wear some RAM “boots.” They’ll be familiar with industrial environments and the importance of plant equipment. Then, let’s train our current production workers to care more for their machines than they did in the past, and, in the process, become the eyes and ears for reliability, availability, and maintenance improvement.TRR
by Gary Mintchell | Aug 23, 2021 | Automation, Technology
Augmented Reality (AR) and Virtual Reality (VR) are not enough deviations from Reality for today’s technology. Now we have XR, which includes virtual reality, augmented reality, mixed-reality, and future immersive technology. Two organizations—Perkins Coie and the XR Association—came together to conduct an Insider Survey probing the perceived state of the art.
Jason Schneiderman, co-lead of the Immersive Technology (AR/VR/XR) vertical and Emerging Company Venture Capital attorney at Perkins Coie, discussed the news regarding opinions of post-pandemic growth with me recently. One of the limiting factors is limited content. But, 71% of those surveyed thought investment would increase post-pandemic. For manufacturing, people weigh the cost of not flying in an expert as part of the AR/VR investment. Automotive #1 in optimism, manufacturing number 7.
From the news release:
Disruption caused by COVID-19 has strengthened immersive technology’s prospects, which could get even stronger as the world emerges from the pandemic, according to the fifth annual XR Industry Insider Survey (previously the Augmented and Virtual Reality Survey) released today by Perkins Coie and leading industry group the XR Association.
The survey of 164 industry stakeholders involved in XR, which encompasses virtual, augmented, and mixed reality, shows the pandemic may have actually supercharged the industry’s momentum of the past half-decade. In light of COVID-19’s effects—namely, social distancing guidelines that forced a global shift toward remote connectivity and increased comfort and familiarity with online and virtual experiences—more than eight in 10 respondents say investment in immersive technology will increase in 2021 compared with 2020.
The survey, conducted in April 2021, was preceded and informed by group interviews with experts in the field. Overall, the results show that immersive technologies continue to expand beyond gaming and entertainment, with sectors like healthcare, education, retail, and workforce development and training seen as areas of potential growth.
Eighty-three percent of respondents expect immersive technology investment in 2021 to be higher than in 2020, compared with 68% in our previous survey, who said 2020 investment would be higher than in 2019. Additionally, 37% said investment would be significantly higher in this year’s survey, compared with 26% last year. Notably, optimism is higher for some industries; after the reliance on remote work over the past year, 95% of respondents say their organizations plan to increase spending on immersive technology for better remote collaborations and trainings.
The biggest barrier to mass adoption revolves around access to software. Half of respondents said development of more accessible software (not specifically regarding open-source software or software built for persons with disabilities) to meet the needs of all users will be the biggest driver of enterprise adoption. For consumer adoption, respondents said the top driver was availability of and access to open-source software and communities (61%).
Other challenges are consistent with results from past year’s surveys. User experiences was cited by 65% of respondents as the top barrier to greater adoption of immersive technology, while content offerings (53%) and costs to consumers (27%) were other hurdles.
Validating some anecdotal evidence that the industry is becoming more diverse, six in 10 survey respondents identified as working for a minority-owned or a female-owned company.
Findings showed that minority-owned companies were more likely to strongly agree, while respondents from women-owned businesses were more likely to agree or strongly agree on the lack of compelling content. For respondents from minority-owned companies, existing content is especially lacking when it comes to being interactive and immersive. In addition, respondents from female-owned companies were more likely to think educational content is lacking even though they think the pandemic has affected the use of immersive technology in education more than any other sector except marketing and advertising (the full survey pool chose healthcare).
Perkins Coie is a leading international law firm that is known for providing high value, strategic solutions and extraordinary client service on matters vital to our clients’ success.
The XR Association promotes the dynamic global growth of the XR industry, which includes virtual reality, augmented reality, mixed-reality, and future immersive technology. XRA is leading the way for the responsible development and adoption of XR by convening stakeholders, developing best practices and research, and advocating on behalf of our members and the greater XR industry.
The XR Association represents the broad ecosystem of the XR industry including headset manufacturers, technology platforms, component and peripheral companies, internet infrastructure companies, enterprise solution providers, and corporate end users. The founders of XRA are Google, HTC Vive, Microsoft, Oculus from Facebook, and Sony Interactive Entertainment.
