From Statistics to Metrics: Jim McConnell’s Guide to Measuring Security’s Effectiveness

0:00

Manish:

Jim McConnell has over 30 years of experience in corporate security and over 33 years across critical infrastructure industries, most recently serving as a fellow at a Fortune 25 security organization while supporting 400 plus NGOs pro-bono. He is now the principal owner for Ask McConnell LLC, serving clients by solving key security pain points from a converged security perspective. Jim has 15 patents and published two books on converged security metrics and another on converged safety metrics. Jim is also a volunteer first responder, adjunct instructor at Texas A&M TEEX, and a mentor to our law enforcement and military heroes. Please join me in welcoming Jim to our Connected Intelligence Podcast. Welcome, Jim.

Jim:

Thank you, sir. Honored to be here and a great organization to partner with and we go back a while, and so great to be here and support the organization and the audience that we have.

Manish:

Terrific, and it’s an honor to have you, Jim. So for our audience, maybe tell us a little bit about your background for those that don’t know you. Many do, but for those that don’t.

Jim:

Sure. So beyond the bio, been involved in the various areas of security, started out as a kind of classic IT, then went to cyber, from there got introduced to incident response that drove into supply chain security and physical security and executive protection work as an operator and a PMO. So just what a privilege and honor to have a path that allows me to cross all of those different areas. And my class that I do for military and law enforcement is they’re mentoring them, I get to cover all of those areas, not just one particular area so that they can kind of focus on where they think they would be excited about getting into the security industry. So great to be here and great to support however we can.

2:37

Manish:

And look at you now, you’re an author, how about that? And an author for some time. So two books, one, and we’re gonna play a lot with words in this podcast, but one on converge safety and one on converge security. So maybe disambiguate that for us, maybe why the two differences and help educate us.

Jim:

Sure, as part of my career, I just, you know, overall just struggled with terminology that I and others use, the industry use, standards use, and then the scope. And so spent, as I tell people, everybody should write a book, but I spent more time on the dictionary, on the definition section than all the others, just because I wanted to be consistent. What the industry does in different documents, great, but I just wanted for my audience, whether it’s a client, an NGO or any other group, wanted to be consistent in my side. So, I started out with security metrics, convert security metrics, and then later on had a lot of requests for the safety side. So here’s how I define it. Maybe different for different people in the audience. So security to me is the prevention, detection and response to a crime or a violation of an organization’s rules or policies. Safety to me, and this was not the United States OSHA version of it, but just the Jim McConnell operational side is same thing, prevention, detection and response. But in this case to accidents, spilt milk, broken glass, those types of things. And people kind of say, well, I use this word and they use that word type thing. And I tell people what we really talk about a lot is we want people to feel safe because it’s a feeling. And so how do we feel safe? We have to do security and safety. At Ontic, if I walked into your building, I want to see and experience some security things and some safety things to be able to say I’m comfortable walking into the Ontic office. No different than a single mom dropping off Johnny at a daycare. They want to see both of those so they feel safe to move forward and take advantage of those resources.

5:03

Manish:

So Jim, you’ve spent quite a bit of time in a large enterprise in corporate America. There are many of our listeners that began their career in military or at a three-letter agency, then they came to this sector, the private sector. And the language, the semantics are a little bit different. Another two words that often get interchanged are statistics and metrics. especially inside of corporate America. So how do you separate the two and when should one be used over another?

Jim:

Sure, so I separate them from an operational standpoint, maybe not the super scientific I’ll leave it up to the great folks at NIST that do that better than I ever will, and I’ve worked with them many times. But to me, statistics are the counting of something, the counting of how often a door opens or closes, how often an event happens or doesn’t happen. And it’s just, to me, a measure of those ebb and flows of things that happen. And they can be valuable and they can be on pretty graphs and dashboards and things like that. To me, metrics is where we’re really driving for, are we improving on the statistics that can be a part of the formula of metrics? And so if I say, let’s take, if I walked around the building that Ontic is headquarters at and I counted all of the fire extinguishers, And I said, okay, there’s 45 fire extinguishers in there. Okay, that’s a stat in my mind. But if I go and say, 40 of those have not been inspected in the last 30 days. So whatever that percentage ends up being, 80% of those have not been inspected. Okay, I can fix 80% and the audience, whether that’s a board member or a safety person or a regulator or something like that can say, 80% sounds bad. And so, as I say in my book and I talk about all the time, red is bad, green is good. So 80% is red, 20% is good. So now can we find the owner of that 80%? I think the natural communications of different audiences can take and say, how do I get less red? Everybody knows 80% is bad. And so let’s reduce that down. Now, if it’s fire extinguishers, they may say 1% is bad too, but it’s relevant. So to me, stats are those things that we need to use around volume and metrics, things like that. Smaller companies, statistics may be more valuable. Larger companies, when it’s not 40 fire extinguishers, it’s 5,000 fire extinguishers. You know what? So we got 5,000 fire extinguishers. Great. So I think as you get bigger organizations, more complex organizations, metrics, hands down, will move the needle much easier.

Manish:

And Jim, I noticed as a segue, in your book, you use percentages over absolute numbers. So just more out of curiosity, why percentages? You have littered the book with lots of percentages.

Jim:

Sure, I kind of went back and put on my third grade hat and used pie charts and percentages solely to say, let’s keep this simple. If you don’t have something, let’s keep it simple. Let’s start out simple. If we want to get more complex, more colors, more types of grass, great. But if we’re starting out or trying to mature, we’re not getting the feedback and the ROI out of our metrics program, maybe we’ve overcomplicated it. So to me, I just, it was kind of a simplification thing. And again, 80% of the fire extinguishers are bad. You don’t need a lot of PhDs around to figure out that we’ve got to go fix that. We’ve got to find the facilities person or the vendor that takes care of those, uh, before, you know, the fire department comes in and does their inspection. So just, um, I think everybody gets percentages, and they can grab onto that, find an owner, and say, get that number down.

9:22

Manish:

It’s interesting. In many, many of our podcasts, we talk about silos, silos that exist organizationally, culturally, through tools, technology, etc. And here you are promoting convergence as do we here at Ontic, but I’m curious how you define convergence and keep in the back of your mind as you answer the question, how to overcome these silos? Because convergence sounds like a utopian case, but what have you experienced?

Jim:

Yeah, so my view is if we’re good security professionals and smart board members, we’re thinking about all, and I use the term all in every language means all. And so to me, from a convergence standpoint, people need to not get so tied up on immediately going to an org chart that says all security rolls up to a particular human. What it is really is around the awareness at the engineer level all the way to the board that says, the security of our organization is all of these different areas. There may be different owners within that, and that’s an exercise I call who’s who that needs to take place. But it’s really around all the different threats, all the different vulnerabilities, all the different mechanisms that we do that deal with that, going back to my original definition of this term called security. So is that cyber? Is that fraud? Is that investigations? Again, yes, for the standpoint of reducing risk and probability of bad happening with an organization. Let’s make sure we think about all of those different areas. Again, I always caution people, don’t focus on immediately going and saying they all report to the same person. That’s what we get. I think a lot of times in industry we get all tied up on. Again, if we’re good security professionals, we’re good relationship building within our organizations. Let’s get together and work on the betterment of security for the whole organization; converge the relationships, converge the technology, converge the intelligence, all the different areas to better the security and reduce risk.

11:41

Manish:

You made a bold move in your book. You went beyond top 10. You went to 25 metrics, which is great. Love to hear the context behind that. But can you cite one or two critical metrics, irrespective of size of organization or maturity of organization, that our listeners should be thinking about?

Jim:

Sure. I always put together one of the chapters is for CEOs. And in fact, if we want to have that particular chapter or that table available for the audience, we’ll add it into the show notes here to have that available. But it’s kind of that if there was 25 that the CEOs or the boards or us as general practitioners could say, let’s start somewhere. I like to start there. And so in that we think about things like, what percentage of the employees have been trained on a particular topic of security? What percentage of our buildings have a first aid AED kit in them? So again, a number of those things run it. But I really like to think about, you know, right of bang or right at bang. And so how many security incidents are in a trackable database? as a really important one. And so training and tracking incidents, a big deal in my mind. And so part of that then gets into, you take, whether you go up a stack or down a stack, percentage of vulnerabilities that have an owner tied to them. And when I say owner, a human’s name, not a department’s name.

14:00

Manish:

There’s an old adage in large enterprises, those that are run operationally well, of, if you can’t measure it, you can’t manage it. But then there’s this notion of taking action. So it’s one thing to just measure. It’s another thing to close that loop and take action. How do you think about that and what advice do you have, again, for our listeners?

Jim:

Yeah, it’s counterintuitive, but sometimes I tell people don’t start a metrics program unless the audience is prepared for the answer. And so from my experience, I had a situation one time, we were doing some great analytics on some stats and creating some pretty dashboards and found a large number of new issues that we were running into for this particular audience. And I went to what I perceived as the owner that would fix it. And he says, Jim, you want to know how many of those I want to be dealing with? I’m already understaffed and dah, dah, dah, dah. And he, you know, gave me the big fat zero kind of thing. Didn’t change the facts that they were there. And so part of this is a culture of the results of metrics, making sure your culture and your people are ready for that change. I think the other side of it is, I kind of use the joke of the golf course solution. And so if you’re doing good metrics across your organization, and again, denoting them by people, not by departments or legal entities or business units, but by people’s ownership, you start to get some interesting conversations on the proverbial golf course. Hey, I got 80% you only have 20%, either how’d you do that? Or, Hey, I’m going to, I’m going better than you. And so there’s some amazing things that can happen when you start to put up those things with people’s names on it, that you really start to move people to say, I don’t want to be up there anymore. How do I get rid of the red with my name on it? That also may identify some great opportunities of finding better ownership. Because you find, you take, it’s Billy Bob, and Billy Bob says, Jim, I can see why you think I own it, but I don’t. And I’m here to help you find who that owner is. Great. Then now you start to get into accountability and support going on. But it is definitely a baby step that has to take place, is the audience ready for this? And then ownership,tThat’s why I start with this kind of top 25, is don’t go to a hundred. We’ve got a lot of options for you, but start with 10. Cause if you can get that culture moving to say, this is kind of a normal thing, that’s going to come out, call it a state of security, a dashboard or whatever like that. You’ll baby steps this thing because it is a culture shock, particularly with folks that have to think about, hey, now that we know we got to fix, we didn’t budget for it, or we may have some regulatory reporting that we may have to do.

17:05

Manish:

So, Jim, this is a fun one. Measurement or how to measure the effectiveness of a security organization. You know, this is a combination of trying to prove that they’re aliens or UFOs that have landed and complex advanced calculus. For many, measuring effectiveness of something that hasn’t happened, particularly in physical security where you’re a cost center, is challenging and tricky. So how do you think about measuring the effectiveness of a security organization?

Jim:

Yeah, and so I think it starts with I think it’s three different areas, maybe in the wrong order of some people, but I think about the kind of people first here and a little bit of wisdom I’ve been granted over the years. One is for the security department itself. How is the security team doing in the way of emotionally, career path, how are they feeling about their training and their career path, things like that. So if the security team is, you know, happy and excited about their why of being there, to me, that’s, can you measure that? Sure. Can you do employee surveys of the security team? Could you do something like net promoter score within your own security team? Interesting. I think the second thing is the business. I got trained by an individual that was an expert in network promoter score. And I kind of said, well, why don’t we do net promoter score to the audience or to the clients, particularly internal, of the security function? And I’ve rarely seen that out in the real world of different organizations that I’ve had the privilege of working with. And so how are we doing? Are we surveying our customers, our clients, internally, net promoter score being one of those obviously survey things. And so there, and then we kind of can get into the harder stuff of threats and vulnerabilities and incidents and things like that. And I think that absolutely drives into some things around going back to, there are some things in those areas where we need to do stats. People are going to say, we’ve had less break-ins. Okay, that’s a stat. But I think then you get into the opportunity to say, people will understand if you’ve reduced your number of vulnerabilities across the converged security spectrum by 75%, that, you know, somebody is going to feel better about that. Hey, what do you need financially, organization, resource wise to get us from 75% down to 10% or 25 down to 10%. Great conversation type thing. So I think it’s the staff, your clients, and then the security stuff.

20:00

Manish:

This is a really interesting one, and we talked to many clients about this, which is how to communicate at the right levels to your various audiences and stakeholders. So there’s one form of communication, especially in measuring effectiveness, communicating metrics, sharing statistics, et cetera, with the frontline versus way up to leadership, and then all the way up to the boardroom. So how do you think about those audiences and how to tailor your communications depending on the audience.

Jim:

Again, sounds strange, but I always talk about it. What form did they, do they import input data into their eyes and brain every single day? So the example that I use in my class is the first thing, the, if you’re going to kind of think about justifying this at the board and the executive level, it sounds strange. Call their executive assistant, find out what model phone that they have. Because if what you’re about to deliver to them doesn’t fit on their phone, because they’re hardly ever on a laptop or looking at the big screen, might want to adjust your user interface, your report, or something like that. It sounds strange, but does it fit on here? Do they have to scroll 85 times with their finger to read your report? So sometimes it’s, how do we deliver that in its user experience? User experience, of course, as we know, is not just a technical term, but it’s that user experience. So to me is, how do they deliver that? How do we deliver that to them? Do they go into a particular war room every now and then and like to see something on the big screen? Okay, we’ll deliver that. But then it gets into, um, you could deliver every metric in my book to them. Sure. But what’s the five that you’d like to get their attention on? Ask them what five they would like to get. Again, that could be a survey or something like that. And so that’s that side. I think as we get lower down into the operational areas, I think it gets a lot more into getting face-to-face, getting out in the field. And you’re going to show some stats and they, the lower level folks love the stories. I’m doing some work right now on the job seeker recruiter fraud that’s going on in the world right now. And it’s great. I’ve got lots of stats and lots of things like that. But boy, when I get face-to-face with an audience of legal people or HR people or job seekers, and they see live on the screen, real live events, they go, oh, my goodness. And they’re like, I’m going to respond to that. So I think those lower folks around and I’m not here to design what that lower means, get out into the field, get off of Zoom, get your boots on, get out in the field and start talking to people face-to-face live. Budget for travel for this thing. We call it security awareness, call it a lot of things. And then you say, listen, and part of that is I need your help in reducing this metric down. Will you help me? So deliver it how the user experiences it.

23:18

YManish:

Well, Jim, shame on us if you and I didn’t touch on a topic that’s on everyone’s mind, which is AI. So pick your time horizon, two years, five years, 10 years. How do you, one, think security metrics are going to evolve? And then what role will AI play in all of this?

Jim:

Sure. I used AI the other day to see what it would do and said, let’s take a good known security standard, ISO 27,000, and I said, what’s missing from this classic standard, which of course goes back to something called the BS 7799 standard. And I asked one of the tools, I said, what is missing from this standard that AI challenges us with? And, you know, there was some things in there and I went, nah, that’s confidentiality. No, that’s integrity. No, that’s, that’s normal stuff. But there was two or three in there, that’s interesting, so those are some unique things in there that we need to kind of think about as security practitioners. And so I think that’s part of that conversation of how do we make sure people understand that it’s not as different in so many ways from a security perspective as our existing standards. In other words, don’t over-invent something new with solutions that we already have. I think the other side of it is, of course, I think the threats are tremendous. I have this job seeker investigation I’m working on right now. The amount of AI being used against the job seeker and HR teams is tremendous. Talk to an HR leader a couple of weeks ago, they had one job and had within a week had what? 2,000 applicants for that job, and they knew most of them were fraudulent type thing. So huge challenges, I think, on the threat side, both physical, cyber, and video, just tons of things there that we got to keep up with. And then I think the third thing is, how do we use it responsibly to start to be a force multiplier for us security professionals and everywhere that we look at. So if I’m out doing executive protection and I see something that is say a protest, and I don’t know the threat level of that protest, can I take some pictures and send it into a platform that can give me something that fits on my screen? It’s a response that says, Jim, that protest is related to X, Y, and Z, and gives him some analysis of that protest. Those are huge. So force multiplier from an AI, particularly for what I call the field or the eyes on glass type of folks that are working with us every day that need all the force multipliers we can give them.

26:15

Manish:

Jim, again, thank you for joining us on our podcast today. Two final questions for you. First one’s really straightforward. We have many listeners who might be a little nervous, a little unsure about how to go and begin a metrics measurement type of program. What advice or guidance would you have for them?

Jim:

Sure. I think it’s first gathering a small set of stakeholders into a physical or virtual room and saying, are we ready to go down this path? Are we ready to make this decision, make this path, determine ownership? of the things that we’re going to do. I think the next piece is to find to make sure that in the security team that’s creating these is do we have the skill set? And that skill set is not necessarily security people. It could be some graphics capability, some dashboarding capability, some folks that understand the business to be able to tell the stories behind the metrics. So do we have that skill set? And it may not be full time within the security team, but maybe it’s a collaborative team that says we have the skill sets to be able to produce great metrics within a user interface for user experience for the audience. And then the third area, I think, is really around starting with the top five, the top 10. Start small. And start with five that you’re comfortable with. But maybe they’re a little tight, a little bit more challenging. Overall, but start with five making sure people are understanding them can communicate them they come out in the form and the place that they need to be and then start to improve on those things making sure that we can say over a 60-day period we’ve been able to prove those metrics we can show the improvements overall and then start tightening the screws adding more on there and involving more organizations. And, you know, part of that is, of course, making sure that you’re reporting based on individuals and staying away from departments or business units.

28:20

Manish:

All right. Well, as is tradition here, we end every podcast with one final question, and that is, Jim, what does Connected Intelligence mean to you?

Jim:

Great question. I love the two words. We’ve talked about word and scope and definitions a lot. So to me, intelligence is things that we can deliver from the security department, and that intelligence could be intelligence we can gather for due diligence on a merger and acquisition all the way up to the threat of the day that goes on. So across the converged environment, we gather intelligence to help the business, one, feel safe, two, making sure that it helps the business make better decisions around risk, resources, and prioritizations of what they’re doing from a security and safety perspective. I think the connected side is we’ve got to connect the people that gather this information, whether that’s third parties or internal, and connect all the different security functions within the organization, some of those security functions might be handled by facilities or finance or something like that, but connect them all so that the audience of your Connected Intelligence really get a complete picture of what security and safety intelligence connects, reconnects all the different parts of the company to support the mission of the business.

Manish:

Our guest today, Jim McConnell. Jim, thank you for joining us on the Connected Intelligence Podcast.

Jim:

It has been an honor and a great organization to support. Thank you to Ontic and great looking at the future for you all and what you are going to be able to support our industry and organizations going forward and here to help and here to serve in the future.

Manish:

Thanks again.