How can you Measure how Secret a Secret is?

Download the full Article

When a few know something and want to keep others from learning, that’s a secret. Everyone has secrets, some small, like eating a bar of chocolate when you are on a diet, some are personally important, like a embarrassing personal or professional mistake in the past. There are as many types of secrets as people and organisations that keep them, among them:

• Personal secrets and family secrets, normally related to the moral and taboos of the culture where they live.
• Business secrets, like financial information, strategy and trade secrets.
• Law enforcement secrets, like forensic or methods, investigation information and details about ongoing investigations.
• Crime secrets, like insider trading, organised crime and gangs.
• Political secrets, (Most nations have some form of Official Secrets Act and classify material according to the level of protection needed) like:

o Weapon designs and technology (nuclear, cryptographic, stealth).
o Military plans.
o Diplomatic negotiation positions.
o Intelligence information, sources and methods.
o International relations, secret treaties like:
• Molotov-Ribbentrop pact.
• Cuba crisis agreement.
• Dover treaty.
• Quadripartite agreement.
• Sykes-Picot agreement.

• Social secrets, like certain religions or secret societies as the masonry.
• Professional secrets, like health workers, social workers and journalists.
• Other, like video tape rental and sale records in the USA.

While we all have an intuitive way to distinguish small secrets from high secrets, there hasn’t been so far a way to measure it.

By using the following formula:

Secret = Log C*(Sum Tdk / Sum Tk) = Log C + Log ( Sum Tdk / Sum Tk )

Where C is the quantity of information, Tk is the time someone has known the secret, Tdk the time has had interest in knowing the secret.

If C=1, we can see some examples:

¿Who did “Famous for Nothing” went with for a dirty weekend last summer? If two people know since the 1st of August, 48 more since the 1st of September and 100.000 coach potatoes would like to know and they find out after five months, just before revelation the secrecy is:

S = Log ((100.000 * 5) / (2 * 5 + 48 * 4)) = 3,39

if only 8 people had found out in September, the secret would be S = 4,04

¿Who killed Kennedy? Let's suppose two people know since 1963, and 300 million Americans would like to know. After 42 years:

S = Log( 300 million * 42 / 2 * 42 ) = 8,1

¿Who was Deep Throat? Just before it was found, 4 people knew, and 300 millions were interested. After 33 years:

S = Log( 300 million * 33 / 4 * 33 ) = 7,8

What if 2 more people had found out after 30 years?

S = Log( 300 million * 33 / (4 * 33 + 2 * 30) ) = 6,7

At a business, perhaps a secret is important for a few years, and all your competitors would be eager to know. If 10 people in the company know for a year, and 150 people from other companies would like to know:

S = Log( 150 * 1 / (10 * 1) ) = 1,17

If after two years the market is more competitive and more people (1500) is interested:

Secret = Log (1500 * 1 / (10 * 1) ) = 2,17

For the sake of example, I will use the following groups for measuring secrets:
• Family 10
• Social environment 100
• High school 300
• Competition 300
• Gang 50
• Police 100000
• Army 1000000
• Population 30000000
• Foreign Armies 15000000

Applying the formula, the following approximate values can server as examples of measured secrets:
• A social secret, like the confidence of a friend or an alibi of where you slept, 0,70
• Secret signs and telling signs from a gang, 0,95
• Hiding a mistake, like breaking something, 1,00
• Keeping the privacy of others like a confession, or the social situation of someone, 1,00
• Keeping the privacy of an average person like a list of video rentals of records of library use, 1,04
• Cheating on your wife/husband, 1,44
• Secrets from a Masonry, Scientology or Mirrahism, 1,48
• A regular trade secret, 2,18
• A mistake or wrongdoing of a politician 2,40
• Mafia, Yazuka or insider trading, 3,30
• Keeping the privacy of a politician like a list of video rentals of records of library use, 3,48
• Identity of a witnesses in a criminal case, 5,70
• A journalistic source, 5,78
• The Coca cola formula, 6
• Corruption, misuse of public funds 7,57
• Nuclear weapons, cryptography, 8,00
• Osama Bin Laden location 8,50

Mysteries, secrets known by no one, like those discovered by Champollion when deciphering Egyptian hieroglyphics have S=infinite.

Ignorance on the existence of the secret makes it less secret, as the interest on learning it is less, and the effort on keeping it secret is lower as well. Unfortunately it is very difficult to estimate the number of people interested in a secret, so the accuracy measuring secrecy won’t normally be very high. This way of measuring secrets can lead to some interesting exercises, like adding a factor to the formula for how intense is the interest for learning the secret or analyzing the diffusion of a secret in a group depending on the likelihood of every member of the group of revealing it.

Measuring secrecy can help to gain an understanding on how secret is the information we handle and the kind of efforts to make to keep it secret. To have a clear understanding on the reasons for keeping the secrecy and the influence of time, interest and group of people who know the secret, can give insights on how to manage secrets properly. Two conclusions can be easily drawn from the formula. Firstly, preventing others to know the existence of the secret makes it easier to keep it; Secondly keeping the group of people who know the secret as small as possible prevents leakages more effectively than any technical measure.

A clear understanding on the type of secrets in a organisation, how secret they seem to be, the impact of their revelation and a measure of their secrecy is the first step to a cost effective and efficient classification and protection of secrets.

If you liked this article, consider taking advanced ISMS training online via Udemy

Process Management with Security Metrics

A metric is a quantitative measurement that can be interpreted in the context of a series of previous or equivalent measurements. Metrics are necessary to show how security activity contributes directly to security goals; measure how changes in a process contribute to security goals; detect significant anomalies in processes and inform decisions to fix or improve processes. Good management metrics are said to be S.M.A.R.T:

  • Specific: The metric is relevant to the process being measured.
  • Measurable: Metric measurement is feasible with reasonable cost.
  • Actionable: It is possible to act on the process to improve the metric.
  • Relevant: Improvements in the metric meaningfully enhances the contribution of the process towards the goals of the management system.
  • Timely: The metric measurement is fast enough for being used effectively.

Metrics are fully defined by the following items:

  • Name of the metric;
  • Description of what is measured;
  • How is the metric measured;
  • How often is the measurement taken;
  • How are the thresholds calculated;
  • Range of values considered normal for the metric;
  • Best possible value of the metric;
  • Units of measurement.

Security Metrics are difficult to come by
Unfortunately, it is not easy to find metrics for security goals like security, trust and confidence. The main reason is that security goals are “negative deliverables”. The absence of incidents for an extended period of time leads to think that we are safe. If you live in a town where neither you nor anyone you know has ever been robbed, you feel safe. Incidents prevented can’t be measured in the same way a positive deliverable can, like the temperature of a room.

Metrics for goals are not just difficult to find; they are not very useful for security management. The reason for this is the indirect relationship between security activity and security goals. Intuitively most managers think that there is a direct link between what we do (which results or outputs) and what we want to achieve (the most important things: our goals). This belief is supported by real life experiences like making a sandwich. You buy the ingredients, go home, arrange them, and perhaps toast them and voilá: A warm sandwich ready to eat. The output sought (the sandwich) and the goal (eating a home made sandwich) match beautifully.

Unfortunately, there is no direct link every time. A good example can be research. There is not direct relationship between goals (discoveries) and the activity (experiments, publication). You can try hundreds of experiments and still not discover a cure for cancer. Same thing happens with security. The goals (trust, confidence, security) and the activity (controls, processes) are not directly linked.

When there is a direct link between activity and goal, like the temperature in a pot and the heat applied that pot, we know what decision to take if we want the temperature to drop: stop applying heat But, how will we make a network safer, adding (more accurate filtering), or summarising (less complexity) filtering rules? We don’t know.  If a process produces dropped packets, more or less dropped packets won’t necessarily make the network more or less secure, just like a change in the firewall rules won’t necessarily make the network safer of otherwise.

The disconnect present in information security between goals and activity prevents goal metrics from being useful for management, as you can never tell if you are closer to your goals because of decisions recently taken on the security processes.

Goal metric examples:

  • Instances of secret information disclosed per year. What can you do to prevent people with legitimate access to disclose that information?
  • Use of system by unauthorised users per month. What can you do to prevent people from letting other users to use their accounts?
  • Customers reports of misuse of personal data to the Data Protection Agency. Even if you are compliant, what can you do to prevent a customer to fill a report?
  • Risk reduction per year of 10%. As risk depends on internal an external factors, what can you do to actually modify risk?
  • Prevent 99% of incidents. How do you know how many incidents didn’t happen?

Actually useful security metrics
If metrics for goals are difficult to get, and are not very useful; what is a security manager to do? Measuring process outputs can be the answer. Measuring outputs is not only possible but very useful, as outputs contribute directly or indirectly to achieve security, trust and confidence. Using output metrics you can:

  • Measure how changes in a process contribute to outputs;
  • Detect significant anomalies in processes;
  • Inform decisions to fix or improve the process.

There are seven basic types of process output metrics:

  • Activity: The number of outputs produced in a time period;
  • Scope: The proportion of the environment or system that is protected by the process. For example, AV could be installed in only 50% of user PCs;
  • Update: The time since the last update or refresh of process outputs.
  • Availability: The time since a process has performed as expected upon demand (up time), the frequency and duration of interruptions, and the time interval between interruptions.
  • Efficiency / Return on security investment (ROSI): Ratio of losses averted to the cost of the investment in the process. This metric measures the success of a process in comparison to the resources used.
  • Efficacy / Benchmark: Ratio of outputs produced in comparison to the theoretical maximum. Measuring efficacy of a process implies the comparison against a baseline.
  • Load: Ratio of available resources in actual use, like CPU load, repositories capacity, bandwidth, licenses and overtime hours per employee.

Examples of use of these metrics:

  • Activity: Measuring the number of new user account created per week, a sudden drop could lead to detecting that the new administrator is lazy, or that users started sharing user accounts, so they are not requesting them any more.
  • Scope: In an organization with a big number of third party connections, measuring the number of connections with third parties protected by a firewall could lead to a management decision not to create more unprotected connections.
  • Update: Measuring the update level of the servers in a DMZ could lead to investigating the root cause if the level goes above certain level.
  • Availability: Measuring the availability of a customer service portal could lead to rethinking the High Availability Architecture used.
  • Efficiency / Return on security investment (ROSI): Measuring the cost per seat of the Single Sign On systems of two companies being merged could lead to choose one system over the other.
  • Efficacy / Benchmark: Measuring backup speed of two different backup systems could lead to choose one over the other.
  • Load: Measuring and projecting the minimum load of a firewall could lead to taking the decision to upgrade pre-emptively.

There is an important issue to tackle when using output metrics; what I call the Comfort Zone. When there are too many false positives, the metrics is quickly dismissed, as it is not possible to investigate every single warning. On the other hand, when the metric never triggers a warning, there is a feeling that the metric is not working or providing value. The Comfort Zone (not too many false positives, pseudo-periodic warnings) can be achieved using an old tool from Quality Management, the control chart. The are some rules used in Quality Management to tell a warning, a condition that should be investigated from a normal statistical variation (Western Electric, Donald J. Wheeler's, Nelson rules), but for security management the best practice is adjusting the multiple of the standard deviation that will define the range of normal values for the metric until we achieve the Comfort Zone, pseudo-periodic warnings without too many false positives.

Using Security Management Metrics
There are six steps in the use of metrics: measurement, representation, interpretation, investigation and diagnosis.

Measurement: The measurement of the current value of the metric is periodic and normally refers to a window, for example: “9:00pm Sunday reading of the number of viruses cleaned in the week since the last reading” Measurements from different sources and different periods need to be normalized before integration in a single metric.

Interpretation: The meaning of a measured value is evaluated comparing the value of a measurement with a threshold, a comparable measurement, or a target. Normal values (those within thresholds) are estimated from historic or comparable data. The results of interpretation are:

  • Anomaly: When the measurement is beyond acceptable thresholds.
  • Success: When the measurement compares favourably with the target.
  • Trend: General direction of successive measurements relative to the target.
  • Benchmark: Relative position of the measurement or the trend with peers.

Incidents or poor performance take process metrics outside normal thresholds. Shewhart-Deming control charts are useful to indicate if the metric value is within the normal range, as values within the arithmetic mean plus/minus twice the standard deviation make more than 95.4% of the values of a normally distributed population. Fluctuations within the “normal” range would not normally be investigated.

Investigation: The investigation of abnormal measurements ideally ends with identification of the common cause, for example changes in the environment or results of management decisions, or a special cause (error, attack, accident) for the current value of the metric.

Representation: Proper visualisation of the metric is key for reliable interpretation. Metrics representation will vary depending on the type of comparison and distribution of a resource. Bar charts, pie charts and line charts are most commonly used. Colours may help to highlight the meaning of a metric, such as the green-amber-red (equivalent to on-track, at risk and alert) traffic-light scale. Units, the period represented, and the period used to calculate the thresholds must always be given for the metric to be clearly understood. Rolling averages may be used to help identify trends.

Diagnosis: Managers should use the results of the previous steps to diagnose the situation, analyse alternatives and their consequences and make business decisions.

  • Fault in Plan-Do-Check-Act cycle leading to repetitive failures in a process -> Fix the process.
  • Weakness resulting from lack of transparency, partitioning, supervision, rotation or separation of responsibilities (TPSRSR) -> Fix the assignment of responsibilities .
  • Technology failure to perform as expected -> Change / adapt technology.
  • Inadequate resources -> Increase resources or adjust security targets.
  • Security target too high -> Revise the security target if the effect on the business would be acceptable.
  • Incompetence, dereliction of duty -> Take disciplinary action.
  • Inadequate training -> Institute immediate and/or long-term training of personnel.
  • Change in the environment -> Make improvements to adapt the process to the new conditions.
  • Previous management decision -> Check if the results of the decision were sought or unintended.
  • Error -> Fix the cause of the error.
  • Attack -> Evaluate whether the protection against the attack can be improved.
  • Accident -> Evaluate whether the protection against the accident can be improved.

What management practices become possible?
A side effect of an Information Security Management System (ISMS) lacking useful security metrics is that security management becomes centred in activities like Risk Assessment and Audit.  Risk Assessment considers assets, threats, vulnerabilities and impacts to get a picture of security and prioritise design and improvements while Audit checks the compliance of the actual information security management system with the documented management system with an externally defined management system or an external regulation. Risk Assessment and Audit are valuable, but there are more useful security management activities like monitor, test, design & improvement and optimisation that become possible with output metrics. Theses activities can be described as follows:

  • Monitor—Use metrics to watch processes outputs, detect abnormal conditions and assess the effect of changes in the process.
  • Test—Check if inputs to the process produce the expected outputs.
  • Improving -  Making changes in the process to make it more suitable for the purpose, or to reduce usage of resources.
  • Planning - Organising and forecasting the amount, assignment and milestones of tasks, resources, budget, deliverable and performance of a process.
  • Assessment -  How well the process matches the organisation's needs and compliance goals expressed as security objectives. How changes in the environment or management decisions in a process change the quality, performance and use of resources of the process; Whether bottlenecks or single points of failure exist; Points of diminishing returns; Benchmarking of processes between process instances and other organisations. Trends in quality, performance and efficiency.
  • Benefits realisation. Shows how achieving security objectives contributes to achieving business objectives, measures the value of the process for the organisation, or justifies the use of resources.

While audits can be performed without metrics, monitoring, testing, planning,  improvement and benefits realisation are not feasible without them.

What needs to be done?
S.M.A.R.T security managers need metrics that actually help them performing management activities.

While it is not necessary to drop goal metrics altogether, the day to day focus of information security management should be on security monitoring, testing, design & improvement and optimization using output metrics, which are the ones which will show what are the effect of management decisions, if things are getting worse or better, if processes work as designed, and if there are changes out of our direct control that cause abnormal conditions in security processes. All these activities are perfectly feasible using outputs metrics and control charts.

If you liked this article, consider taking advanced ISMS training online via Udemy