Return On Security Investment

The information security industry recognizes both the necessity and the difficulty of carrying out a quantitative evaluation of ROSI, return on security investment.

The main reason for investing in security measures is to avoid the cost of accidents, errors and attacks. Direct costs of an incident may include lost revenues, damages and property loss, or direct economic loss. The total cost can be considered to be the direct cost plus the cost of restoring the system to its original state before the incident. Some incidents can cost information, fines, or even human lives.

The indirect cost of an incident may include damage to a company’s public image, loss of client and shareholder confidence, cash-flow problems, breaches of contract and other legal liabilities, failure to meet social and moral obligations, and other costs.

Measuring Return

What do we know intuitively about the risk and cost of security measures? First, the relationship between the factors that affect risk - such as window of opportunity, value of the asset and its value to the attacker, combined assets, number of incidents and their cost, etc. - is quite complex.  We also know that when measures are implemented to reduce risk, the ease of using and managing systems also decreases, generating an indirect cost of the security measures.

How do we go from this intuitive understanding to quantitative information? There is some accumulated knowledge of the relationship between investment in security measures and their results. First, there is the Mayfield paradox, according to which the cost of universal access to a system and absolutely restricted access is infinite, with more acceptable costs corresponding to the intermediate cases.

An empirical study was also done by the CERT at Carnegie Mellon University, which states that the greater the expenditure on security measures, the smaller the effect of the measures on security. This means that after a reasonable investment has been made in security measures, doubling security spending will not make the system twice as secure.

The study that is most easily found on the Internet on this subject cites the formulas created during the implementation of an intrusion detection system by a team from the University of Idaho.

R: losses.
E: prevented losses
T: total cost of security measures

(R-E)+T= ALE

R-ALE = ROSI, therefore ROSI = E-T

The problem with this formula is that E is merely an estimate, and even more so if the measure involved is an IDS, which simply collects information on intrusions, which means that there is no cause-effect relationship between detecting an intrusion and preventing an incident. Combining this type of estimate with basing it on mathematical formulas is like combining magic with physics.

What problems do we face in calculating return on investment of security measures? The most important is the lack of concrete data, followed closely by a series of commonly accepted suppositions and half-truths, such as that risk always decreases as investment increases, and that the return on the investment is positive for all levels of investment.

Nobody invests in security measures to make money; they invest in them because they have no choice. Return on investment demonstrates that investing in security is profitable, in order to select the best security measures with a given budget, and to determine whether the budget allocated to security is sufficient to fulfill the business objectives, but not to demonstrate that companies make money off of the investment.

In general, and also from the point of view of return on investment, there are two types of security measures: measures to reduce vulnerability and measures to reduce impact.

  • Measures that reduce vulnerability barely reduce the impact when an incident does occur. These measures protect against a narrow range of threats. They are normally known as Preventive Measures. Some of these measures are firewalls, padlocks, and access control measures. One example of the narrowness of the protection range is the use of firewalls, which protect against access to unauthorized ports and addresses, but not against the spread of worms or spam.
  • Measures that reduce impact to very little to minimize vulnerability if an incident does occur. These measures protect against a broad range of threats and are commonly known as Corrective Measures. Examples of these measures include RAID disks, backup copies, and redundant communication links. One example of the range of protection is the use of backups, which do not prevent incidents, but do protect against effective information losses in the case of all types of physical and logical failures.

The profitability of both types of measures is different, as the rest of the article will show.

Preventive or Vulnerability-Reduction Measures

A reduction in vulnerability translates into a reduction in the number of incidents. Security measures that reduce vulnerability are therefore profitable when they prevent incidents for a value that is higher than the total cost of the measure during that investment period.

The following formula can be used:

ROSI = CTprevented / TCP

CT  = Cost of Threat = Number of Incidents * Per Incident Cost.
TCP = Total Cost of Protection

When ROSI > 1, the security measure is profitable.

Several approximations can be used to calculate the prevented cost. One takes the prevented cost into account as the cost of the threat in a period of time before and after the implementation of the security measure.

CTprevented = ( CTbefore – CTafter)

Calculating the cost of the threat as the number of incidents multiplied by the cost of each incident is an alternative with respect to the traditional calculation of the incident probability multiplied by the incident cost, provided that the number of incidents in the investment period is more than 1. To calculate a probability mathematically, the number of favorable cases and the number of possible cases must be known. Organizations rarely have information on possible cases (but not “favorable” cases) of incidents. It is impossible to calculate the probability without this information. However, it is relatively simple to determine the number of incidents that occur within a period of time and their cost.

For a known probability to be predictive, it is also necessary to have a large enough number of cases, and conditions must also remain the same. Taking into account the complexity of the behavior of attackers and the organizations that use information systems, it would be foolish to assume that conditions will remain constant. Calculating the cost of a threat using probability information is therefore unreliable in real conditions.

One significant advantage of calculating the cost of a threat as the product of the number of incidents and their unit cost is that this combines the cost of the incidents, the probability, and the total assets (since the number of incidents partly depends on the quantity of the total assets) into a single formula. To make a profitability calculation like this, real information on the incidents and their cost is required, and gathering this information generates an indirect cost of an organization’s security management. If this information is not available, the cost of the threats will have to be estimated to calculate the ROSI, but the value of the calculation result will be low as the estimate can always be changed to generate any desired result.

The profitability of a vulnerability reduction measure depends on the environment. For example, in an environment in which many incidents occur, a security measure will be more profitable than in the case of another environment in which they do not occur. While using a personal firewall on a PC connected to the Internet twenty-four hours a day may be profitable, using one on a private network not connected to the Internet would not. Investing in a reinforced door would be profitable in many regions of Colombia, but in certain rural areas of Canada, this investment would be a waste of money.

Sample profitability calculation:

  1. Two laptops out of a total of 50 are stolen in a year.
  2. The replacement cost of a laptop is 1800 euros.
  3. The following year, the company has 75 laptops.
  4. The laptops are protected with 60€ locks.
  5. The following year only one laptop is stolen.

ROSI = ( Rbefore – Rafter) / TCP

ROSI = ( ( 1800+Vi )*3 - (( 1800+Vi )*1+75*60) )/( 75*60 )

(The number of incidents is adjusted for the increase in the number of targets).

If a laptop was worth nothing (Vi=0), the security measure would not be profitable (ROSI < 1). In this example, the 60€ locks are profitable when a laptop costs more than 2700€, or when, based on historical information, the theft of 5 laptops can be expected for the  year in question.

Using this type of analysis, we could:

  • Use locks only on laptops with valuable information.
  • Calculate the maximum price of locks for all laptops (24€ when Iv=0).

Corrective or Impact-Reduction Measures

Since impact-reduction measures do not prevent incidents, the previous calculation cannot be applied. In the best case scenario, these measures are never used, while when there are two incidents which could result in the destruction of the protected assets, they are apparently worth twice the value of the assets. Now then, who would spend twice the value of an asset on security measures? Profitability of corrective measures cannot be measured. These measures are like insurance policies; they put a limit on the maximum loss suffered in the case of an incident.

What is important in the case of impact-reduction measures is the protection that you get for your money. The effectiveness of this protection can be measured, for example depending on the recovery time after an incident. Depending on their effectiveness, there are measures that range from backup copies (with some added cost) to fully redundant systems (which cost more than double).

One interesting alternative to calculating the ROSI of a specific security measure is to measure the ROSI of a set of measures – including detection, prevention, and impact reduction – that protect an asset. In this case, the total cost of protection (TCP) is calculated as the sum of the cost of all of the security measures, which the effort to obtain the information on the cost of the threats is practically identical.

Budget, cost, and selection of measures

The security budget should be at most equal to the annual loss expectancy (ALE) caused by attacks, errors, and accidents in information systems for a tax year. Otherwise, the measures are guaranteed not to be profitable. The graph below shows the expected losses as the area under the curve. To clarify the graph, it represents a company with enormous expected losses, of almost 25% of the value of the company. In the case of an actual company, legibility of the graph could be improved using logarithmic scales.

An evaluation of the cost of a security measure must take into account both the direct costs of the hardware, software, and implementation, as well as the indirect costs, which could include control of the measure by evaluating incidents, ethical hacking (attack simulation), audits, incident simulation, forensic analysis, and code audits.

Security measures are often chosen based on fear, uncertainty and doubt, or out of paranoia, to keep up with trends, or simply at random. However, the calculation of the profitability of security measures can help to select the best measures for a particular budget. Part of the budget must be allocated to the protection of critical assets using impact-reduction measures, and part to the protection of all of the assets using vulnerability-reduction measures and incident and intrusion detection measures.

Conclusions

The main conclusions that can be drawn from all of this are that:

  • To guarantee maximum effectiveness of an investment, it is necessary, and possible if the supporting data is available, to calculate the return on the investment of vulnerability-reduction measures.
  • In order to make real calculations, real information is needed regarding the cost of the incidents for a company or in comparable companies in the same sector.
  • Both incidents and security measures have indirect and direct costs that have to be taken into account when calculating profitability.

Download the Article

O-ISM3 challenges the CIA triad

The O-ISM3 Challenge, an equivalent in the consulting arena to the popular “Capture the Flag” competitions of the ethical hacking scene. In order to solve the Challenge participants have to solve a Use Case where they play the role of a consultant who is in charge of finding the security needs of a company. In order to solve the Challenge, participants have the option of using traditional concepts like confidentiality, integrity, availability, or new concepts like O-ISM3’s security objectives. The purpose of the Challenge is comparing the success solving the Use Case of participants using either option, which are mutually exclusive.

The Challenge will develop in 2017, and the winners will be published later in the year. Among those who pass the test, a prize of 500 pounds and a spot in an O-ISM3 course will be randomly assigned. Furthermore, the conclusions of the Challenge will be published, depending on the number of participants, their approach to solve the Challenge, and their comparative success solving the Challenge.

About O-ISM3
The Open Group Information Security Management Maturity Model (O-ISM3) Standard defines security processes for managing an enterprise’s Information Security Management System (ISMS). The O-ISM3 Standard places the onus on the business to define its required business security targets in its Security Policy, and then offers a set of security management processes from which the business selects which ones to deploy in a coherent ISMS. Each security control process in the ISMS then returns metrics to indicate how well that process is contributing towards achieving the business's security targets.
The O-ISM3 Standard metrics feedback is a major differentiating feature compared with other ISMS systems, because it enables the ISMS manager to present quantitative evidence (as opposed to qualitative subjective judgements) to show:

  • Which security control processes are revealing IT operational areas that are under-achieving regarding security targets
  • Which processes need to be tuned to improve performance to achieve or exceed critical targets
  • Which processes are not contributing sufficient to justify continuing to use them

The ISMS manager is thereby informed on which processes to retire and which to add to their ISMS. (More) Importantly, they are also informed with metrics that they can report as objective evidence to their CxO-level management on how well their ISMS is performing (based) on their security targets, and how effective their investment in security is.

The O-ISM3 Challenge

The current state of affairs is that most information security professionals use the concepts of Confidentiality, Integrity and Availability for audits, consulting projects, risk assessment and management, and development of new standards. These concepts present a series of problems that have yet to be solved:

  • They're incomplete. Some professionals supplement them with concepts like Possession, Utility(Parkerian Hexad), Risk, Authentication, Authorization, Audit, Non-Repudiation, Accountability and Assurance (CIA-cubed). This means performance and delivery vary greatly depending on what professional or company you use.
  • They're ambiguous. Many professionals and even published standards give different definitions of Confidentiality, Integrity and Availability. This adds more undesirable variance.
  • They are not operational. Consequently, Threats, Incidents, Vulnerability and Weakness among other concepts can't be reliably defined in terms of Confidentiality, Integrity and Availability reliably, increasing the ambiguity of definition. Even seasoned professionals don't agree on just what an Incident or a Vulnerability is. If you can't agree on what something IS, how can you manage it?
  • They don't have units of measurement. This makes it impossible to manage information security quantitatively. Bye, bye, optimization of resources.

The use of ambiguous, incomplete, not operational concepts without units of measurement has created a number of problems for information security management. Communication with between specialists and non-specialists in information security is difficult. Demonstrating the value of information security is difficult. Generally speaking, the use of these proxy concepts that don't add value makes information security management more difficult that it needs to be. Time is wasted, security projects that need funding don't get it, and trendy projects with little return get the green light. Luckily, change is possible.

Reasons to participate

  • To prove you skills as an information security consultant and win the prize.
  • To show off the worth of the professional certificate you hold and win the prize.
  • To prove that Confidentiality, Integrity and Availability are indeed fundamental an useful concepts.
  • To prove that Confidentiality, Integrity and Availability are neither necessary nor useful concepts.

The O-ISM3 Challenge
The O-ISM3 Challenge presents participants with a Use Case scenario (see below). The Use Case is a fictional Travel Agency in Madrid, Spain. Participants will act as information security consultant who need to determine what are the information security needs of the Travel Agency. Please note that there is a difference between finding out what the Travel Agency needs and what the Travel Agency might do regarding information security. If we were to compare the security practices of the Travel Agency with some standard, we could find out that the Travel Agency is not doing everything that a standard says can be done. There is a difference between doing everything that is standards state is possible, and everything that meets the needs of the business.

Determining the security needs of the Travel Agency will enable the consultant to determine the reasonable security measures to be applied, which are likely to be different, and cheaper, than all the security measures that could be taken.

In order to prove that they can successfully determine the security needs, the participants have to create a list of Questions to ask the Travel Agency managers or employees. This should be easy, since all the answers are included in the Use Case, and they will be listed individually in a spreadsheet that the participants will have to fill out.

Participants have a choice:

  • CIA Option: Questions can ONLY ask about Confidentiality, Integrity and Availability.
  • O-ISM3 Option : Questions can NOT ask about Confidentiality, Integrity or Availability.

Development of the Challenge

  • Participants can sign-up between the 28th of February and the 14th of March (CET Timezone).
  • To participate there is a registration fee of 5 euro payable using the"Register via PayPal" button below.
  • Communication between the participants and Inovement will be only through the mail address used to register and learn@inovement.es
  • The challenge is limited to 500 participants.
  • Every participant will receive a copy of a spreadsheet with the Answers and a registration number within 24h of registration.
  • In the spreadsheet, the participant must choose either the CIA Option or the O-ISM3 Option, and mention any information security professional certifications they hold.
  • Each participant must fill in Questions that would give the Answers provided and send the spreadsheet attached in an e-mail to learn@inovement.es, with their registration number in the subject of the mail.
  • Questions will be evaluated. A single ambiguous question, where the answer is not a perfect match causes the participant to FAIL the challenge (Example of FAIL: "What is the Integrity of the Data?· for the answer "Data needs to be kept for 5 years", example of PASS: "When does the system needs to be Available? for the answer: "Between 8 and 5 Monday to Friday") .
  • Grammatically, logically inconsistent questions, not in the English language, or longer than 255 characters or inclusion of malicious macros will result in a FAIL.
  • For the CIA Option failure to use at least one of Confidentiality, Integrity and Availability (or Confidential, Integer, Available) in any question will result in a FAIL.
  • For the O-ISM3 Option using any of Confidentiality, Integrity or Availability in any question will result in a FAIL.
  • All answers must be sent by midnight on 16 of January (CET Timezone).
  • Those who have PASSED will be announced on 24 of January. The best ones will be published.
  • The results of the entry evaluations may not be contested.
  • The names of the participants will not be published without their permission.
  • A 500€ prize and a free seat for an O-ISM3 Course will be awarded. The winner will be chosen from those who PASSED and will be given to the one with the most significant digits (two at least) in their registration number matching the first prize of Loteria Nacional of 21st of January (a random number). The prize will be paid within seven days via Paypal to the same account used to register.
  • The number of participants both the total number and those with correct answers for both Options will be published, along with an analysis of the results comparing how the CIA Option and the O-ISM3 Option fared, and what that means for the use of concepts like Confidentiality, Integrity and Availability.

If you want to get posted on the development of the Challenge, follow us on Twitter, or request begin added to our mail list.

Results
If the number of participants who PASS using the CIA Option is higher than the number of participants who PASS using the O-ISM3 Option, this will indicate that Confidentiality, Integrity and Availability are valid concepts in order to analyze the security needs of a company like Ambiguous SL. If the number of participants who PASS using the O-ISM3 Option is higher than the number of participants who PASS using the CIA Option this will indicate that Confidentiality, Integrity and Availability are not particularly useful in order to analyze the security needs of a company like Ambiguous SL. As a secondary result, we will compare how different information security professional certificate holders performed.

The Use Case
Ambiguous SL is a travel agency located in Madrid, Spain. Their business is selling retail travel packages both online and through their offices, which are street level on a main street. The most important system they own and operate is the Package Sales System, which they use for advertising, sales, and bookings. This system interfaces with the Amadeus GDS system (checking availability and bookings), with VISA (payments), and with an equivalent system of a Moroccan partner (MTravel), as it is a popular destination for Spanish tourist and represents a significant part of the company's business.

The owner of Ambiguous SL has put Myrna in charge of IT, among other responsibilities. Myrna has hired you do find out which security measures (controls or processes) would provide the highest return of investment for Ambiguous SL. Myrna will take care of implementation. Your first (and only) task is to make an assessment of Ambiguous SL security needs.

Myrna has named Ignatius as the project manager for the Package Sales System. He is an employee of the company (Confederacy SL) that develops and maintains the Package Sales System for Ambiguous SL.

The Package Sales System functionality is as follows (please note this a Use Case, so it is simpler than a real life case):

  • Create, Modify and Delete Travel Packages.
  • Sell Travel Packages both online and at the office.
  • Receive feedback from customers and the public in general.
  • Send Travel Package offers to subscribers.
  • Manage Claims and Issues.

A high level view of the Package Sales System Database reveals the following data resources:

  • Travel Package Archive
  • Sales Archive
  • Feedback Archive
  • Offers Archive
  • Claims, Feedback and Incidences Archive

The following list of actions can be performed on each data resource:

  • Travel Package Archive: Create, Update, Retire, Publish, Unpublish.
  • Sales Archive: Book, Release, Sell, Refund, Update.
  • Feedback Archive: Create, Update, Close.
  • Offers Archive: Create, Update, Retire, Publish.
  • Claims, Feedback and Incidences Archive: Create, Update, Close
  • Sales Statistics Report Archive: Create, Close

There are certain requirements about who can do what, and where they can do it:

  • Only the sales manager can Create, Update and Publish Travel Packages.
  • Each salesperson can only view the personal information of his or her own clients.
  • Only the sales manager and the person assigned to Feedback and Claims can view the personal information of all clients.
  • Only the owner of the company can access the Sales Statistics Report.
  • Only the sales manager can create Offers

Certain parts of the Package Sales System are licensed, namely the Operating System, Application Server and Database.
As the company and systems are located in Spain, the Package Sales System needs to comply with the Spanish Privacy Law (LODP). Since the Package Sales System manages VISA payments, it needs to comply with PCI-DSS.

Some of the users of the Package Sales System are employees of Ambiguous SL, some are temps from Adecco. The administrators of the Package Sales System are employees of Confederacy SL. The general public of Spain is a user and they can purchase Travel Packages through the application. The application does not serve the public of countries other than Spain. Persons under the age of 18 can ask for feedback and signup for offers, but they can't purchase Travel Packages.

The system is located in a properly conditioned room inside the office. The system interfaces with Internet via a high speed fiber optic connection. The system interfaces with the interconnected systems and users via mail, file transfers and a VPN that connects directly with the MTravel network.

The system is expected to work 24x7, but because of maintenance stoppages of no more than one hour per week during no business hours (from 9 to 5 from Tuesday to Sunday) are acceptable. The longest time that the system can be offline during business hours is 2 hours, because sales can be performed with TPV and handwritten notes can partially replace the use of the system. In case of a major malfunction of the system, it would be acceptable to lose one day of data, since most data could be reconstructed checking with VISA, Amadeus and Mtravel. It is understood that all "live" transactions would be lost in case of an incident.
Data needs to be archived for 5 years in order to meet tax regulations. After ten years data should be deleted permanently, as customer behaviour changes over time and data is no longer useful for Business Intelligence.
Sales representatives and customers sometimes make mistakes entering data. This is acceptable as long as there is no more than one percent of the records contain innacurate information.

In order to create an account in the Package Sales System, potential clients can login using Facebook or create an account linked to their email address. They can unlink or delete the account at any time, but that does not delete any data in the database if they have purchased a Travel Package, even if they cancelled the purchase. In order to create an account in the Package Sales System, the Sales Manager sends an email to the Administrator. The email states what functions the user should be able to perform. The general public doesn't need an account to provide feedback or sign up for the Offers newsletter.

Customer who lose their passwords to the Package Sales Systemcan request a new one and a link will be sent to their email address. Users who lose their password to access the Package Sales System need to physically visit the Administrator, who resets the password and give it to them in a written note.

As some Offers expire at midnight, the Package Sales System should prevent customers from purchasing Travel Packages after they have expired, even by a few seconds.

There is a development environment, that Confederacy SL maintains in their own data center, and a pre-production environment, at Ambiguous SL office.
The current administrator is subscribed to email lists that notify him of security updates. The Administrator has configured the system using security guidelines found on Internet for every component. Security patches have not been applied since a patch caused a half day downtime.
The Administrator changes about once every six months.

The system has no malware protection.
The domain has been registered with Piensasolutions.es. The digital certificates used by the system are from Thawte. No one has been assigned with the responsibility to manage the domain or the certificates.
The systems logs all the sales activity, but not any other activity.
There is no Firewall. The internet connectivity provider (Telefonica) provides a service that is supposed to provide "clean" traffic.

No part of the Package Sales System is located in a publicly accesible location.
No part of the Package Sales System is accesible via Mobile application, but there are plans to incorporate a solution for this.
No part of the Package Sales System is exposed to extreme environmental conditions.

Pages