SURVEY: Top 3 Artificial Intelligence (AI) Commercial Vendors

Getting Right or Wrong?

Commercial Developments

---

A review of open-source industrial sector literature reveals a predominant use of internal data to detect cyber-threats.  This analysis includes Columbus’ (2019a) article on major cybersecurity device manufacturers. Three of the significant cybersecurity device providers were analyzed as part of this review; they include Vectra AI, Darktrace, and Cisco Systems.  Two use heterogeneous data in some capacity for threat detection, while one was determined to rely upon internal data alone.

In the case of Vectra AI, there were apparent disconnects between what they describe in their open-source vice information released by its senior leadership (Lunden, 2018; Sheu, 2019). This portion of the review is supportive of the view that industry is only sparsely using external data in its cybersecurity detection and prevention devices.

Vectra AI.

In 2016, Vectra AI raised $36M to increase its research and development into creating an AI-based solution. In 2018, Vectra deployed its solution, Cognito ®, an AI-based solution, using ANNs as a basis of its technology (Lunden, 2018). The Chief Executive Officer (CEO) declared that while there were other players in the marketplace, to include Darktrace and Cisco, Vectra AI’s primary solution is Cognito®. Vectra AI describes its solution as not a “bolt-on,” after-the-fact, solution as compared to its market competitors (para. 9).

---

ESSAY: Can the Human “Poet” Bring Value to Predictive Analysis

---

A review of additional open-source information from Vectra’s threat detection materials describes the use of the government’s STIX format as an external data component of Cognito®. The Cognito® solution imports “local and industry-specific indicators of compromise consisting of malicious IP address, domains, URLs [Uniform Resource Locators] or user agents expressed in STIX” (Vectra AI, 2017, para. 3). Vectra states it does incorporate STIX data as part of its solution (M. Teranen, personal communication, October 15, 2019); however, the question remains whether there is any quantified value with its employment either based on its internal solution or in comparison with its competitors.

Notably, there is a disconnect with Vectra’s declared use of heterogeneous data and its company information. Kevin Sheu (2019), Vice President, Product Marketing, Vectra AI, illustrates one of the study’s contentions that commercial companies remain more invested in internal datasets with minimal consideration of external or heterogeneous data non-resident to the targeted IT environment.  Sheu (2019) states that “metadata and file capture deliver much better investigative value—it is easier and faster to find things” (para. 14). Contentiously, intrusion detection is more than what is more comfortable or faster; it should be a matter of accuracy of identifying and defeating cyber-threats against organizational IT environments.

Vectra AI’s 2019 White Paper, the data science behind Cognito AI threat detection models, reflectsSheu’s (2019) perspective of how Vectra’s Cognito® automated threat detection and response platform address various types and kinds of data (Vectra AI, 2019). The Vectra solution describes its solution as reliant on “local learning techniques” that may be inferred as reliance on data that is discovered resident to the IT environment (p. 6). It appears from this contradiction that Vectra AI’s solution has a general disregard for the importance of using external data. Vectra AI (2019) suggests that “[w]hile global learning is critical; some things can only be learned based on local experiences” (p.5). (While it is not the objective of this study to summarily dismiss conflicting points of view from within the same company, it does suggest a lack of consistency on the perceived value of data heterogeneity within the cybersecurity defense market.)

Darktrace.

Vectra’s CEO identified Darktrace as a market competitor, and a review of online material shows no use of external data (Lunden, 2018). It received a 2019 award as the “Best Application of AI in the Enterprise” and uses a non-specified ML solution that may or may not include ANNs. The core technology is its Darktrace Antigena®, which identifies “normal ‘pattern of life’ [activities] for every user, device, and associated peer group in the business” (Darktrace, 2019b, p. 1). While Darktrace’s solution describes the use of data as either based upon previous data or current data, it does not demonstrate an integrated inclusion of external or heterogeneous data.

Cisco Systems.

A review of their 2019 White Paper highlights Cisco’s use of a “network analytics engine” that uses AI/ML in its intent-based networking solution (Cisco, 2019, p. 3). Specifically, Cisco leverages its global access to data as a mechanism for the usage of different data sources. “By feeding large quantities of data and diverse categories of data, [Cisco] can use ML to calculate very accurately…statistical outcomes” (p. 5).

As noted in Figure 8, Cisco demonstrates that specific organizational network audit logs are captured, anonymized, and processed by Cisco’s AI/ML processes—note the transition from the upper left of diverse data, to your network, to a worldwide data platform that anonymizes customer data to the AI/ML predictive outputs of the Cisco DNA Center. Cisco’s solution leverages the synergies of ML and diverse data to identify threats more effectively, see Figure 8.

Figure 1. Cisco AI network analytics. Reprinted from AI and machine learning primer: A technology overview for business decision-makers, by Cisco, 2019. Retrieved from  https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/digital-network-architecture/nb-06-cisco-dna-ai-ml-primer-cte-en.html?oid=sowen018703 

While this is not an exhaustive review of every AI-based cyber-intrusion solution, it provides insight and direction from the primary market leaders in the cyber-defense marketplace. The reviewer’s position is not that these companies are solely ignoring heterogeneous data, but that they are understating its use and importance to protecting vital IT infrastructures. Furthermore, there was no quantified or comparative suggestion of how one commercial solution is measurably better than another. There remains the need to identify metrics that can assist cyber-defenders and corporate decision-makers in fighting cyber-attacks.

Selected References

Anagnostopoulos, C. (2016). Quality-optimized predictive analytics. Applied Intelligence, 45(4), 1034–1046. Retrieved from
http://franklin.captechu.edu:2123/10.1007/s10489-016-0807-x

Anitha, P., & Patil, M. M. (2018). A review of data analytics for supply chain management: A case study. International Journal of Information Engineering and Electronic Business, 10(5), 30–39. Retrieved from http://franklin.captechu.edu:2123/10.5815/ijieeb.2018.05.05

Carse, B., & Oreland, J. (2000). Evolution and learning in neural networks: Dynamic correlation, relearning and thresholding. Adaptive Behavior, 8(3–4), 297–311. Retrieved from https://doi.org/10.1177/105971230000800305

Chimento Jr, J. J. (2019). Toward an Understanding of Using High Entropic Digital Communication Techniques in Cybersecurity Decision Making (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 13897847)

Chesney, R. (2015, October 13). Cybersecurity in 1989: Looking back at Cliff Stoll’s classic The Cuckoo’s Egg [Blog post]. Lawfare. Retrieved from https://www.lawfareblog.com/cybersecurity-1989-looking-back-cliff-stolls-classic-cuckoos-egg

Cisco. (2019). Artificial intelligence/machine learning for intent-based networking – primer [White paper]. Cisco. Retrieved from https://www.cisco.com/c/en/us/solutions/collateral/enterprise-networks/digital-network-architecture/nb-06-cisco-dna-ai-ml-primer-cte-en.pdf

Clarke, R. A., & Knake, R. K. (2014). Cyber war. New York, NY: Harper Collins.

Columbus, L. (2019a, June 16). Top 10 cybersecurity companies to watch in 2019. Forbes. Retrieved from https://www.forbes.com/sites/louiscolumbus/2019/06/16/top-10-cybersecurity-companies-to-watch-in-2019/#4b683b696022

Columbus, L. (2019c, January 23). Data scientist leads 50 best jobs in America for 2019 according to Glassdoor. Forbes. Retrieved from https://www.forbes.com/sites/louiscolumbus/2019/01/23/data-scientist-leads-50-best-jobs-in-america-for-2019-according-to-glassdoor/#457226e77474

Committee on National Security Systems. (2015, April 6). CNSS glossary. CNSS. Retrieved from https://rmf.org/wp-content/uploads/2017/10/CNSSI-4009.pdf

Corrigan, J. (2019, September 4). Pentagon, NSA laying groundwork for AI-powered cyber defenses. Nextgov. Retrieved from https://www.nextgov.com/cybersecurity/2019/09/pentagon-nsa-laying-groundwork-ai-powered-cyber-defenses/159649/

Cummins, E. (2018, August 29). Another AI winter could usher in a dark period for artificial intelligence. Popular Science. Retrieved from https://www.popsci.com/ai-winter-artificial-intelligence/

Cybersecurity and Infrastructure Security Agency Act of 2018, Pub. L. 115-278, 132 Stat. 4186, codified as amended at 6 U.S.C. §§651–674.

Darktrace. (2019a, June 12). Darktrace wins ‘best application of AI in the enterprise’ at the AIconics awards [Press release]. Darktrace. Retrieved from https://www.darktrace.com/en/press/2019/292/

Darktrace. (2019b). Autonomous response: Threat report 2019. Darktrace. Retrieved from https://www.darktrace.com/en/resources/wp-cyber-ai-response-threat-report-2019.pdf?utm_source=darktrace&utm_medium=technology

Darktrace. (2019c). Machine learning in the age of cyber AI: A review of machine learning approaches for cyber security and Darktrace’s underlying technology [White paper]. Darktrace. Retrieved from https://www.darktrace.com/en/resources/wp-machine-learning.pdf?utm_source=darktrace&utm _medium=technology

Davis, B., Whitfield, C., & Anwar, M. (2018, August). Ethical and Privacy Considerations in Cybersecurity. In 2018 16th Annual Conference on Privacy, Security and Trust (PST) (pp. 1–2). IEEE. doi:10.1109/PST.2018.8514188

Denning, D. (2017, August 18). Tracing the sources of today’s Russian cyberthreat. Scientific America. Retrieved from https://www.scientificamerican.com/article/tracing-the-sources-of-today-rsquo-s-russian-cyberthreat/

Department of Justice. (n.d.). Privacy act of 1974. DOJ. Retrieved from https://www.justice.gov/opcl/privacy-act-1974

Devlin, H. (2019, July 5). Hacked forensic firm pays ransom after malware attack. The Guardian. Retrieved from https://www.theguardian.com/science/2019/jul/05/eurofins-ransomware-attack-hacked-forensic-provider-pays-ransom

Director of National Intelligence. (2018, October). 2018 Information Sharing Environment. DNI. Retrieved from https://www.dni.gov/files/documents/FOIA/2018_Information_Sharing_Environment_Annual_Report.pdf

Djekic, M. (2019, July 5). Cyber security analytic purposes [Blog post]. Cyber Defense Magazine. Retrieved from https://www.cyberdefensemagazine.com/cyber-security-analytics-purposes/

Elder, J. (2013, June). It is a mistake to…lack relevant data [White paper]. Charlottesville, VA: Elder Research.

European Union (E.U.). (n.d.). GDPR key changes. EU. Retrieved from https://eugdpr.org/the-regulation/

Ezeife, C. I., Dong, J., & Aggarwal, A. K. (2008). SensorWebIDS: A web mining intrusion detection system. International Journal of Web Information Systems, 4(1), 97–120. Retrieved from http://franklin.captechu.edu:2123/10.1108/17440080810865648

Fang, X., Xu, M., Xu, S., & Zhao, P. (2019). A deep learning framework for predicting cyber attacks rates. EURASIP Journal on Information Security, 2019(1), 1–11. Retrieved from http://franklin.captechu.edu:2123/10.1186/s13635-019-0090-6

Forcepoint. (n.d.). What is spoofing? Spoofing defined, explained, and explored [Blog post]. Forcepoint. Retrieved from https://www.forcepoint.com/cyber-edu/spoofing

Funke, D., & Benkleman, S. (2019, May 23). How Russia’s disinformation strategy is evolving. Poynter. Retrieved from https://www.poynter.org/fact-checking/2019/how-russias-disinformation-strategy-is-evolving/

Garamone, J. (2018, February 13). Cyber tops list of threats to U.S. director of national intelligence says. Defense.gov.  Retrieved from https://www.defense.gov/Newsroom/News/Article/Article/1440838/cyber-tops-list-of-threats-to-us-director-of-national-intelligence-says/

Galloppo, G., & Previati, D. (2014). A review of methods for combining internal and external data. The Journal of Operational Risk, 9(4), 83–103. Retrieved from https://franklin.captechu.edu:2074/docview/1648312043?accountid=44888

Guccione, D. (2019, July 4). What is the dark web? How to access it and what you’ll find. CSO Online. Retrieved from https://www.csoonline.com/article/3249765/what-is-the-dark-web-how-to-access-it-and-what-youll-find.html

Gupta, D., & Rani, R. (2018). A study of big data evolution and research challenges. Journal of
            Information Science, 1–19. Retrieved from https://doi.org/10.1177/0165551518789880

Gwynne, P. (2013). Predicting the progress of technology. Research Technology Management, 56(4), 2–3. Retrieved from https://franklin.captechu.edu:2074/docview/1458287915?accountid=44888

Haider, M. (2015). Getting Started with Data Science: Making Sense of Data with Analytics. New York, NY: IBM Press.

Hair, Joe F., Jr. (2007). Knowledge creation in marketing: The role of predictive analytics. European Business Review, 19(4), 303–315. Retrieved from doi:http://franklin.captechu.edu:2123/10.1108/09555340710760134

Halladay, S. D. (2013). Using predictive analytics to improve decisionmaking. The Journal of Equipment Lease Financing (Online), 31(2), 1–6. Retrieved from https://franklin.captechu.edu:2074/docview/1413251757?accountid=44888

Harvard Business Review (HBR). (2018). Artificial intelligence and machine learning driving tangible value for business [Briefing paper].

Hayden, L. (2010). IT security metrics: A practical framework for measuring security & protecting data. New York: McGraw Hill.

Hensel, N. (2016). The defense industry: Tradeoffs between fiscal constraints and national security challenges. Business Economics, 51(2), 111–122. Retrieved from http://franklin.captechu.edu:2123/10.1057/be.2016.16

Hu, Z., Gnatyuk, V., Sydorenko, V., Odarchenko, R., & Gnatyuk, S. (2017). Method for cyberincidents network-centric monitoring in critical information infrastructure. International Journal of Computer Network and Information Security, 9(6), 30. Retrieved from http://franklin.captechu.edu:2123/10.5815/ijcnis.2017.06.04

Homeland Security Systems Engineering and Development Institute. (n.d.). Threat intelligence sharing using STIX and TAXII. Secure360. Retrieved from https://secure360.org/wp-content/uploads/2014/05/Threat-Intelligence-Sharing-using-STIX-and-TAXII.pdf

Hubbard, D. (2009a, February 11). I am concerned about the CI, median and normal distribution [Blog post]. Hubbard Decision Research. Retrieved from https://hubbardresearch.com/i-am-concerned-about-the-ci-median-and-normal-distribution/

Hubbard, D. (2009b). The failure of risk management: Why it’s broken and how to fix it. Hoboken, NJ: John Wiley & Sons.

Hubbard, D., & Seiersen, R. (2016). How to measure anything in cybersecurity risk. Hoboken, NJ: John Wiley & Sons.

Jahan, A., & Alam, M. A. (2017). Intrusion detection systems based on artificial intelligence. International Journal of Advanced Research in Computer Science, 8(5) Retrieved from https://franklin.captechu.edu:2074/docview/1912629399?accountid=44888

Jasim, Y. A. (2018). Improving intrusion detection systems using artificial neural networks. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 7(1), 49–65. Retrieved from http://franklin.captechu.edu:2123/10.14201/ADCAIJ2018714965

Johns, A. (n.d.). What is internal data? – Definition & sources [Blog post]. Study.com. Retrieved from https://study.com/academy/lesson/what-is-internal-data-definition-sources.html

K, P. C., & Shivakumar, B. L. (2014). A review of trends and technologies in business analytics. International Journal of Advanced Research in Computer Science, 5(8), 225–229.  Retrieved from https://franklin.captechu.edu:2074/docview/1658426584?accountid=44888

Kaplan, F. (2016). Dark territory: The secret history of cyber war. New York, NY: Simon & Schuster.

Koerner, B. (2016, October 23). Inside the cyberattack that shocked the US government. Wired. Retrieved from https://www.wired.com/2016/10/inside-cyberattack-shocked-us-government/

Kulp, P. (2019). Active cyber defense: A case study on responses to cyberattacks (Doctoral dissertation). Available from ProQuest Dissertations & Theses Global. (2247845452). Retrieved from https://franklin.captechu.edu:2074/docview/2247845452?accountid=44888

Kumar, P., & Verma, R. S. (2017). A review on recent advances & future trends of security in honeypot. International Journal of Advanced Research in Computer Science, 8(3). Retrieved from https://franklin.captechu.edu:2074/docview/1901458306?accountid=44888

Lau, C.H. (2019, January 10). 5 steps of a data science project lifecycle. Towards Data Science. Retrieved from https://towardsdatascience.com/5-steps-of-a-data-science-project-lifecycle-26c50372b492

Lee, A. J. (2015). Predictive analytics: The new tool to combat fraud, waste and abuse. The Journal of Government Financial Management, 64(2), 12–16. Retrieved from https://franklin.captechu.edu:2074/docview/1711620017?accountid=44888

Levy, N. (2019, July 26). Amazon R&D and infrastructure spending spike as tech giant staffs up on talent. GeekWire. Retrieved from https://www.geekwire.com/2019/amazon-rd-infrastructure-spending-spikes-tech-giant-staffs-technical-talent/

Lighthill, J. (1972). Artificial intelligence: A general survey. Chilton computing. Retrieved from http://www.chilton-computing.org.uk/inf/literature/reports/lighthill_report/p001.htm

Lis, P., & Mendel, J. (2019). Cyberattacks on critical infrastructure: An economic perspective 1. Economics and Business Review, 5(2), 24–47. Retrieved from doi:http://franklin.captechu.edu:2123/10.18559/ebr.2019.2.2

Lunden, I. (2018, February 21). Vectra raises $36M for its AI-based approach to cybersecurity intrusion detection. Techcrunch. Retrieved from https://techcrunch.com/2018/02/21/vectra-raises-36m-for-its-ai-based-approach-to-cybersecurity-intrusion-detection/

Loy, J. (2019). Neural network projects with Python. Birmingham, UK: Packt.

Lyngaas, S. (2019, April 23). Someone is spoofing big bank IP addresses-possibly to embarrass security vendors. Cyberscoop. Retrieved from https://www.cyberscoop.com/spoofed-bank-ip-address-greynoise-andrew-morris-bank-of-america/

Maloney, D. (2017, October 19). Books you should read: The cuckoo’s egg. Hackaday. Retrieved from https://hackaday.com/2017/10/19/books-you-should-read-the-cuckoos-egg/

Mandiant. (2013, February 18). APT1: Exposing one of china’s cyber espionage units. Fireeye. Retrieved from https://www.fireeye.com/content/dam/fireeye-www/services/pdfs/mandiant-apt1-report.pdf

Marz, N., & Warren, J. (2015). Big Data: Principles and best practices of scalable real-time data systems. New York: Manning Publications Co.

McGibony. (2015, June 30). Be a data detective [White paper]. Charlottesville, VA: Elder Research.

Mitchell, B. (2019, October 7). Computer ports: Usage & role in networking: Discover the wide range of computer connections. Lifewire. Retrieved from https://www.lifewire.com/computer-port-usage-817366

MITRE. (2012). Standardizing Cyber Threat Intelligence Information with the Structured Threat Information eXpression (STIX). MITRE. Retrieved from  https://www.mitre.org/sites/default/files/publications/stix.pdf

MITRE. (n.d.). Structured threat information expression (STIX). MITRE. Retrieved from https://makingsecuritymeasurable.mitre.org/docs/stix-intro-handout.pdf

Nagrecha, S., & Chawla, N. V. (2016). Quantifying decision making for data science: From data acquisition to modeling. EPJ Data Science, 5(1), 1–16. Retrieved from doi:http://franklin.captechu.edu:2123/10.1140/epjds/s13688-016-0089-x

National Commission on Terrorist Attacks upon the United States. (2004). The 9/11 Commission report: Final report of the National Commission on Terrorist Attacks upon the United States. Authorized ed., 1^st ed. New York: Norton.

National Association of State Chief Information Officers. (2016). Advanced cyber analytics: Risk intelligence for state government. NASCIO. Retrieved from https://www.nascio.org/Portals/0/Publications/Documents/2016/NASCIO_AdvancedCyberAnalytics_FINAL_4.18.16.pdf

National Institute of Standards and Technology. (2018, June 7). Protecting Controlled Unclassified Information in Nonfederal Systems and Organizations. NIST. Retrieved from https://csrc.nist.gov/publications/detail/sp/800-171/rev-1/final

National Institute of Standards and Technology. (2015, January 22). Security and Privacy Controls for Federal Information Systems and Organizations. NIST. Retrieved from https://csrc.nist.gov/publications/detail/sp/800-53/rev-4/final

Naylor, B. (2016, June 6). One year after OPM data breach, what has the government learned? National Public Radio. Retrieved from https://www.npr.org/sections/alltechconsidered/2016/06/06/480968999/one-year-after-opm-data-breach-what-has-the-government-learned

Ng, C., Pan, L., Xiang, Y. (2018). Honeypot frameworks and their applications: A new framework. Singapore: Springer.

Nield, T. (2019, February 7). Is another AI winter coming? Hackernoon. Retrieved from https://hackernoon.com/is-another-ai-winter-coming-ac552669e58c

Novetta. (n.d.). Know your network: Arm your analysts [Blog post]. Novetta. Retrieved from https://www.novetta.com/products/novetta-cyber-analytics/

Olson, P. (2012). We are anonymous: Inside the hacker world of LulzSec, Anonymous, and the global cyber insurgency. New York, NY: Little, Brown, and Company.

Oltramari, A., & Kott, A. (2018). Towards a reconceptualisation of cyber risk: An empirical and ontological study. Journal of Information Warfare, 17(1), 4–73. Retrieved from https://franklin.captechu.edu:2074/docview/2059071274?accountid=44888

Orgera, S. (2019, August 5). How to use TOR browser for anonymous web browsing. Lifewire. Retrieved from https://www.lifewire.com/tor-browser-tutorial-4103599

Paliwal, D. (2013). Mathematical analysis of problem statements: Artificial intelligence. International Journal of Advanced Research in Computer Science, 4(3). Retrieved from https://franklin.captechu.edu:2074/docview/1443744864?accountid=44888

Paliwal, D., Vaya, D., Khandelwal, S. (2013). Mathematical analysis of problem statements: Artificial intelligence. International Journal of Advanced Research in Computer Science, 4(3). Retrieved from https://franklin.captechu.edu:2074/docview/1443744864?accountid=44888

Palo Alto. (n.d.). What is an endpoint [Blog post]? Palo Alto. Retrieved from https://www.paloaltonetworks.com/cyberpedia/what-is-an-endpoint

Peasland, P. (2017, October 9). What problems can data science solve? Medium. Retrieved from https://medium.com/@philippa.peasland_69295/what-problems-can-data-science-solve-46f0b744da5a

Pham, T. M. (2018). Exploring strategies for incorporating population-level external information in multiple imputation of missing data (Doctoral dissertation). Retrieved from EBSCO Open Dissertations. http://search.ebscohost.com/login.aspx?direct=true&db=ddu&AN=788945D34A68B6CD&site=ehost-live

Project Management Skills. (2010, September 5). Qualitative risk analysis and assessment. Retrieved from Project Management Skills: https://www.project-management-skills.com/qualitative-risk-analysis.html

Prusak, L. (2010, October 7). What can’t be measured. Harvard Business Review. Retrieved from https://hbr.org/2010/10/what-cant-be-measured

Radziwill, N. M., & Benton, M. C. (2017). Cybersecurity cost of quality: Managing the costs of cybersecurity risk management. ArXiv. Retrieved from https://arxiv.org/ftp/arxiv/papers/1707/1707.02653.pdf

Rashid, T. (2016). Make your own neural network. Amazon Digital Services, LLC: Tariq Rashid.

Ray, T. (2019, September 12). No, there will be no AI winter. Forbes. Retrieved from https://www.forbes.com/sites/tiernanray/2019/09/12/no-there-will-be-no-ai-winter/#5815439d46a5

Riemer, K., & Seidel, S. (2013). Design and design research as contextual practice [Editorial]. Information Systems and eBusiness Management, 11(3), 331–334. Retrieved from doi:http://franklin.captechu.edu:2123/10.1007/s10257-013-0223-2

Rodriguez, L., & Da Cunha, C. (2018). Impacts of big data analytics and absorptive capacity on sustainable supply chain innovation: A conceptual framework. LogForum, 14(2), 151–161. Retrieved from doi:http://franklin.captechu.edu:2123/10.17270/J.LOG.267

RSA. (2016, February 5). The role of TOR in cybercrime [Blog post]. RSA. Retrieved from https://www.rsa.com/en-us/blog/2016-02/role-tor-cybercrime

Russo, M. (2019). Critiques paper: Cybersecurity and data science join forces. Unpublished manuscript.

Russo, M. (2018). The Risk Reporting Matrix is a Threat to Advancing the Principle of Risk Management. Unpublished manuscript.

Schroer, A. (2019, April 10). 25 Companies merging AI and cybersecurity to keep us safe and sound. Built-In. Retrieved from https://builtin.com/artificial-intelligence/artificial-intelligence-cybersecurity

Schuchman, S. (2019a, May 12). History of the first AI winter. Toward Data Science. Retrieved from https://towardsdatascience.com/history-of-the-first-ai-winter-6f8c2186f80b

Schuchman, S. (2019b, May 12). History of the second AI winter. Toward Data Science. Retrieved from https://towardsdatascience.com/history-of-the-second-ai-winter-406f18789d45

Shaikh, F. (2016, October 3). Deep learning guide: Introduction to implementing neural networks using TensorFlow in Python. Analytics Vidhya. Retrieved from https://www.analyticsvidhya.com/blog/2016/10/an-introduction-to-implementing-neural-networks-using-tensorflow/

Shankland, S. (2019, October 25). Google’s quantum supremacy is only a first taste of computing revolution. Cnet. Retrieved from https://www.cnet.com/news/google-quantum-supremacy-only-first-taste-of-computing-revolution/

Sheu, K. (2019, April 30). Why network metadata is just right for your data lake [Blog post]. Vectra AI. Retrieved from https://www.vectra.ai/blogpost/why-network-metadata-is-just-right-for-your-data-lake

Siegel, E. (2016). Predictive analytics: The power to predict who will click, buy, lie, or die. Hoboken, NJ: John Wiley & Sons.

Silver, N. (2012). The signal and the noise: Why so many predictions fail–but some don’t. New York, NY: Penguin.

Smith, A. (2019, July 4). 7 fundamental steps to complete a data project [Blog post]. Dataiku. Retrieved from https://blog.dataiku.com/2019/07/04/fundamental-steps-data-project-success

Soni, S., & Vyas, O. P. (2010). Using associative classifiers for predictive analysis in health care data mining. International Journal of Computer Applications, 4(5), 33–37. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.206.4564&rep=rep1&type=pdf

Somers, M. J., & Casal, J. C. (2009). Using artificial neural networks to model nonlinearity: The case of the job satisfaction–job performance relationship. Organizational Research Methods, 12(3), 403–417. Retrieved from https://doi.org/10.1177/1094428107309326

Spitzner, L. (2003). Honeypots: tracking hackers. Boston, MA: Addison-Wesley.

Starks, T. (2019, July 9). Cyber incidents were expensive in 2018. Politico. Retrieved from https://www.politico.com/newsletters/morning-cybersecurity/2019/07/09/cyber-incidents-were-expensive-in-2018-675243

Starr, B. (2015, July 31). Military still dealing with cyberattack ‘mess.’ CNN. Retrieved from https://www.cnn.com/2015/07/31/politics/defense-department-computer-intrusion-email-server/index.html

Stoll, C. (2005). The cuckoo’s egg: Tracking a spy through the maze of computer espionage. New York, NY: Simon and Schuster.

Stoll, C. (1988, May). Stalking the wily hacker. Communication of the ACM, 31(5), 484–497. Retrieved from http://mars.umhb.edu/~wgt/cisc4370/wilyhacker.pdf

Strand, M., Wangler, B., & Niklasson, M. (2004). External data incorporation into data warehouses: an exploratory study of identification and usage practices in banking organizations. In Proceedings of the CAiSE Forum at the 16th International Conference on Advanced Information Systems Engineering (CAiSE’04) (pp. 103–112). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.99.8300&rep=rep1&type=pdf

Sukamolson, S. (2007). Fundamentals of quantitative research. Language Institute Chulalongkorn University, 1, 2–3. Retrieved from https://s3.amazonaws.com/ academia.edu.documents/ 48405070/Suphat_Sukamolson.pdf?response-content-disposition=inline%3B%20filename%3DFundamentals_of_quantitative_research.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A %2F20191024%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191024T184218Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=a31eb071d79bde2117a1fd1bf742b7eb152f770dd17f501a114ab4257ede2dbd

Tadjdeh, Y. (2019, September 6). Defense logistics agency embraces AI. National Defense. Retrieved from  https://www.nationaldefensemagazine.org/articles/2019/9/6/defense-logistics-agency-embraces-ai

Taylor, M. (2017). Neural network math: A visual introduction for beginners. Vancouver, Canada: Blue Windmill Media.

Taylor-Sakyi, K. K. (2016). Big data: Understanding big data. arXiv. Retrieved from https://arxiv.org/ftp/arxiv/papers/1601/1601.04602.pdf

Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The art and science of prediction. New York, NY: Crown Publishers.

Thurber, M. (2018, April 6). What is data wrangling and why does it take so long [Blog post]. Elder Research. Retrieved from https://www.elderresearch.com/blog/what-is-data-wrangling

Toth, E. (2017, December 21). Which are the most scanned ports [Blog post]? Bitninja. Retrieved from https://bitninja.io/blog/2017/12/21/port-scanning-which-are-most-scanned-ports

Trullen, J., & Bartunek, J. M. (2007). What a design approach offers to organization development. The Journal of Applied Behavioral Science, 43(1), 23–40. Retrieved from https://franklin.captechu.edu:2074/docview/236352484?accountid=44888

Udemy. (n.d.). Machine learning: Build neural networks in 77 lines of code. Retrieved from https://www.udemy.com/machine-learning-build-a-neural-network-in-77-lines-of-code/learn/lecture/13179726#overview

Under Secretary of Defense for Acquisition, Technology, and Logistics. (2006, August). Risk management guide for DOD acquisition. Retrieved from https://www.acq.osd.mil/damir/documents/DAES_2006_RISK_GUIDE.pdf

United States Health and Human Services. (n.d.). Health information privacy. HHS. Retrieved from https://www.hhs.gov/hipaa/index.html

University of Melbourne. (2018, May). Risk assessment methodology. Retrieved from  https://safety.unimelb.edu.au/__data/assets/pdf_file/0007/1716712/health-and-safety-risk-assessment-methodology.pdf

van Veen, F. (2016). A mostly complete chart of neural networks. Asimov Institute. Retrieved from https://www.asimovinstitute.org/wp-content/uploads/2016/09/neuralnetworks.png

Vectra AI. (2019). The data science behind Cognito AI threat detection models [White paper]. Vectra AI. Retrieved from https://www.vectra.ai/download/the-data-science-behind-cognito-ai-threat-detection-models

Waddell, D. (2015, May 14). Global information security workforce study [Press release]. ISC2. Retrieved from https://www.isc2.org/-/media/ B7E003F79E1D4043A0E74A57D5B6F33E.ashx

Walsh, K. (n.d.). Audit log best practices for information security [Blog post]. Reciprocity. Retrieved from https://reciprocitylabs.com/audit-log-best-practices-for-information-security/

Walters, T. (n.d.). Incorporating external data into the data warehouse. SAS. Retrieved from https://support.sas.com/resources/papers/proceedings/proceedings/sugi22/DATAWARE/PAPER116.PDF

Warwick, K. (2010). Cultured neural networks. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 224(2), 109–111. Retrieved from https://doi.org/10.1243/09596518JSCE916

Waterman, S. (2017, November 15). Experts say government’s information sharing program is all take and no give. Cyberscoop. Retrieved from https://www.cyberscoop.com/dhs-ais-program-house-homeland-committee/

Watkins, L. A., & Hurley, J. S. (2015). Cyber maturity as measured by scientific-based risk metrics. Journal of Information Warfare, 14(3), 57–65. Retrieved from https://franklin.captechu.edu:2074/docview/1967314091?accountid=44888

Weng, B. (2017). Application of machine learning techniques for stock market prediction (Doctoral dissertation). Retrieved from EBSCO Open Dissertations. http://search.ebscohost.com/login.aspx?direct=true&db=ddu&AN=DE0B8B4C2E217AE3&site=ehost-live

Wilner, A. S. (2018). Cybersecurity and its discontents: Artificial intelligence, the Internet of Things, and digital misinformation. International Journal, 73(2), 308–316. Retrieved from  https://doi.org/10.1177/0020702018782496

Wilson, H.J., & Daugherty, P.R. (2018, July-August). Collaborative intelligence: Humans and AI are joining forces. HBR. Retrieved from https://hbr.org/2018/07/collaborative-intelligence-humans-and-ai-are-joining-forces

Yu-Zhong, C., Zi-Gang Huang, Xu, S., & Ying-Cheng, L. (2015). Spatiotemporal patterns and predictability of cyberattacks. PLoS One, 10(5). Retrieved from https://franklin.captechu.edu:2074/docview/1685181563?accountid=44888

Zhan, Z., Xu, M., & Xu, S. (2013). Characterizing honeypot-captured cyber attacks: Statistical framework and case study. IEEE Transactions on Information Forensics and Security, 8(11), pp.1775–1789. doi: 10.1109/TIFS.2013.2279800

Zuech, R., Khoshgoftaar, T. M., & Wald, R. (2015). Intrusion detection and big heterogeneous data: A survey. Journal of Big Data, 2(1), 1–41. Retrieved from http://franklin.captechu.edu:2123/10.1186/s40537-015-0013-4