Shadow IT in the era of artificial intelligence: how to deal with shadow AI?
With readily available AI-based tools, the phenomenon of "shadow IT" in companies is booming. What are the risks of this "shadow AI" and how should we respond to them?
Markéta Melenová, Jan Kubíček
Shadow IT in the era of AI
In an era of massive safeguards against the spread of covid, working from home has become far more common than at any other time in history. This has accelerated the shift to the use of cloud-based solutions. Users have also since become much more likely to process company data on your own devices .
The situation where a company's IT department loses track of which applications and solutions are being used in the company has come to be known as Shadow IT. This phenomenon is associated with a number of risks - safe handling of corporate and personal data, security of their transmission, etc.
The trend of using in-house ICT tools has not disappeared with the latest anti-pandemic measures. On the contrary: a new boom in shadow IT has come with easily accessible AI-based tools. Nowadays, the most common content generation tools are LLM-based tools called genAI. The pressing topic of shadow IT is once again emerging - this time in the form of "shadow AI".
What is shadow AI
Like Shadow AI refers to readily available generative AI tools (such as public LLMs) in the corporate environment that are used without prior approval. IT managers lose sight of this and the more difficult it can be to defend against cyber risks.
Meanwhile, IT managers' approaches to AI tools vary from "let's disable it and maybe gradually enable some" to "a new version of the model is out, why don't we have it in production yet?" Which one is more correct?
How to introduce AI into an organisation?
When looking for an answer, we always start from the existing the risk profile of a particular organisation. The right approach is one that is appropriate to the risks, business objectives, capabilities (financial, technical, business) and current corporate know-how in AI.
A balanced approach therefore means think about the risks, set up a basic governance process and then start with the first use cases. It would be a shame to forgo greater work efficiency where AI tools can serve well. (The second thing is how to deploy the AI tools you choose and how to use them effectively.)
A friendlier attitude towards AI tools is typical for start-ups and organisations where the use of new technologies is part of their DNA and can be part of their corporate strategy.
In the case of organisations with a high risk profile However, it may be prudent to hard-code publicly available AI tools first prohibit and make inaccessible (not just relying on internal guidelines and governance process).
Typically, this applies to financial institutions and organisations in highly regulated or conservative sectors such as healthcare or banking (although even among them there are enthusiastic early adopters). Why the caution? To prevent internal information of such companies from appearing in responses generated by AI tools outside the organization.
The IT department can then gradually enable the AI tools that the organization's employees need and that are in accordance with the company's Acceptable Use Policy for AI (Acceptable Use AI Policy).
How AI tools work
AI solutions are usually based on a certain model of reality. Users enter relatively accurate and incomplete data and expect some output. The correctness of the output depends on the correctness and completeness of our query and the correctness of the model.
Also plays a role the user's ability to correctly interpret the results of the AI tool. It is not just about how well the user knows the subject. The level of understanding of how the model arrives at its outputs is also important.
In everyday life, we ask a colleague: "How did you come up with this?" and we expect an answer that allows us to check the correctness of the thought process, possibly also authoritativeness of the sourcefrom which my colleague draws. It's not so easy with the model.
It should also be remembered that users' knowledge of reality is limited. We rarely remember all the details and facts, and we rarely know the exact mechanisms that govern, for example, what happens on the stock market. Nor do we rarely answer the same question in the same way as in the past.
The answers are often just guesses (with greater or lesser accuracy) and we shape them on the basis of our own experience. And with generative models that work with probabilities, it is similar: their the answers (and therefore the factual correctness) to the same query may differ significantly.
What are the risks of using Shadow AI?
An organisation can deploy AI tools on its own platformwhere they will be under its full control (or you can pay for a premium version with higher data protection guarantees).
The counterpart is represented by publicly and freely available online toolsthat users use uncontrollably directly in Web browser. Today's shadow AI most often takes this form. This creates several risks.
Reputational risk
The organisation could present in its public outputs incorrect information. In particular, there are risks where misleading or erroneous information can cause health, financial and other harm to people.
Nor is it without risk internal use the outputs of free AI tools - especially when this provides the basis for decision-making.
Risks to information security
To get useful answers from an AI tool, the user often has to enter a prompt, which may include sensitive information. No one knows or is watching at this moment, what happens to them afterwards.
This may include trade secrets, industrial designs, personal data, financial information, pricing information, etc. This way, the data gets outside the control of the organisation, which is unacceptable especially for organisations that handle highly sensitive information.
There are, of course, technical solutions preventing data exfiltration. However, the most stringent may not be financially optimal for some organisations and acceptable to users.
Wondering how to ensure security in the cloud? Eight basic principles we wroteand here.
Useful tool for shadow AI: AWS GenAI Scoping Matrix
In relation to Shadow AI, the framework developed by AWS has proven itself to us. It is called GenAI Scoping Matrix and distinguishes five ways (or more precisely "scopes") of implementing genAI tools.
Each method of implementation involves a different range of risks and therefore appropriate security measures. The scopes model of risk and security management (scopes) facilitates communication between developers and operational security staff.
Includes:
- 2 types of scopes with a solution acquired by purchase (or available for free - see below)
- and 3 types of scope, which are developed according to the wishes and needs of the customer.
AWS GenAI Scoping Matrix and five ways to implement AI tools (source: Securing Generative AI: Introduction to the Generative AI Security Scoping Matrix)
Tools acquired by purchase
- Scope 1: Includes the use of licensed and publicly available AI solutions (these are the tools used in Shadow AI).
An example is the lively use of ChatGPT, Copilot in Edge (formerly Bing Chat), etc. Here, the user (and unknowingly, his employer) usually relies on the security measures of the tool provider. But has anyone checked the terms and conditions? Has anyone thought about how to use the tool correctly and safely?
Users should only feed publicly available data into this type of AI tool. Inputting sensitive data, data subject to trade secrets or personal data is undesirable.
- Scope 2A: AI solutions are acquired by an organisation as software as a service (SaaS).
These applications give customers greater control over the fate of data embedded in the tool, including the ability to opt-out of automatic data sharing with the tool provider.
(Microsoft 365 Copilot, for example, inherits existing DLP policies. Enterprise licenses can respect document classification and user-specific access rights. So if a user doesn't have access to something, neither does Copilot.) For security management, the determinants are license conditions, compliance with them, and user discipline in handling data (some data may not be entered into AI tools).
Shadow AI: Can your provider use your users' data?
We already know that Shadow AI falls under AWS scope 1. dictated by the terms of the contract providers and thus, what data the user enters into the tool. The contractual terms of each instrument are specific.
Example 1: Gemini
Google's (formerly Bard) Gemini user terms state that conversations and other information (e.g. geolocation or tool usage information) are automatically stored and used by Google to further develop and improve these technologies.
If the user does not wish to use the data in this way, it is necessary to configure the tool properly and has opted out of this functionality (opt-out). In contrast, for users under the age of 18, there is an opt-in mode - data is not collected unless the user actively opts in.
Example 2: Copilot
Microsoft's Copilot is commonly available in the Edge browser. Here, too, the Statement by Microsoft on privacy policy the inquisitive user learns that his data can be used to develop and train Microsoft AI models.
In addition, there is a link to FAQ pages of the web applicationace Copilotwhere, on the contrary, the provider's statement says that for users from the European Economic Area (EEA), the model does not use user input for training until further notice.
Because the Czech Republic is part of the EEA (where the legal framework restricts such use of data), data of Czech private users (whether logged in or not) are not used for training models.
If you're wondering whether and how data can travel between continents, read more in this Encyclopedia articlethe cloud.
Shadow AI and terms and conditions
Let's look at another risk: when using Microsoft Copilot from a private user account, the employer cannot assumethat the user has actively checked the terms and conditions. At the same time, he cannot check the situation because he does not know about the use of Shadow AI tools.
The user would have to look up the terms and conditions themselves and interpret them correctly (which requires a high level of expertise and experience in law, digital governance and data security). In addition, they would have to check the contractual arrangements proactively and continuously, as AI tool providers make changes to terms and conditions, often unilaterally and without notice (it is not as convenient as, for example, with banks, which notify us in advance of a change in contractual terms).
The trained eye of a lawyer, for example, will also notice:
- that these restrictions apply to users "in EEA countries"but not to the user "from EEA countries" (as in the case of MS Copilot),
- that they concern the protection of personal data, but not the protection of data in general - for example, data embedded in or produced by a model (here we are talking about Gemini),
- that opting out of the use of data for model training does not automatically mean opting out of further use of that data for other purposes (again, Microsoft Copilot).
So how to approach Shadow AI?
In the case of Shadow AI, it is most often the user (and not the company) who holds the greatest degree of control over how risky their use of the genAI tool will be. Only the user can set up the tool appropriately.
What is too risky for organisation A may be acceptable for organisation B. Therefore it is more appropriate to allow employees access to AI tools, but Controlled. Risk treatment depends primarily on the organisation's business objectives and strategies, as well as its risk profile.
And since this is our professional topic, we're happy to help you with risk management and the implementation of AI tools.
ABOUT THE AUTHORS
Jan Kubicek
Legal IT Consultant | LinkedIn
Jan will support you wherever compliance is concerned. He has a legal background (regulatory, compliance and data protection in banking). He often scrutinises contracts to ensure that the numerous regulatory requirements are met. When analyzing contracts, he likes to anticipate what could happen... and figure out how to treat potential risks.
Markéta Melenová
Compliance Consultant | LinkedAnd
Markéta is a compliance consultant, with a particular focus on DORA, AI Act and NIS2. She specializes in industry standards in the field of information security and artificial intelligence and their implementation in highly regulated international entities. In doing so, he leverages his previous experience in data analytics, standards development (legislative, industry and internal) and working on the development of International Telecommunication Union standards.
