Feeding Customer Data into AI Systems: When It's Allowed and How to Build It

The short answer

Yes, under certain conditions. The GDPR doesn't prohibit AI — it sets requirements for how data is processed.

Here's a more interesting observation: in the projects we see, things almost never fail because of the legal situation. They fail because no one in the company can say which data in the system in question actually qualifies as personal data. As long as that question is open, any discussion of legal bases is purely theoretical.

So the rest of this article follows two parallel tracks: the four legal checks, and the four architectural patterns that reduce the risk on a technical level before it ever needs a legal assessment.

‍

Part 1: the four checks before the first prompt

Legal basis: Article 6 GDPR

Every processing of personal data needs a legal basis. For most use cases at mid-sized companies, three come into question: performance of a contract, a legal obligation, or legitimate interest. Consent is the least popular basis, because it can be withdrawn at any time, which ends the processing.

With legitimate interest, you need to document a balancing test: your interest against the interests of the data subjects. This balancing test has to exist in writing before the system goes live, not afterward.

Purpose limitation: does your original purpose also cover AI?

This is where most projects stumble. Data was collected for a specific purpose, say, contract processing. Feeding it into an AI system that recognizes patterns or generates suggestions can count as a different purpose. In that case, you need either a compatibility assessment under Article 6(4) or a new legal basis.

In practice, that means: don't just ask whether you're allowed to have the data. Ask whether you're allowed to use it for this new purpose.

Data minimization: what the task actually needs

A language model tasked with categorizing support requests needs the text of the request. It doesn't need the name, the customer number, or the billing history. Every field you leave out is a field that can't leak.

This isn't legal advice, it's engineering. Data minimization is the one measure that still works even when everything else fails.

Transparency: is it in your privacy policy?

Data subjects have a right to know how their data is processed. If an AI system has been added to the mix, that belongs in the disclosures under Articles 13 and 14, and the processing activity belongs in the record of processing activities under Article 30.

‍

Part 2: the provider

As soon as an external model is involved, its provider is a data processor. Three points need to be settled before the first data record leaves the building:

Data processing agreement under Article 28. Without one, the processing is unlawful — on your side.

Third-country transfer. If the provider is based outside the EU — which is the norm for the major models — you need the EU Standard Contractual Clauses and a Transfer Impact Assessment, i.e. a documented evaluation of the level of protection in the destination country. For US providers, there's also the certification under the EU-US Data Privacy Framework, which is publicly viewable.

Training on your inputs. In personal and standard tiers, using inputs to improve the model is often the default; in business and enterprise tiers, it's contractually excluded. The difference lies in the contract, not a toggle in the settings that you should rely on. Ask to be shown the relevant clause in the contract.

‍

Part 3: the four architectural patterns

This is the part most articles on this topic skip. The legal situation is the same for everyone. The difference comes down to architecture.

‍

1. Reduce before the call

Between your system and the model sits a layer that replaces names, customer numbers, addresses, contract numbers, and free-text fields containing personal data before the call goes out, and reverses the placeholders in the response. The model works with "Customer A"; your user sees the real name.

This takes development effort, and it's the single most effective lever available. Important: a substitution that can be reversed is pseudonymization, not anonymization. Pseudonymous data remains personal data and remains subject to the GDPR. Anyone who confuses the two is building on a false assumption.

2. Look up instead of training

Training your own model on your data permanently bakes that data into the model. Getting it back out again — say, after a deletion request — is practically impossible from a technical standpoint.

The alternative: the model stays unchanged, and the relevant content is looked up from your database at runtime and passed to the model as context. Delete the record, and it's gone on the next call. That's the difference between a deletion process that actually works and one that's merely claimed to.

3. Check permissions at runtime

An internal assistant that accesses documents has to apply the same permissions as the source system. Otherwise it'll answer a case worker's questions about salaries because the file happened to sit in a folder nobody checked.

This isn't a hypothetical problem — it's the most common finding when an assistant is pointed at a file system that's grown organically over years. Permissions have to apply on every single call, not just at setup.

4. No automated decisions without a human

Article 22 GDPR gives data subjects the right not to be subject to a decision based solely on automated processing that produces legal effects or similarly significantly affects them. Credit decisions, applicant pre-screening, terminations.

The practical way out is rarely a legal justification and almost always a design choice: the system proposes, a human decides and can change the suggestion. That means the decision isn't made solely by automated processing. What matters is that this review is real. A checkbox that gets rubber-stamped 99 percent of the time doesn't count.

Part 4: two things that come next

Data protection impact assessment. Required under Article 35 when the processing is likely to result in a high risk. For new technologies, large-scale processing of special categories of data, or systematic evaluation of personal aspects, it's the rule rather than the exception. The supervisory authorities maintain lists of processing operations where it's mandatory.

EU AI Act. As of August 2, 2026, most of the regulation applies. It comes on top of the GDPR; it doesn't replace anything. In practice, what matters first is the allocation of roles: whoever develops an AI system and makes it available under their own name is a provider. Whoever deploys it within their own operations is a deployer. The role determines the obligations, and the role isn't negotiable.

‍

The uncomfortable part

The most common response to this article will be: "But we anonymize."

In most cases, that isn't true. Replacing names while date of birth, postal code, and contract start date stay in the record isn't anonymization. A combination of just a few attributes is regularly enough to re-identify a person. And free-text fields where employees have left notes over the years almost always contain personal references that no field list captures.

A record only counts as anonymous once the person can no longer be identified with reasonable effort. Everything short of that is pseudonymization, and pseudonymization stays under the GDPR. The difference isn't semantic — it decides whether the entire regulation applies or not.

‍

Anyone who honestly pseudonymizes and calls it that is better off than someone who claims to anonymize and gets proven wrong in an audit.

‍

Frequently asked questions

Are we allowed to enter personal data into ChatGPT?

Not on the personal tier. There, a data processing agreement is typically missing, and inputs can be used to improve the model. In business and enterprise tiers, both of those are covered. What matters isn't the tool, it's the contract behind it.

What's the difference between anonymized and pseudonymized?

Anonymized means: the person can no longer be identified with reasonable effort, and the GDPR no longer applies. Pseudonymized means: the link can be restored with additional information, and the GDPR still applies. Most procedures that get called anonymization are actually pseudonymization.

Do we need our customers' consent to use AI?

Not necessarily. Consent is just one of several legal bases and often the least practical one, because it's revocable. Contract performance or legitimate interest frequently work just as well. The choice of basis should be documented before the system goes live.

Do we need to carry out a data protection impact assessment?

If the processing is likely to result in a high risk to the rights of data subjects, yes. That's frequently the case with AI systems handling personal data. Your data protection officer or your supervisory authority can give you the list of processing operations where it's mandatory.

Can we use customer data to train our own model?

Legally possible, practically risky. Once data is trained into a model, it's nearly impossible to remove again, so a deletion request ends up going nowhere. Runtime lookup achieves the same goal in almost all cases and stays deletable.

Where does the data need to be stored?

Within the EU is the simplest option. Outside it, it's permissible if Standard Contractual Clauses, a Transfer Impact Assessment, and, where relevant, a Data Privacy Framework certification are in place. "In the cloud" is not a location — get the region and data center location confirmed in writing.

Who is liable if the AI system outputs something wrong?

Toward data subjects, the controller remains responsible — that is, the company that determines the processing. That's one of the reasons for the fourth pattern above: where a human decides, there's a decision that can be reviewed.

What we take care of

We build the layer between your system and the model: reducing data before the call, lookup instead of training, runtime permission checks, and logging that lets you prove, if it ever comes to that, what went where and when.

What we don't do is the legal assessment of your specific case. For that, we work with our external data protection officer, heyData GmbH, and for your case, your own data protection officer or a specialist attorney is the right contact.

How we ourselves handle AI tools, which ones are in use, and what's contractually behind them is laid out openly on our AI Compliance and Data Protection page.

Why these questions need to be settled this early, and what else causes AI projects to fail, is covered in Why AI Projects Fail.

‍

Note: This article is a professional perspective from a software developer's point of view, not legal advice.

On how this was created: Our content is produced with the support of AI, but every text goes through a double human review before it's published.

‍

Sources

• Regulation (EU) 2016/679, General Data Protection Regulation, in particular Articles 5, 6, 13, 14, 22, 28, 30, 32, 35, and 44 to 49.
• Regulation (EU) 2024/1689, Artificial Intelligence Act.
• Recital 26 GDPR on the identifiability of a person.
• Supervisory authority lists of mandatory data protection impact assessments.
• EU-US Data Privacy Framework, public participant list of the U.S. Department of Commerce.