Every data opportunity is a nail

The big tech approach to AI sees every data opportunity as a nail for an LLM to hammer.

I've watched that habit spread for years. Send it all up, train on it, work out the rest later. It's become so normal that many people no longer notice it's a choice. It is one. It's just usually someone else's.

The default

In October 2025, researchers at Stanford HAI (opens in a new tab) reviewed the privacy policies of six major US AI developers: Amazon, Anthropic, Google, Meta, Microsoft and OpenAI. All six "employ users' chat data by default to train their models."

By default is the important part. Opting out is the user's job, where it's possible at all.

The researchers found more. Some of the developers keep chat data indefinitely. Some allow people to review transcripts. And they point out that a chatbot can draw inferences from harmless input. Ask for heart-friendly recipes, in their example, and the system may decide you fit a classification as a health-vulnerable individual. The Stanford team's recommendation was straightforward: "affirmative opt-in for model training." In other words, ask first.

Business accounts are different, which tells you something

Here's the detail I find most revealing. Business plans often work the other way.

Anthropic's page for financial advisors, for example, says: "On Team and Enterprise plans, we don't train our models... on your data by default." Computerworld (opens in a new tab) makes the same point across vendors: business and enterprise accounts often come with stronger terms.

That's good news for firms that pay for those plans and read the terms. But think about what it says. The vendors already know how to not train on your data. They offer it. It's simply a feature of the more expensive tier.

Privacy you have to pay for is still privacy, and I'm glad it exists. It's also a reminder that the default was a business decision, not a technical necessity.

Default settings are a decision someone else made

For a regulated firm, the question isn't only "is our plan set not to train?" It's broader: what is our data for?

Client records exist to serve clients. Patient information exists to care for patients. Case files exist to support a case. Each comes with obligations about who may see it, how long it's kept and what it may be used for. "Improving a vendor's general-purpose model" is rarely on that list.

So the first decision belongs to people at the firm. What is this data for, and what isn't it for? Write it down. Then pick tools and terms that fit the answer, rather than adjusting the answer to fit the tool.

Not every problem needs an LLM

This is where the hammer habit does the most damage. Once every data opportunity looks like a nail, every problem gets the same tool, whether it fits or not.

Large language models are remarkable at some things. They're also expensive to run, hard to explain and hungry for data. For a lot of work in regulated firms, something smaller does the job better.

Take a hypothetical. A firm wants to sort incoming documents into a dozen categories and route them to the right team. One option is to send every document to a large external model. Another is a set of clear rules written with the people who handle those documents every day, plus a small model for the tricky cases, and a person for anything the system isn't sure about. The second option may be less exciting. It's also cheaper, easier to explain to an auditor, and it keeps the documents where they are.

Sometimes the right tool is a large model. Sometimes it's a small one, a well-written rule or a knowledgeable person. Often it's a combination. Back in 2019 I wrote that it's always valuable to call a thing a thing. Calling a problem by its real name is usually the first step to picking the right tool for it.

The test I'd suggest is simple. Before data goes into any model, ask three questions. Does this problem actually need this much data? Does it need this kind of model? And who decided?

Data has a purpose

None of this is anti-AI. I've spent about 12 years in AI and enterprise data, and I'm optimistic about what it can do. But the firms that get the most from it will be the ones that decide what their data is for before a default setting decides for them.

That's not a job for a vendor's terms of service, and it's not a job for a model. Data has a purpose. People should set it.

A hammer is a fine tool. It's a bad strategy.