Most postmortems on failed production ML models don’t actually end at the model. They end somewhere upstream, often in a spreadsheet or an old Slack thread, where someone tried to turn a fuzzy piece of business logic into a column.
“Active customer” meant three different things to three different teams. A “days since last contact” feature got computed one way in training and another way in serving. A label leaked through a feature that, on paper, looked perfectly reasonable. None of these are model problems. They’re feature problems, and they’re far more common and far more expensive than the architecture debates that usually get the attention.
Large language models are starting to show up in this part of the workflow. The early instinct on a lot of teams is to let them generate features directly: paste in some column names, ask for “20 features for churn prediction,” take what comes back. That’s the wrong job for an LLM here. Feature engineering is exactly the kind of task where the model has no access to your data, your label definition, your serving infrastructure or your business rules. Those four things are usually where the real risk lives.
A more useful way to think about things: an LLM can be a genuinely good feature reviewer, but it’s a bad feature factory. It’s strong at locating ambiguity, drafting language and pressure-testing definitions someone else has already grounded in real data. It’s weak, and occasionally risky, as a source of truth about what a feature actually means or whether it’s safe to use. So let’s walk through an effective workflow for keeping it on the right side of that line.
How to Use LLMs in Feature Engineering
Most production ML models fail due to upstream feature problems like fuzzy business logic, poor documentation and data leakage rather than model architecture issues. Using LLMs as feature factories to generate features directly without access to real data or pipeline constraints fails in production. Instead, deploy them as feature reviewers in a human-guided workflow.
- Brainstorm candidate feature hypotheses from domain knowledge.
- Draft structured, plain-language feature definitions.
- Pressure-test definitions for temporal data leakage and ambiguity.
- Translate technical feature logic for non-technical stakeholders.
Where Feature Work Actually Breaks
Before getting into the workflow, it helps to be specific about the failure modes since they’re what the LLM is being asked to catch.
Fuzzy Business Logic
A feature like is_high_value_customer often exists as a vague consensus rather than a written rule. Different stakeholders carry different versions of it in their heads, and the first time it gets written into code, someone’s mental model wins by accident.
Weak Documentation
Feature stores and old notebooks accumulate columns named things like flag_2, ratio_v3 or adj_score, with no record of what they encode, what window they cover or why anyone built them. Six months later, nobody, including the original author, can say with confidence what the feature represents.
Leakage-Prone Transformations
This is the most expensive failure mode of the three. A feature computed from data that wouldn’t actually be available at prediction time, like a refund flag set only after a decision was made or an aggregate that quietly includes the outcome window, produces a model that looks great in validation and falls apart the moment it meets reality.
These are knowledge problems and process problems before they’re statistical ones. That’s exactly the kind of problem an LLM, used carefully, can actually help with.
A 4-Part LLM-Assisted Workflow
1. Turning Domain Knowledge Into Candidate Features
Most feature work doesn’t start with data. It starts with a domain expert’s intuition about what matters. An LLM is useful here as a structured brainstorming partner, turning loose domain knowledge into a list of testable hypotheses, which a human and the actual data then filter down.
A reasonable prompt pattern looks something like this:
You are helping a data science team brainstorm candidate features for a
[prediction task] model. Domain context: [2-3 sentences of business
context]. Available raw data: [list of tables/columns at a high level].
Propose candidate features grouped by category (behavioral, temporal,
aggregate, relational). For each, state the hypothesis for why it might
be predictive, in one sentence.The output is a list of hypotheses, not features ready to ship. The value is in coverage. An LLM will reliably suggest categories (recency, frequency, ratio, trend, interaction) that a single domain expert under time constraints might skip. The value isn’t in correctness, which still has to be checked against the real schema and the real label definition.
2. Drafting Feature Definitions and Naming Conventions
Once a feature seems worth building, the next failure point is documentation, specifically the gap between what a feature is called and what it actually computes. This is a place where LLMs do genuinely solid, low-risk work because rewording precise logic into clear language is squarely in their wheelhouse.
A useful pattern is to ask for a structured definition rather than freeform prose:
Given this feature logic: [paste the actual SQL/pandas logic],
produce a definition card with these fields:
- Name (snake_case, following pattern: <entity>_<metric>_<window>)
- Plain-language definition (one sentence, no jargon)
- Computation window
- Data sources used
- Known edge cases (nulls, new entities, boundary dates)
- Refresh cadenceFeeding it the actual logic, not a description of the logic, is what keeps this step grounded. The model is rewording something true into something clear rather than guessing at something true. That’s the difference between a documentation aid and a hallucination risk.
3. Pressure Testing Definitions for Ambiguity and Leakage Risk
This is the highest value use case, and the one worth building a habit around. Before a feature goes anywhere near a training pipeline, run its definition past the LLM as an adversarial reviewer, one specifically prompted to look for leakage and ambiguity rather than to approve the feature.
Review this feature for a model predicting [target, with its exact
definition and prediction timing]. Feature definition:
[paste definition card from step two].
List, specifically:
1. Any way this feature could include information unavailable at
prediction time
2. Any ambiguity in the definition that could cause two engineers to
implement it differently
3. Any dependency on a field that is itself derived from the outcome
being predicted
4. Edge cases not addressed in the definitionTake a feature like num_support_tickets_resolved for a churn model predicting 30 days out. That should raise a flag almost immediately: Tickets resolved after a customer has already churned or resolved as part of a retention call that only happens because someone flagged the churn risk can leak the outcome backward into the feature. A reviewer prompted this way will usually catch the shape of that problem, the temporal misalignment between the feature window and the label window, even without seeing the underlying data because that’s a structural question rather than a factual one.
The output here is a checklist of risks worth verifying, not a verdict. Someone with access to the actual pipeline still has to confirm which of the flagged risks are real.
4. Explaining Engineered Features to Stakeholders
The last mile of feature work is usually a conversation with a product manager, a compliance reviewer or a business stakeholder who needs to understand and sign off on what a model is using without reading the transformation code. This is a strong, low-risk use case for the same reason step two is. Turning precision into generality is what these models are good at.
Explain this feature to a non-technical stakeholder in three or four sentences.
No jargon. Include: what it measures, why it's relevant to
[business problem], and one limitation worth knowing.
Feature definition: [paste definition card]Requiring a stated limitation in every explanation is a small habit worth keeping. It pushes back against the tendency for LLM-generated explanations to sound more complete and more authoritative than the underlying feature actually is.
The Guardrails That Keep This Safe
None of the above works without a few firm rules about what the LLM doesn’t get to be the final word on.
Treat every output above (the candidate list, the definition card, the risk list, the explanation) as a draft for a human to accept, edit or reject. None of it ships unreviewed. Whether a feature actually leaks is a question about the pipeline and the data, and it gets answered by checking timestamps and running point in time correctness tests, not by how convincing the LLM’s reasoning sounds on the page.
Step one only works because it’s seeded with real domain context from a person who understands the business; an LLM brainstorming without that input just produces plausible sounding noise. A feature’s actual value gets decided by held out model evaluation and, ideally, a champion-versus-challenger comparison in production, not by how cleanly its definition reads.
And when an LLM is used to draft a definition or flag a risk, it’s worth keeping a record of the prompt, the output and what a human did with it afterward. That matters for audits, and it matters for catching the same mistake before it repeats three sprints later.
LLMs Should Review, Not Build
Asking an LLM to “generate the features” treats it like it has access to information it doesn’t have: your label timing, your serving constraints, your data quality issues. Asking it to review, draft and explain features that a human has already grounded in real data treats it like what it actually is, a fast, well-read collaborator with no access to your production system and no authority to approve anything on its own.
Used that way, the time saved isn’t really in modeling. It’s in the unglamorous work that actually breaks production systems: writing things down clearly, asking whether something could leak before deployment instead of after, and giving stakeholders an honest, plain language account of what a feature does and doesn’t capture. That’s a smaller claim than “LLMs can build your features for you,” but it’s the one that still holds up once the model is in production.
