AI Mini-Series Part II: Garbage In, Hallucination Out: Choosing What an AI Is Allowed to Know

Part 2 of a series on building and testing an AI layer for our systematic framework. This one is about the data the model is allowed to think about, and the one rule we will not bend.

In the first piece of this series, I described the AI layer we are building as a second reader of the market: an instrument pointed at the same Five Forces our deterministic strategies already read, but built to take in information a price rule was never designed to hold. This piece is about the least glamorous part of that work, and the part I am most convinced matters most. It is the data the model is allowed to think about.

A language model will produce a confident, fluent reading of whatever you put in front of it. That is its great strength, and it is its central danger. Give it good information and it reasons well. Give it bad information and it reasons just as well, and just as confidently, over something that is wrong. Fluency is the one thing an AI model is never short of. So the first discipline, before any of the parts people find interesting, is deciding what the model is allowed to see, where each piece of it comes from, and what happens when it is missing.

The one rule we do not bend

The model never sees a number that is not real, current, and from the source that is supposed to produce it.

That sounds obvious until you sit with how a language model fails. A traditional program that loses a data feed throws an error or returns a blank. A model handed a bad number does none of that. It reasons over the bad input with the same confidence it brings to the truth, and hands me a conclusion built on something that was never true. The failure is invisible, because the wrong output looks just as polished as a correct one.

Most of the integrity problems I have fought are not exotic; they are numbers. The classic issues are a figure that is a day old when it needs to be current, or a value pulled from a convenient secondary source that does not match the one canonical source it is supposed to come from. These never announce themselves. They look like good data, and a clean-looking number from the wrong place is far harder to catch than an obvious blank.

This is why a wrong number frightens me more than a missing one. A missing number is a dead end the system knows how to handle: show the model that the value is unavailable, give the reason, and move on. A wrong number is a contaminant. The model takes it in, reasons over it without hesitation, and carries it into every conclusion it touches. One bad input does not stay where it landed. It spreads through the read like a virus and corrupts the outcome, and because the final output is as fluent as ever, nothing about it looks sick. So a number is allowed in only if it is real, current, and from the source that is supposed to produce it. Being current is large part of that test, not a separate nicety: the inputs that can move between last night and this morning are refreshed right before the model reasons, so a quiet overnight is never mistaken for the world as it stands when a decision gets made. Everything else that’s not proven to be real and current is shown as unavailable, and unavailable always beats a guess.

You cannot point it at everything

This is the real reason you cannot simply hand a capable model a firehose and let it sort things out. Every source you add is a source you are now implicitly trusting, and that trust must be earned before an input goes anywhere near the read. It cannot be assumed just because the data was easy to get, or even from a reputable source.

A social feed is the clearest example of what we will not do. You cannot point the model at X, or Reddit, or any open stream of unverified posts, and treat what comes back as fact. There is real information in those places, surrounded by an ocean of rumor and noise, much of it planted, and a model cannot reliably tell them apart any better than a person scrolling quickly can. Feeding that to a system that will then reason from it with total confidence is how you manufacture a thesis built on a rumor. Being certain that a new source cannot poison the trusted ones is a precondition for letting it in at all, not an afterthought once something breaks.

News is not just news

Sources that look authoritative carry a subtler version of the same risk, and this one took me longer to respect than it should have. A segment on a business news channel is not just news. It is opinion wrapped in the credibility of a news desk, and a language model is the kind of reader that absorbs the opinion along with the facts. The framing, the adjectives, the choice of what to lead with and what to bury, all of it shapes the model’s read in ways that have nothing to do with what actually happened.

Articles make it worse by being selective with the very numbers I care about most knowing. A story is written to make a point, so it reports the figures that support the point and quietly omits the ones that would complicate it. Hand that to the model and you have given it half the data with a conclusion already drawn on top. Half the numbers under a confident narrative may be the most dangerous input of all, because it arrives pre-reasoned and persuasive, and a language model is more vulnerable to a persuasive frame than to a missing fact.

So we hold the line between the two. Numbers come from the sources that produce them, never lifted out of an article that happened to mention them. News and commentary, where the model encounters them at all, are treated as what they are: opinion to be weighed with skepticism, never as fact, and never as a source of figures.

Every input has a Force and a job

We also do not pour everything we have collected into one pile and ask the model what it thinks. Every input enters through one of the Five Forces with a specific job to do through a dedicated agent tasked with only evaluating that Force. Macro inputs come from the actual economic and policy releases and feed the macro agent that creates the macro read. Positioning inputs come from the options and futures markets and feed the structural read. Sentiment inputs come from investor surveys and market-based fear gauges and feed the read on crowd behavior.

This is the same point I made about the framework last time, now applied to the inputs rather than the output. A defined question built from defined inputs is one you can audit. When something looks wrong in the read, you can trace it back to the input and the specific agent that caused it. A capable model handed an undifferentiated pile of data gives you no such thread to pull. An input without a Force and a job is not information. It is noise with a number attached.

Deciding what to leave out

Choosing what to exclude is as much of the work as choosing what to include.

Some inputs we take from exactly one source and nowhere else. A key commodity reference price, for example, comes from a single exchange feed, with no backups wired in behind it. That is deliberate. A helpful fallback that quietly substitutes a number from a source of unknown quality is how a price that no real market printed ends up inside the read. If the one trusted source fails, the model is told the value is unavailable. That is the safe way to fail.

Free data sources degrade quietly, and we assume they will. A feed that returned clean numbers last month starts returning an access-denied page, or the same file reuploaded in a slightly different layout with the header sitting in a new place. A naive pipeline parses whatever comes back and passes it along, and now the model is reasoning over a web-server error it mistook for market data. So before we parse anything, we check that what arrived is the thing we asked for, in the shape we expect. When it is not, we refuse it and record the gap. A refresh job can also run, report success, and quietly insert nothing, so we track when each input last changed, not just when we last asked.

Some data we also choose not to give the model in raw form at all, because it is too noisy to reason over directly. Raw options-dealer positioning is the clearest example. It carries real information in aggregate and is treacherous in the particular, so rather than ask the model to interpret the raw figures, we let it use them only as corroboration for signals drawn from steadier sources.

Why this is most of the work

None of this is the part of building with AI that anyone puts in a headline. There are no clever prompts here, nothing to admire about the reasoning. It is sourcing and verification, and above all the discipline of deciding what not to use. We spend more time on it than on any other part of the system.

I care about it more than is probably fashionable, because I have watched careful analysis ruined by a single bad input, long before any of this involved AI, and because the capital being managed is, first and foremost, my own. A second reader of the market is only worth having if I can trust what it says. That trust does not start with the reasoning. It starts here, in the unglamorous decisions about what the machine is allowed to know.

Even with the data right, a fluent model can still reason its way to a confident mistake. Making the model itself trustworthy, as opposed to merely well-fed, is a different and harder problem, and it gets a piece of its own later in this series. First, though, comes a question that has to be answered before trust even enters it: once the model has read the Forces, what form should its conclusion take? Our deterministic strategies decide by counting votes. A language model’s reading does not reduce to a vote, and what it has to become instead is the subject of the next piece.

Blackworks Capital LLC manages funds through Blackworks Capital Management LLC, an Exempt Reporting Adviser. Nothing here is an offer or solicitation or investment advice. The systems described are in research and paper-testing and do not manage client capital. Past performance does not guarantee future results.