The Dupoux-LeCun-Malik paper makes a point that should be obvious but apparently isn't obvious to most people building AI systems: observation-based learning and action-based learning need each other. System A (observation) without System B (action) produces systems that know a lot but can't actually do anything useful with that knowledge. System B (action) without System A (observation) produces systems that do things without understanding why, which works great until it doesn't and then fails catastrophically. Most AI companies build one or the other. Integration is the hard part that nobody wants to do because it requires both systems to "speak the same language" about patterns, confidence, and outcomes. We spent 18 months on the integration. I'm not going to pretend it was fun.
The failure modes are instructive if you're paying attention. Pure observation, which is what most language models are: trained on massive text corpora, excellent at pattern recognition, can generate fluent text and answer questions and summarize documents. Can they actually do things in the world? Sort of. They can generate instructions. They can describe how to do things. But executing actions in the real world—clicking buttons, submitting forms, handling unexpected errors, adapting when the payer system returns something weird—that's where they fall apart. The paper puts it clearly: observation-based learning "lacks mechanisms for selecting informative data" and is "disconnected from action capabilities." You've watched a shedload of cooking videos but you've never touched a stove—good luck making dinner. Pure action, which is what reinforcement learning game-players are: they learn through trial and error, millions of attempts, eventually achieve superhuman performance at the specific game they trained on. But they have no abstract understanding of what they're doing. They can't generalize to new situations. They can't explain their reasoning. They just do what worked before. Change the game slightly and they fall apart. The paper: action-based learning is "notoriously sample-inefficient" and "struggles with high-dimensional action spaces."
In our system, observation and action aren't separate modules living in different services. They're interleaved in every execution loop. Observation informing action: the PatternLearner watches guidance requests, extracts signatures (problem type, domain, context features), compresses observations into patterns. When a new request arrives, the system doesn't start from scratch considering every possible approach. It matches against observed patterns. The observation history reduces the action space. Instead of "what are all the ways we could handle this?", it's "what approaches have we observed working in similar situations?" Much smaller search space. Much faster convergence. Action informing observation: every execution generates outcomes. These outcomes feed back to the PatternLearner. The system doesn't just observe external data—it observes its own results. Did the pattern work? How well? What went wrong if it failed? This creates a closed loop: observe patterns from guidance requests, apply patterns to new requests, observe outcomes from applications, refine patterns based on outcomes, apply refined patterns to next requests. Neither system can function without the other. Observation without action would be pattern-matching without validation—statistical associations without causal evidence. Action without observation would be trial-and-error without learning—reinventing the wheel on every request.
The specific mechanisms, and I hope you're taking notes: observation helping action includes abstract representations (PatternSignature compresses request features into matchable format), predictive models (confidence scores predict execution success), uncertainty guidance (low confidence triggers exploration rather than exploitation), and transfer knowledge (similar patterns share learned insights across contexts). When the action system faces a new request, the observation system provides context: "This looks similar to Pattern X, which has 0.87 confidence. Consider applying Pattern X with modifications for the specific context." Without this, the action system would be trial-and-error on every request, which is hopelessly inefficient at scale. Action helping observation includes grounded data (execution outcomes are real, not simulated—you know what actually happened), causal evidence (success/failure reveals what actually works versus what merely correlates), goal-directed observation (active patterns determine what to watch and learn from), and rich training signals (outcomes provide supervised learning data that pure observation couldn't generate). The observation system doesn't just watch external requests. It watches what the action system does with those requests. It learns "this approach worked" and "this approach failed" from actual executions.
Integration requires shared representations, which is the hard part everyone skips. The observation system and action system need to "speak the same language" about patterns, confidence, and outcomes. Most systems skip this entirely. The observation component has one data format. The action component has another. Translation layers in between. Information loss at every boundary. We spent months on this before building anything else. Our PatternSignature format, our confidence scoring system, our outcome classification scheme—all designed to be readable by both observation and action components. No translation needed. Patterns are patterns. Confidence is confidence. Outcomes are outcomes. The cost: significant upfront design work before you can build anything interesting. The benefit: tight integration that actually works rather than two systems that happen to run in the same process.
The paper emphasizes that biological organisms don't have separate "learning from observation" and "learning from action" systems. They have integrated systems that do both, controlled by meta-cognition that decides when to observe versus act. We implemented the same thing. Our meta-control layer decides: is this a situation where we should observe and accumulate, or should we act and learn from results? High uncertainty? Observe more, act cautiously. High confidence? Act more, observe results. New domain? Observe heavily before acting. Familiar domain? Act based on established patterns. The decision isn't hardcoded. It emerges from confidence levels and pattern matching. The system learns when to learn from observation versus when to learn from action. If your AI vendor can explain their observation system OR their action system but not how they integrate, they haven't solved the hard problem. At any rate.