A Second Brain That Thinks With You: Building a Knowledge System with AI

In this comprehensive tutorial, Matt Wolfe demonstrates how to construct a powerful "second brain" knowledge management system that goes far beyond simple information storage. The core insight driving this system is that most people's digital collections—transcripts, articles, podcasts, tweets—end up as dumping grounds where data goes to die unless you actively revisit and search through it. Wolfe's solution is to build an intelligent, interconnected system using Obsidian as the front-end markdown organizer, Codeex as the AI coding assistant, and Andrej Karpathy's LLM wiki architecture as the foundational blueprint. The result is not just a place to store information, but a living, queryable brain that grounds your journaling and decision-making in the knowledge you've already collected.

The Three Pillars: Wiki, CRM, and Journal

Wolfe structures his second brain around three core components that work together seamlessly. The first pillar is the wiki/knowledge base, which stores everything from YouTube transcripts and articles to podcast notes and tweets. This is not a flat repository; the AI layer automatically summarizes each source, extracts key entities like people, companies, tools, ideas, and themes, and then cross-links related notes so that clicking into a concept reveals all the saved content that mentions it. The second pillar is a CRM for tracking personal and professional relationships. Whenever Wolfe meets someone at an event or on a Zoom call, he records how they met, what they discussed, contact details, and any relevant context. This information becomes part of the interconnected web, so future journal entries or queries can surface past conversations. The third pillar is the journal, which serves as the primary interface for interacting with the entire knowledge base. When Wolfe writes about his struggles—say, a creative block or a video underperforming—the AI doesn't just give generic ChatGPT-style advice. Instead, it searches the wiki for relevant saved videos or articles, checks past journal entries for recurring patterns, and even looks at CRM notes for any conversations that touched on the same topic. The system then crafts a response grounded in what Wolfe himself found valuable and saved, making advice deeply personalized.

Building the Foundation: From Karpathy's Blueprint to a Working Wiki

The technical build begins with creating an empty Obsidian vault and then using Codeex to implement Karpathy's LLM wiki architecture. Wolfe prompts Codeex to build out the folder structure based on the GitHub repository provided by Karpathy, which yields a minimal but functional setup: a raw folder for immutable source files, a wiki folder for AI-generated markdown pages, an agents.md file that defines how the system should operate, an index.md that catalogs everything, and a log.md that records all actions. He then configures the Obsidian Web Clipper to automatically save any webpage or YouTube video (complete with transcripts) directly into the raw folder. After seeding the system with a handful of videos on discipline, motivation, and productivity, Wolfe runs the processing command. The AI extracts key concepts, creates wiki pages for each (e.g., "Discipline without Willpower," "Temptation Bundling"), and updates the index and log. The graph view in Obsidian begins to show interconnected nodes, a visual representation of the growing knowledge network. Wolfe also tweaks the agent instructions to add channel names to YouTube clips and to move processed files into a raw/processed subfolder, keeping the workspace clean.

Integrating Journal and CRM for Interactive, Grounded Intelligence

With the wiki functional, Wolfe expands the agents.md file to add two new capabilities: journaling and CRM management. He creates folders for journal and CRM and prompts Codeex to update the agent's rules. For journaling, any chat prefaced with "journal" becomes a dated markdown entry, added to a journal index. The AI's response must ground itself in the wiki, past journal entries, and CRM data—not just in its own general knowledge. For the CRM, any message prefaced with "add to CRM" creates or updates a contact file with name, meeting details, and context, all indexed alphabetically. Wolfe demonstrates both features live: a test "add to CRM" for a person named Matthew Berman correctly creates a record noting they met at Qualcomm and CES; a sample journal entry about struggling with clickbait titles versus literal titles triggers a response that pulls from saved videos on "YouTube valley of death" and "creator persistence," showing that the system truly uses the user's own curated content to generate advice. The journal entry is saved with a timestamp, the AI's response, and a synthesis section, while the CRM record can later be queried with "Where did I meet Matthew Berman?" and the system retrieves the stored context.

Automation and Backup: Making the Second Brain Self-Maintaining

To avoid manual processing every time new content is clipped, Wolfe sets up an automation in Codeex that runs hourly. The automation checks if any unprocessed files exist in the raw directory, processes them into wiki pages, and—after processing—commits and pushes the entire vault to a private GitHub repository. This creates a continuous backup and version history. The automation uses GPT-5.5 with high reasoning to ensure thorough extraction. The result is a system that requires minimal ongoing effort: clip a webpage with one click, and within an hour it's summarized, linked, and backed up. Wolfe emphasizes that the entire behavior is controlled by the agents.md file, which is just a set of natural-language instructions that can be edited at any time. Users can add new folders for specific categories (clients, workouts, recipes) and update the agent accordingly, making the system highly adaptable.

Practical Implications and Final Thoughts

Wolfe's second brain system transforms the way personal knowledge management works. Instead of passively collecting information, you build an active, AI-assisted partner that surfaces relevant content exactly when you need it. The integration of journaling with the knowledge base means that your daily reflections are enriched by everything you've ever saved, and your CRM ensures that professional relationships are never forgotten. The entire setup uses free or affordable tools: Obsidian is free, Codeex offers a free tier (with paid upgrades for more usage), and the Obsidian Web Clipper is free. The only recurring cost is potentially the AI model usage if you exceed free limits. Wolfe notes that the system gets smarter over time—the more content you add, the more interconnected and useful the graph becomes. He encourages viewers to adapt the system to their own needs, whether that means replacing the CRM with classroom notes or adding a workout log. The underlying architecture—a wiki at the center, with journaling and CRM as interactive layers—provides a flexible, powerful, and deeply personalized second brain that doesn't just store your knowledge but actively helps you use it.