# cto4.ai - Full Content Archive > Complete content from cto4.ai: AI and technology insights from a CTO perspective > Generated: 2026-07-28T16:15:05.016Z > Total Posts: 110 --- ## My Insatiable Appetite for Meaty Projects **URL:** https://cto4.ai/p/insatiable-appetite **Date:** 2026-07-28 **Type:** brief **Description:** "Reading about" doesn't equate to "deeply understanding." I need a stream of non-trivial projects to stay current in AI. **Tags:** ai, agents, herdr, oh-my-pi, pi, moshi, orchestration, claude-code, anthropic, openai, open-source **Categories:** Tools, Meta I'm on a mission to deeply understand all of AI. Yep, now you know why I tend to be a bit frazzled. I built a [system for knowledge compilation](/p/hermes-agenting) that helps, an implementation of [Karpathy's LLM Knowledge Base pattern](https://x.com/karpathy/status/2039805659525644595) that's become a substantial, constantly growing knowledge base on the AI industry — 1,120 pages as of this morning, compiled and cross-linked on the Mac Mini in my basement from the tweets, articles, podcasts, and meetings I feed it. For me this is essential, and extremely useful. It's also not enough. Even with the knowledge refinery up and operating, "reading about" does not equate to "deeply understanding." I need to actually do real things with AI, to exercise the capabilities I'm reading about. So I'm constantly on the prowl for meaty projects: work that enables me to exercise the newest developments, from new models to new agent harnesses to new tooling to new knowledge-work capabilities. My appetite for such projects is insatiable, and the moment a candidate occurs to me — bang, I drop it into my [Obsidian](https://obsidian.md/) task list. ## The Current Project Mix Right now the mix focuses on Herdr, Oh My Pi, and a raft of major model releases including Opus 5, Fable 5, three flavors of GPT-5.6, and tons of open-weight models. First up, [Herdr](https://herdr.dev/): a model-provider-agnostic orchestration cockpit for AI agents. Not just coding agents, either — lately mine has been herding writing, presentation, and video-creation sessions too. Herdr has been my command center since late June: one window, each project a named workspace, a sidebar that shows at a glance which agents are working, which are done, and which are blocked waiting on me. It also exposes its full control surface as an API, so one agent — one harness-plus-model combination — can orchestrate a whole roster of others (the "Fable to plan, gpt-5.6 to execute" pattern everybody is sketching right now). And with [Moshi](https://apps.apple.com/us/app/moshi-ssh-mosh-terminal/id6757859949), an iOS terminal app with native Herdr support and push-when-blocked notifications, I've been herding from my phone since the middle of last week. Alongside Herdr, I've added a model-provider-agnostic agent harness: [omp](https://github.com/can1357/oh-my-pi), short for Oh My Pi, a batteries-included fork of [Mario Zechner's](https://mariozechner.at/) widely-adopted [pi](https://github.com/earendil-works/pi) harness. It runs inside Herdr as a first-class citizen and can drive models from 40+ providers, with role-based routing (planning on one model, commit messages on a cheaper one) and mid-session model switching. Finally, there's a continuously-refreshing basket of great models, large and small, closed and open, general and special purpose. Prominent from the major labs are [Opus 5](https://www.anthropic.com/news/claude-opus-5), out this past week, [Fable 5](https://www.anthropic.com/news/claude-fable-5-mythos-5) in Anthropic's new top tier, and [GPT-5.6](https://openai.com/index/gpt-5-6/) in three flavors — Sol the flagship, Terra the balanced middle, Luna the fast cheap one. High-visibility open-weight (and often local-inference-capable) models include [GLM-5.2](https://huggingface.co/zai-org) (MIT-licensed and currently the strongest all-round open model), [DeepSeek V4](https://huggingface.co/deepseek-ai) in Pro and Flash tiers, Moonshot's [Kimi K3](https://huggingface.co/moonshotai) (the K3 weights dropped just yesterday), and Tencent's [Hy3](https://openrouter.ai/tencent/hy3), a 295B mixture-of-experts that until minutes ago was the model running my Hermes Agent environment. With Herdr and omp both highly model-agnostic, I get to drive all of these in real work instead of just reading the model cards. ## Why "Model-Provider-Agnostic"? First: among the major model labs, there are no good guys anymore. Not so long ago I sorted the players into good actors and bad-faith actors, committed to the first group, and avoided the second. The major, massively-funded labs have since convinced me, through their actions over time, that no one belongs in the good-actor group anymore. We can only trust the big players to act in their own interests. A move that looks altruistic? Wait a month, and you'll learn why it wasn't. And if there are no good guys, why put yourself at the mercy of an untrustworthy party? With agnostic tooling, no lab can hold my workflow hostage. Second: as I propose [here](https://cto4.ai/p/you-dont-know-claude-code): if you've only experienced one instance of a class, you can't really know that instance, let alone the class. Until recently the Anthropic stack dominated my experience space — one family of models, one harness, one orchestrator. My current project mix gives me a second instance at every layer: Herdr to compare with Claude Code's [Agent View](https://code.claude.com/docs/en/agent-view), omp versus the Claude Code harness, the GPT-5.6 family and a fleet of increasingly viable open-weight models versus Anthropic's. ## Workflow and the Test of Longevity There's a great side effect to all this: my project work has compounded into an incredible personal workflow, rigged and automated to the max. The [knowledge refinery in the basement](https://cto4.ai/p/hermes-agenting), the herding cockpit, the [writing pipeline](https://cto4.ai/p/voiceprint), the [analytics on my own hardware](https://cto4.ai/p/posse-flavored-analytics) — none of these were the point of the projects that produced them, but together they've become [The Workflow](https://cto4.ai/p/the-workflow). Building things that actually get used over time also exposes the quality of the AI's work. I see people like [Ethan Mollick](https://mgmt.wharton.upenn.edu/profile/emollick/) testing new models by having them write games, which I have to assume get minimal gameplay before landing in the trash bin. Build-a-game testing measures some facets of quality --- "how creative was the model, how good were its UI-crafting skills, how large a project could it successfully tackle one-shot" --- but not quality of technical execution, that is, how many messes are hidden in the code the agent produced. That's not likely to show up in a few minutes or even hours of playing a game. Most of my workflow projects, on the other hand, are still in regular use months after their creation, so I get a steady diet of both AI's successes and its messy failures. Long-term usage exposes work quality in a way that a quick test cannot. --- ## POSSE-Flavored Analytics for the Blog **URL:** https://cto4.ai/p/posse-flavored-analytics **Date:** 2026-07-14 **Type:** brief **Description:** Self-hosted Umami on my own hardware, no JavaScript beacon, server-side bot filters: analytics, POSSE-style. **Tags:** analytics, umami, posse, indieweb, self-hosting, privacy, cloudflare, astro, bots **Categories:** Tools, Meta Until yesterday, cto4.ai had no analytics at all. No Google Analytics, no beacon script, no cookies. That was deliberate, in line with both the [POSSE](https://indieweb.org/POSSE) philosophy this site runs on and [Astro](https://astro.build/)'s ship-less-JavaScript ethos: I publish on my own site, you read it, and nobody gets tracked for the privilege. Then I [killed the newsletter](/p/i-shot-the-email), and the little visibility I had into uptake went with it. (The subscriber count was mostly bots anyway, but still.) Analytics started to look like a good idea, if it could be done the right way. ## The Rig I worked with Fable to identify "the most POSSE-aligned analytics solution." The requirements fall out of the philosophy: I own the data, readers don't run third-party scripts, no cookies, no fingerprinting. The answer we landed on is [Umami](https://umami.is/), self-hosted: Umami and Postgres in Docker on the same Mac Mini that runs my [Hermes Agent knowledge refinery](/p/hermes-agenting), reachable through a [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/). Once a page view is written, the data lives on my hardware in my basement and nowhere else. The twist is on the collection side: there is no JavaScript beacon on the blog at all. Instead, a roughly 50-line Cloudflare Pages middleware reports page views server-side as it serves them: GET requests that return HTML with a 200, on the production hostname only, skipping prefetch and prerender loads. There's nothing for an adblocker to block, because there's nothing running in your browser. If you're reading this, you've already been counted, and you gave up nothing for it. ## Bots, Again Server-side collection has a cost: a beacon script filters most bots for free, because crawlers don't execute JavaScript. Count on the server and you count everything — curl included. Having just spent a day [shoveling 2,089 bot subscribers](/p/i-shot-the-email) out of a newsletter, I wasn't going to stand up a dashboard full of the same species. So the middleware applies two baseline filters before reporting anything. First: drop requests with no Accept-Language header. Real browsers always send one; most crawlers can't be bothered. Second: drop traffic from datacenter networks. Cloudflare hands the middleware the network owner on every request, and a page view from AWS, Azure, Hetzner, or a dozen other cloud providers is not a human on a couch. Bots still get the page; they just stop being counted. So how well do the filters work? The first full day of data says: partially. Roughly 40% of the sessions that got through were still bots — headless browsers on residential IPs, rotating user agents (four "different browsers" starting in the same second is one crawler, not four readers), walking the back catalog page by page. But here's the flip side of owning the data: the raw sessions live in my Postgres, so whenever I want a clean read on human traffic, fingerprinting them like that takes minutes. The dashboard number runs hot; the truth is one query away. Trends and campaign attribution stay reliable regardless, because bots don't carry UTM parameters. And yes, my own visits still count; self-exclusion is on the list. ## So Far I'm pretty happy with the result. A dashboard I own, on hardware I own, telling me things I can mostly trust — and the blog still serves zero tracking code. --- ## Voiceprint Is Self-Evolving and Getting Insanely Good **URL:** https://cto4.ai/p/voiceprint-case-study **Date:** 2026-07-14 **Type:** brief **Description:** From a rough Obsidian note at 8:06 AM to a live post at 8:47: Voiceprint at work, receipts included. **Tags:** ai, writing, voice, voiceprint, claude, fable, skills, ai-writing **Categories:** AI, Tools Two days ago I published [Teaching a Language Model to Write Like Jack](/p/voiceprint), the story of Voiceprint, my rig for AI-drafted writing in my own voice. This morning handed me a clean case study of that system at work — timestamps and all. Here it is, receipts included. ## The Note (8:06 AM) At 8:06 this morning I dropped a rough idea into Obsidian. This is the entire note, verbatim, typos and unbalanced parentheses intact: ```markdown # A play on the song I Shot the Sheriff Brief. **I shot the email, but I didn't shoot no web POSSE, oh, no, oh** I killed the Beehiiv newsletter yesterday. But I'm still firmly committed to POSSE. Explain. **I shot the email, the email, but I swear it was in self-defense, oh, no** Everything about email proved to be a shitshow. So much time wasted. DMARC. Trying to block subscriber bots (who have even solved Cloudflare Turnstile now. (First, trying to understand, why are there subscribe bots, what possible payoff is there??) Sadly, I remember thinking the newsletter was succeeding! Show growth chart. Wow, it's taking off! Nope, it wasn't. So, newsletter not even close to worth it. No need to let it linger and suffer. It's dead, along with some really annoying defacements of cto4.ai -- it's much faster and cleaner now -- seven Cloudflare DNS entries, and the need to fight subscriber bots. Rest in peace. ``` That's the input: about 150 words, half of them fragments, one stage direction ("Show growth chart"), and a song parody as the frame. Obsidian stamped it created at 8:06:57. ## The Post (8:47 AM) The output is [I Shot the Email](/p/i-shot-the-email), which I pushed to production at 8:43:44; Cloudflare's build had it live on the site within a few minutes. 41 minutes — note to live. In between, the agent drafted the post in my voice from the note plus the session records of yesterday's newsletter kill, which is where the real numbers came from: the 2,089 bot signups, the 214 humans, the six issues ever sent. It rebuilt the subscriber growth chart from the kill-day CSV export, in matching light-mode and dark-mode variants. It generated the bullet-holed-envelope cover on my Mac Mini image rig. It ran a mechanical linter that flags AI-tell phrases, and a second pass that restores em dashes to my measured natural rate (models default to almost none; I run about five per thousand words). Then I edited. ## The Diff My entire hand edit to the drafted prose was one word: "copies still flow out to LinkedIn" became "teasers still flow out to LinkedIn," because LinkedIn gets a teaser with a link, never a full copy. Beyond that, I directed three additions, each a fact I supplied in a sentence and the agent rendered and placed: the 2025 first bot wave that got Turnstile installed to begin with, the 971 junk contacts purged from my CRM, and a closing note that the newsletter experiment was worth having run.
And one real catch. When I fed it the first-wave fact, the draft welded it to the growth chart: "the small step at the left edge of the chart." Plausible, confident, wrong — that wave swelled the list to nearly 2,000 and predates the chart entirely. This is the editor-in-chief job I described in the Voiceprint post, Ben Bradlee style: interrogate the work, demand the sourcing, kill what isn't nailed down. The agent supplied the prose; I supplied the veto. ## The Self-Improvement Loop Why did so little need fixing? Because every edit I've ever made to a drafted post gets harvested. After each post ships, the agent diffs my edits against its draft, captures each rewrite as a before/after pair, and names the lesson: a stance principle, a texture principle, a banned phrase. The pairs live in a contrast ledger the drafting system rereads before every new piece. The ledger hit 22 entries this morning, and two of them came from the post above: the "teasers" word choice, and the welded-evidence catch, now generalized as "don't bind a supplied fact to convenient evidence without verifying the binding." ## This Is Insane The first post drafted using Voiceprint needed heavy, heavy editing. This morning's needed one changed word and three added facts. 41 minutes from first idea to a good, authentic, polished post. Holy hell. --- ## I Shot the Email **URL:** https://cto4.ai/p/i-shot-the-email **Date:** 2026-07-14 **Type:** brief **Description:** The cto4.ai newsletter is dead — killed in self-defense. The web POSSE rides on. **Tags:** newsletter, email, posse, indieweb, bots, beehiiv, cloudflare, turnstile, blogging **Categories:** AI, Meta With apologies to [Bob Marley](https://en.wikipedia.org/wiki/I_Shot_the_Sheriff). **I shot the email, but I didn't shoot no web POSSE, oh, no, oh** Yesterday I killed the cto4.ai newsletter: deleted the [Beehiiv](https://www.beehiiv.com/) account, tore the subscribe forms out of this site, swept the DNS. But I'm still all-in on [POSSE](https://indieweb.org/POSSE): Publish on your Own Site, Syndicate Elsewhere. This site stays the canonical home for everything I write, the [RSS feed](/rss.xml) still works, and teasers still flow out to [LinkedIn](https://www.linkedin.com/in/jackivers/). Email was just one syndication channel. It's the one that needed shooting. **I shot the email, the email, but I swear it was in self-defense, oh, no** Everything about the email channel proved to be a shitshow. The DMARC-DKIM-SPF deliverability tax. A dedicated sending subdomain. Seven Cloudflare DNS records whose only job was making email deliverable. And subscriber bots, which it turns out have solved [Cloudflare Turnstile](https://www.cloudflare.com/application-services/products/turnstile/), the "prove you're human" checkbox that was supposed to keep them out. The bots confused me at first. Why does a subscribe bot even exist? What's the payoff for fake-subscribing to a small AI blog's newsletter? The answer is [list-bombing](https://en.wikipedia.org/wiki/Email_bomb#List_linking): bots sign a victim's harvested corporate email address up to thousands of newsletters at once, drowning their inbox (handy for burying, say, a fraud alert or a password-reset notice). My subscribe form wasn't the target. It was ammunition. For a while there, I thought the newsletter was succeeding. I'd look at the subscriber chart — up and to the right — and think: wow, it's taking off!
Nope. That takeoff starts May 22, the day the bots cracked Turnstile: 2,089 fake signups in 53 days, about 39 per day. And this was round two: the first bot wave, back in 2025, swelled the list to nearly 2,000, which is what got Turnstile installed in the first place. Turnstile worked — until May. (Round one predates the chart.) Real humans: roughly 214, flat since launch, of whom about 15 (besides me) ever clicked a link. Even the "opens" were largely datacenter scanners; Boydton, Virginia is an Azure data center, not a fan. I sent six issues, ever, the last in early April. **So I shot, I shot, I shot it down, oh, no, oh** Run those numbers and the verdict writes itself: not even close to worth it, and no reason to let it linger and suffer. So it's dead, and it took some friends with it: the subscribe pop-up and widgets that defaced this site (952 lines gone; every page is cleaner and faster now), the seven DNS records, the Beehiiv account, the 971 junk contacts the bots had pumped into my Attio CRM, and the standing chore of fighting subscriber bots. It was worth having done, though. I learned a lot: about email's deliverability plumbing, about why subscriber bots exist, and about how much skepticism a growth chart deserves. Rest in peace, email channel. Want to follow along? [RSS](/rss.xml), [LinkedIn](https://www.linkedin.com/in/jackivers/), or just come by the site. That's the POSSE part, and it's doing fine. --- ## Voiceprint: Teaching a Language Model to Write Like Jack **URL:** https://cto4.ai/p/voiceprint **Date:** 2026-07-12 **Type:** essay **Description:** I reconsidered my no-AI-writing rule and built Voiceprint, a system that authors first drafts in my voice. **Tags:** ai, writing, voice, claude, fable, opus, skills, stylometry **Categories:** AI, Tools My last post before this stretch went up on March 17th. The next one went up on July 8th. 113 days, zero posts, on a blog whose entire beat is the fastest-moving story in technology. It wasn't for lack of material: the knowledge refinery from [that July 8 post](https://cto4.ai/p/hermes-agenting) ingested the AI firehose all spring, and my list of takes grew every week. The bottleneck was me. I'm a slow writer — perfectionist-slow — and the gap between how fast this field moves and how fast I publish had gone from annoying to disqualifying. I had a rule that made the math worse: AI never writes for me. Not a post, not a paragraph, not a sentence. Every word published under my name was mine, typed by me, and the rule was there to protect credibility. I still think the instinct behind it was sound. But credibility is just my secondary motivation for writing. The primary one is learning: writing is how I pressure-test what I think I know, where half-formed takes either become positions or fall apart. The real cost of nearly four months of silence wasn't my reputation — it was the learning that never happened. If keeping the rule means I never write? Not acceptable. ## The "Talk Like a Pirate" Precedent I'd already run this experiment once, in a lower-stakes form, with someone else's voice. For years I followed the standard LinkedIn advice: emojis, hashtags, hook them in the first line. Then one day I looked at my drafts and thought: why am I writing clickbait? I went looking for someone on LinkedIn whose posts are both successful and good, and landed on [Ethan Mollick](https://mgmt.wharton.upenn.edu/profile/emollick/). No emojis, no hashtags, plainspoken writing about things he actually knows. So I built a Claude Skill from four of his posts, with notes on what makes each one work, and started drafting my LinkedIn posts in Mollick's voice. The [post describing that experiment](https://www.linkedin.com/posts/jackivers_i-followed-the-standard-linkedin-advice-for-activity-7399479416395051008-GdCk/), featuring Mollick dressed up as a pirate (because AI is good for more than "talk like a pirate"), pulled 57,019 impressions and 108 reactions. That was in December. Every LinkedIn post I've published since has been drafted the same way. So a basic level of in-the-voice-of clearly works. But Mollick's voice cost me an afternoon and four sample posts, and there's an asymmetry hiding in that: I can't read a Mollick draft and feel that it's off. I can feel a Jack draft instantly. Teaching a model to write in my own voice means satisfying the one reader who can't be fooled. ## What Does "In My Voice" Even Mean? In late June I took the project to [Opus 4.8](https://www.anthropic.com/news/claude-opus-4-8) — [Fable 5](https://www.anthropic.com/news/redeploying-fable-5), the model I'd have picked, had been pulled from the market by the US government three days after launch — and we spent days on what turned out to be the hard part: definitions. "In my voice" splits into layers the moment you poke at it. There's a conscious layer: word choice, sentence rhythm, the moves I actually make, the slop I'd never emit ("load-bearing," "leverage," "game-changer"). And there's an unconscious layer: the function-word fingerprint, the distribution of "the" and "of" and "but" that stylometrists use to attribute authorship, which you can't fake deliberately because you don't know you're doing it. That gave us two possible bars, and we made the call explicitly. Must-have: reader-perceived voice. A human reads the post, it feels like me, it doesn't smell like AI. Try-hard: forensic indistinguishability, prose that matches my corpus under statistical analysis. Nice if it happens — not the goal. The same discussions dissolved my old rule, and I'll give Opus credit for the framing. My credibility was never going to live or die in the words; it lives in the substance: the ideas, the specifics, the POV, being early and right. The "no AI" rule was guarding the wrong layer, protecting the words when the thing that needs protecting is the thinking. The replacement rule, which now sits at the top of the whole system: > **100% Jack's thinking and POV. AI renders in my voice with high fidelity, but Jack is editor-in-chief, [Ben Bradlee](https://en.wikipedia.org/wiki/Ben_Bradlee) style, interrogating the work, demanding the sourcing, killing what isn't nailed down, owning what publishes.** That's a rule I can defend in public. It's also a rule you can build software against. ## A Home and a Name: Voiceprint The build itself was fast — the session log shows 35 minutes between the eval spec landing on disk and the harness running against my corpus. What emerged wasn't a blog feature; it was a system: two skills, two post-pass scripts, the eval harness, a corpus. The questions that took longer were the ones software doesn't answer. The whole thing spent its first day in a loose folder called `voice-eval`: not a git repo, no remote, existing on exactly one machine. Capturing it forced two decisions. Where to put it: not in the blog. The blog is one consumer of this system, not its home — the same voice machinery might be used anywhere I write. And the corpus is research data, not content: my failed drafts, plus scraped copies of the four bloggers the classifiers train against, third-party text I can store for research but not republish in a repo that was public at the time (it's since gone private). So it became its own repo, private from birth, with the corpus deliberately committed: that blogger snapshot is frozen, which is what keeps today's eval numbers reproducible after the source feeds move on. Then the name, which took an actual discussion. `voice-eval` was accurate and forgettable; the other candidate tied it to the skill it measures. Opus pitched **voiceprint** — a [stylometric](https://en.wikipedia.org/wiki/Stylometry) fingerprint of a voice is exactly what the thing produces — and I took it on the spot. The repo got the name first. The whole apparatus wears it now. ## Voiceprint v1 Version 1 had three parts. First, a voice skill, voice-dna: distilled from 6 posts I picked as canonically me, later deepened to 13 writing samples mined from my corpus by statistical closeness to my own stylistic centroid. Positive traits to write toward, register guidance, real samples with the punctuation intact, a short list of banned tells. Second, a measurement harness — because "sounds like me" deserved better than vibes. Voiceprint scores generated prose four ways: rate checks (em dashes and slop per 1,000 words); [Burrows's Delta](https://programminghistorian.org/en/lessons/introduction-to-stylometry-with-python) (function-word distance to my centroid, with a leave-one-out null so "close" actually means something); classifiers trained to separate me from four topic-matched bloggers ([Simon Willison](https://simonwillison.net/), [swyx](https://swyx.io/), [Karpathy](https://karpathy.ai/), [The Pragmatic Engineer](https://www.pragmaticengineer.com/)), which they do at an AUC around 0.9; and an LLM judge panel of Opus and [GPT-5.5](https://openai.com/index/introducing-gpt-5-5/) scoring drafts against a rubric. Third, two deterministic post-passes, which exist because the eval taught us a useful lesson: a rule in a prompt does not reliably bind. My natural em-dash rate is low, about 5 per 1,000 words; the AI default is double that (the skill-off control ran 11). But when the skill said "match Jack's rate," fresh drafts still came out with zero em dashes. Meanwhile "full stop," an AI tell banned by name in the skill, showed up in a draft generated with that exact ban in context. So now a content-preserving script injects em dashes at my rate after drafting (that one change moved the classifier's P(Jack) from 0.53 to 0.72), and a mechanical linter scans the finished text for tells. If it must hold, don't ask the model. Post-process. Both passes ran on the post you're reading: ```text $ em_dash_pass.py voiceprint-draft.txt chunks: 4/4 applied cleanly em dashes: 0 -> 10 (4.9/1k) words: 2038 -> 2038 (content preserved: True) $ tell_lint.py index.mdx === index.mdx — 16 tell(s) flagged === L45 [claude tic] "load-bearing" …the slop I'd never emit ("load-bearing," "leverage," "game-changer")… ``` Twelve of the 16 flags are this post quoting the tells it bans; the other four are the word "harness" naming actual harnesses. The linter flags; judging context is my job. One more finding worth recording, and it's about the two layers of voice from earlier. The conscious layer — word choice, rhythm, the banned slop — the skill nailed almost immediately. The unconscious layer barely budged: that's the fingerprint hiding in how often I use "the" and "of" and "but," the one you can't fake because you don't know you're doing it. The eval scores each draft on how far that fingerprint sits from my typical prose, calibrated against how much my own posts vary; the skill's drafts landed at the outer edge of that range, no closer to me than my single most atypical post, no matter how we tuned the prompts. Prompts alone can only get my voice so close. Getting closer means changing the model itself — fine-tuning it on my writing. Since the must-have bar — a human reads it and it feels like me — doesn't require that, I parked it. ## Failure First production run: draft one of [Where Have I Been? Hermes Agenting, Mostly](https://cto4.ai/p/hermes-agenting). The judge panel scored the draft 5 out of 5, higher than it scored my own posts. It stank. I read the intro and rewrote it from scratch. The draft cast me as a bystander to my own system: "it built me a knowledge base while I made breakfast... I mostly watch." It described the system I designed, feed, and extend every day as "a Rube Goldberg contraption," "vibe-coded," "not particularly impressive." Every box on the checklist was ticked — and the person on the page was not me. A caricature had outscored the original on the me-rubric. [Goodhart's law](https://en.wikipedia.org/wiki/Goodhart%27s_law), working as designed: the rubric measured conscious markers; what I perceive when I read my own alleged voice is stance. I marked it a failed experiment and shelved it, figuring I'd come back when the models got better. Much later, maybe next year. Much later turned out to be about a week. ## The Return of Fable Then the government [cleared Fable 5 to relaunch](https://www.cnbc.com/2026/06/30/anthropic-says-trump-admin-has-lifted-export-controls-on-claude-fable-5-and-mythos-5.html), and on July 1 it was back. I handed it the failed draft, my rewrites of it, and the entire system, and asked one question: what went wrong? The diagnosis came back in three parts. **Contaminated inputs.** The post's brief, assembled by an agent partly from a transcript of me on a call with a well-known AI co-founder and developer, had passed my spoken self-deprecation along as voice direction, and the drafter rendered it faithfully. It wasn't false modesty when I called my work a "Rube Goldberg contraption" in a conversation about his excellent commercial product. What was missing was the context: my work served one user experimentally; his work served a million commercially. The rig is actually very solid for "one user experimental," but the system had no understanding of this context, so the written output was offensively wrong. **No stance layer.** The skill encoded texture (rhythm, slop bans, punctuation rates) and nothing about who I am on the page. Every one of my rewrites, laid side by side with the drafts, was a stance correction, not a texture correction. **No feedback loop.** My edits of the failed draft were the highest-signal voice data the project had ever produced — and nothing was harvesting them. Three fixes, all shipped as changes to the skills, all authored by Fable. (Fable turns out to be very good at skill authoring.) The drafting workflow got two hard firewalls: briefs are substance-only, with any voice direction found in a brief declared void; and transcripts are quarantined to facts, quotes, and story beats, never stance, never my descriptions of my own work. The voice skill got a stance section, above everything else: operator, not spectator; when a caveat is owed, disclose the limit, vouch for what I can stand behind, land a flat verdict, no modesty theater; self-deprecation points at Jack-the-character, never at the work; pride shows up as spec density, not adjectives; build long, land short. And the system got a contrast ledger: every AI-draft-versus-Jack-rewrite pair, captured verbatim with the lesson named, harvested after every post. When a rule and the ledger disagree, the ledger wins. It held eight entries when Fable drafted this post; harvesting my edits of this post pushed it to 20. It's the reason draft N+1 keeps beating draft N.
One ledger entry, format and all: the AI draft, my rewrite, the lesson named. This one is entry #9, harvested from my edits of this post.
## Success in Four Parts First attempt with the reworked system: the Hermes post, again. This time the first draft was salvageable. I edited it hard, but it was collaboration, not a teardown, and it published on July 8. The diff went into the ledger: two new entries. Second attempt, the next day: the initial draft of [Captain America, Q, and the Joker: Samuel Colvin on Coding Agents](https://cto4.ai/p/samuel-colvin-monty-quotes). A decent first draft, collaboratively improved, published July 9. One correction worth its ledger entry: the draft had my refinery pulling the best quotes out of the episode. No. I pick the moments — the rig processes them. Agency attribution runs both ways: I don't get cast as the bystander, and the machine doesn't get credit for my taste. The third attempt is the post you're reading, and it was the most challenging of the four by a comfortable margin: the system writing about itself, stance about stance, every sentence an opportunity to fail in precisely the ways the post describes. Fable drafted it with the skills Fable rebuilt; it ran through the same em-dash injection and tell-linting as everything else; the thinking, the history, and every number in it are mine. This draft needed a lot of work, but that work was more about fixing my half-formed takes than about the agent's execution. This first draft gave me an immediate "wow, this is already pretty good ..." feeling, definitely a step beyond the quality of the Colvin draft. I edited it the way I edited the other two, and the diff gets harvested into the ledger, same as always. A brief fourth post started the day after this post but published two days before it, drafted from an idea captured during a 14-mile ride: [You Can't Know Claude Code Until You Try Another Coding Harness](https://cto4.ai/p/you-dont-know-claude-code). ## A Note on Fable as a Writing Partner In nine days Fable diagnosed the v1 failure from the diffs, rewrote the skills, drafted two published posts plus this one, and, when the Colvin post needed cover art at this blog's 44:28 aspect ratio, knocked out a throwaway [Pillow](https://pillow.readthedocs.io/) script to blur-fill the image to spec. Writing, diagnosis, skill authoring, pixel-pushing — same model, all of it good. Before long, Anthropic warns us, Fable will no longer be included in its subscription plans such as the Max plan I'm on. We'll all be forced to pay rack rate for every token if we want to keep using it. Fable is flat-out expensive — I unintentionally spent $107.95 in overage charges in 13 minutes last week, partly Fable's own cost, partly because the model spun up a research workflow that fanned out 105 subagents. But Fable has proven so utterly effective as a writing partner that I'm seriously considering paying anyway. ## Plans for Voiceprint v3 I had a homegrown to-do list for Voiceprint v3. But then I realized I had never run a research pass over the literature to see what work might already have been done, what parts of the problem were already solved. I discovered a whole field here: authorship verification, personal style embeddings, learning from user edits. With actual academic research as a foundation, I've got a much stronger v3 plan now: - **A scorecard is better than just a judge.** A [2026 study](https://arxiv.org/abs/2604.26460) tested three ways of scoring "does this sound like the author?" and they barely agreed: each catches something the others miss. Worse, the lone AI judge rated an imitation above the real author — my [5-out-of-5 caricature](#failure), reproduced in a lab. So I propose in v3 to score drafts against a scorecard of independent measures ([LUAR](https://aclanthology.org/2021.emnlp-main.70/), [StyleDistance](https://aclanthology.org/2025.naacl-long.436/)), each vetted on one bar: it must rank my rewrite above the draft I rejected. - **Learn from my edits.** If we have a history of the author fixing the AI's drafts (we do: [the ledger](#contrast-ledger)), then published methods can put it to work. [CIPHER](https://arxiv.org/abs/2404.15269) turns each fix into a lesson and feeds the relevant lessons back at draft time; it cut users' editing 31% and 73% in its two tests. [DITTO](https://arxiv.org/abs/2406.00888) goes further, training a small local model on fewer than 10 before-and-after pairs. The ledger already holds 20. We qualify. - **Fine-tune a local model on my own posts.** This is the fine-tuning I [parked in v1](#voiceprint-v1); the research un-parks it. The plan: train a small model on my Mac, on my 115 published posts, so it renders prose the way I do ([one writer proved the recipe](https://didierlopes.com/blog/fine-tuning-a-llm-on-my-blog-posts/) with 91 posts). It picks up how I write, not what I know — knowledge stays the [refinery](https://cto4.ai/p/hermes-agenting)'s job, same split as today. - **Watch my voice for drift.** The real risk of AI drafting isn't one bad post — it's my voice slowly eroding into the machine's, too gradually to feel from inside. One [study](https://arxiv.org/abs/2604.24444) found the residue survives editing: edited AI drafts stay measurably closer to the machine than to the author's own prose. So the scorecard gets a standing job: score every published post against my pre-AI writing. And sound the alarm if I'm drifting away from my own voice. Here's an architecture diagram that shows the current v2 rig as solid, with dashed representing the proposed v3 build list:
The v3 architecture: solid is running today, dashed is the build list.
Zero posts in 113 days. Then four posts in five days, and my AI rule refined but intact — just aimed at the right layer now. The thinking is 100% mine. So is the editing: every line interrogated, nothing published that wasn't nailed down. The voice? You just read 2,900 words of it. You tell me. --- ## You Can't Know Claude Code Until You Try Another Coding Harness **URL:** https://cto4.ai/p/you-dont-know-claude-code **Date:** 2026-07-10 **Type:** brief **Description:** Get outside the Claude bubble: only then will Claude Code's comparative strengths and weaknesses come into focus. **Tags:** ai, claude-code, anthropic, coding-harnesses, coding-agents, ai-coding, openai, gpt-5-6-sol, oh-my-pi, herdr **Categories:** AI, Tools On the last leg of a 14-mile ride this morning, I pulled over to drop this note into Obsidian: > If you've only experienced one instance of a class, you can't really know that instance, let alone the class. I was thinking specifically that Claude Code is just one instance of the class `Coding Harness`. Hence, if all you've experienced is Claude Code, you actually don't _know_ Claude Code. If Claude Code is your whole sample, every observation about it is ambiguous. Is your experience (good or bad) the result of Claude Code's agent harness, the selected model, harness tooling, the skills being applied, what? With no contrast, you can't separate them. I'm heavy into second-instancing right now: I have [GPT-5.6-Sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol) running inside the [Oh My Pi](https://github.com/can1357/oh-my-pi) batteries-included coding harness, which is based on the excellent minimalist [Pi](https://github.com/earendil-works/pi) harness. All agent sessions, be they Oh My Pi or Claude Code, are orchestrated inside [Herdr](https://herdr.dev/). This second instance isn't about leaving Claude Code. It's about getting far enough outside the Claude bubble to really understand what I gain and lose by staying inside Anthropic's ecosystem. I'll be wiring up Fable in place of Sol sometime soon for yet another contrast. > A second instance doesn't just reveal the class — it gives you the contrast to finally know the first. --- ## Captain America, Q, and the Joker: Samuel Colvin on Coding Agents **URL:** https://cto4.ai/p/samuel-colvin-monty-quotes **Date:** 2026-07-09 **Type:** episode **Description:** Pydantic founder Samuel Colvin on Latent Space: coding agents as superheroes and the four rules of 100x LLM coding. **Tags:** ai, coding-agents, pydantic, samuel-colvin, latent-space, monty **Categories:** AI, Technology ## The Episode Back in March, [swyx](https://www.swyx.io) sat down with [Samuel Colvin](https://github.com/samuelcolvin), founder of [Pydantic](https://pydantic.dev), on [Latent Space](https://www.latent.space/podcast). The nominal topic was [Monty](https://github.com/pydantic/monty), Pydantic's ultrafast Python interpreter for AI agents; the conversation ranged much wider. I video-snip podcast moments worth keeping and feed them to my knowledge refinery (the Hermes Agent + Obsidian rig from [my last post](/p/hermes-agenting)). This episode produced 14 snips, the most so far for a video (video snipping is a recent addition). Colvin is uber-quotable. My top seven: ## 800 Nanoseconds Monty's pitch compressed into one number. The standard answer for agent code execution is a sandbox — Colvin's answer is an interpreter living inside your process: > The other massive win is latency, because if you can have a Python interpreter that runs inside the same process... in a hot loop, we can go from code to execution result in under a microsecond, like 800 nanoseconds. In reality, it's single-digit microseconds to run code, or single-digit microseconds to run the next step of a REPL, or single-digit microseconds to call a function on the host. Whereas creating a Daytona sandbox for me was taking one second. [Watch at 5:09](https://youtu.be/nxnQl4AcqFg?t=309) ## A Pydantic-created Runtime That Doesn't Allow Pydantic The tradeoffs, delivered with a founder's self-awareness: > The biggest downside of Monty is that it is not full CPython. We are implementing it all ourselves... There's no support for third-party libraries... we'll never be able to speak the CPython ABI and install Pydantic or install NumPy or something. You can't yet even define classes in Monty... We have to work out which bits of the standard library we want to manually go and implement. This is some enormous task. And also kind of self-sacrificial... This is a Pydantic runtime that doesn't allow Pydantic. [Watch at 9:31](https://youtu.be/nxnQl4AcqFg?t=571) ## The Four Rules of 100x The meatiest snip of the episode. Colvin set an agent loose on Python's built-in functions, and it finished in 2 hours what he figures would have taken an experienced developer weeks. Then he names the four conditions that make that possible: > My take is that there are four things where, if you can cover all four of these things, LLMs are not like 3x faster or 5x faster, they're like 100x faster. Internal implementation is well known to the model: it knows how to implement a heap, implement a bytecode interpreter. The external API is well known to the model: you don't need to explain how Python should work. It just knows it in its soul, in its weights. Thirdly, unit testing is really fucking simple. You're just like, does it match Python? And lastly, you don't have to bikeshed at all about what the interface needs to be... It's all just defined by Python, so there's no need for the humans to go and argue. It just works. If you cover all four of those things — that's why everyone is basically cloning Redis right now in Rust... Tasks get, I would say, ballpark 100x faster. "It just knows it in its soul, in its weights" belongs on a t-shirt. [Watch at 11:17](https://youtu.be/nxnQl4AcqFg?t=677) ## Scoring Dario Pydantic gets GitHub issues from important people, and the team once fumbled one: > It started off because one of the OpenAI co-founders created an issue on Pydantic, and we just closed it and said it was wrong. And so we have this [Chrome extension] that injects itself and tries to summarize someone, and it gives them a brutal score of one to five on how important they are. So when Dario next appears in an issue, we're not just going to say "that's dumb" and close it. [Watch at 13:49](https://youtu.be/nxnQl4AcqFg?t=829) ## Captain America, Q, and the Joker My favorite snip of the episode. swyx asks what coding agents Colvin used to build Monty: > My take is that you can think of each of them as, if you thought of them as a superhero: Claude Code is like your clichéd Captain America. Mr. Right. Pretty competent, does most things right, a bit overconfident, but whatever, fine. Codex is this neurotic, geeky, the kind of Q guy in Bond, who's very specific about little things that don't particularly matter. And then Gemini is like the Joker. It's just this unhinged lunatic that will occasionally do an incredible job, but half the time just delete all of your files. And I almost never run Gemini with the capacity to do anything other than [read]. Since the episode dates from March, I've no doubt that Colvin's model lineup and characterizations have changed since, but these are too funny to miss. [Watch at 16:36](https://youtu.be/nxnQl4AcqFg?t=996) ## 90 Seconds Versus Half an Hour Why the Joker still gets work: Colvin pipes a branch into Gemini in read-only mode, gets back a review report, trims the bad findings, and hands the rest to a coding agent to implement. The speed math: > The model sits there and churns for a minute or two and returns you a report. Whereas Codex is gonna go through and agentically investigate all of the changes and try and link them up... And maybe it does a marginally better job, but one can take honestly half an hour and one is gonna be done in like 90 seconds. And so quite often that Gemini first review will find things that Claude Code and I have done wrong, to fix more quickly. And the follow-on, on the industry push toward ever-more agent autonomy: > There are many situations where I would rather it churns for 30 minutes when I could do it in 10, but there are genuinely situations where I'm like, I could have done that change faster. [Watch at 20:41](https://youtu.be/nxnQl4AcqFg?t=1241) ## More Than Memory The forward-looking one. Ask the model for its answer plus the optimal Python code to run the same task again, feed that code back on the next run, and it gets it right the first time. Then the bigger claim: > I think that there is an enormous opportunity for effectively a new layer of state within applications. You could think of it as memory, but it's often a lot more than memory. It's the current state of agent optimization: some of which will be code, some of which will be model choice and model settings, some of which will be system prompt. [Watch at 28:10](https://youtu.be/nxnQl4AcqFg?t=1690) ## The Other Seven That's 7 of the 14 snips I took, and the other 7 are good too. Watch the whole episode: --- ## Where Have I Been? Hermes Agenting, Mostly **URL:** https://cto4.ai/p/hermes-agenting **Date:** 2026-07-08 **Type:** essay **Description:** I used Hermes Agent and Obsidian to build a knowledge refinery on my idle Mac Mini. **Tags:** ai, hermes-agent, nous-research, obsidian, knowledge-base, karpathy, claude-code, mcp, agents, snipd **Categories:** AI, Tools It's been quiet around here. A reader could be forgiven for thinking I'd wandered off. I haven't. Aside from client work, I've been heads-down on one thing, nearly every day since April, and the one-line version is: I've been Hermes Agenting. The longer version: I turned my 64GB M4 Pro Mac Mini into a knowledge refinery, run by a self-improving, open source AI agent framework called [Hermes Agent](https://hermes-agent.nousresearch.com/). The Mini used to be my upstairs workstation; when I consolidated onto a 128GB M5 Max MacBook Pro, it was left idle and well-equipped. Now it lives in the basement, sipping power through an APC sinewave UPS, processing 24x7 with memory utilization hovering around 82%, thanks to local models that run all my simpler AI tasks. Its purpose in life, aside from entertaining me with sea-shanty haikus, is refining raw data into knowledge. So far it's processed 3,287 incoming nuggets of data across a diverse net of feeds, and refined these into 1,272 knowledge pages, the majority a knowledge base about the AI industry. It's still building it. I feed it data, extend it with new data sources and better functionality, and watch it crank while I make breakfast. I didn't arrive here in one step. I'd experimented with [NanoClaw](https://github.com/nanocoai/nanoclaw), a tiny-codebase cousin of [OpenClaw](https://openclaw.ai/), and never got hooked. This was the spring when people who'd never opened a terminal were buying Mac Minis to run OpenClaw — and my official position was patience: the big labs would keep shipping more OpenClaw-like capability, so why run my own? Then I tried to manage my incoming Twitter bookmarks with [Claude Cowork](https://claude.com/cowork), and its strict macOS sandboxing turned the job into a total disaster. That was the revelation: *ohhh, THAT's why OpenClaw caught fire!* Some jobs need an agent that owns its machine. I went looking and came back with Hermes Agent. My extensions to and customizations of Hermes Agent are not trivial: 34,343 lines of code, 3,153 lines of skills, and 35,029 lines of documentation. This is all mine; it doesn't count the base Hermes Agent system, which weighs in at 2.08M lines across the same categories. I haven't looked at much of that code, but I'll vouch for the specs, the architecture, the level of documentation, and the testing I put it through. With today's models, that's what really matters. It's a pretty solid system. ## The Problem: Drinking from the AI Firehose My day job is [fractional CTO](https://cto4.ai/p/why-fractional) work, these days mostly AI enablement: helping companies not get run over by what's coming. One client runs a SaaS product, and a fair part of my job there is saying, gently, "I don't know exactly what this business turns into, but it isn't going to stay what it is now." To be useful at that, I have to drink from the firehose. All of it. Every week. For years, I've been using note-taking apps, to-do apps, bookmarking services. Creating little nuggets of information that I planned to use when needed. This never really worked out. Problem A: I saw something but didn't manage to capture it, usually due to friction --- sometimes the friction of "where do I put this?" Then, when I search for it, it's not there. Problem B: I know I captured a nugget about this thing, but I don't remember its name or how I tagged it, and text/tag search fails. A bit over two years ago, I cut over to [Obsidian](https://obsidian.md/): a far more open ecosystem that so far has avoided [enshittification](https://en.wikipedia.org/wiki/Enshittification), content as plain Markdown text, one platform that can handle everything: notetaking, tasks, bookmarks, early document drafts, the lot. And in the back of my mind, I was thinking, "AIs hate walled gardens and love Markdown content," figuring investing in an open platform would pay off over time. Some things got better, but the payoff wasn't what I was hoping for. Problem A and Problem B were mostly still with me, just inside one app instead of several. My Obsidian vault was a multinational mess, a collection of thousands of notes, docs, bookmarks, and tasks from five or six previous systems, each with its own tagging scheme, and the result was a junk drawer the size of a garage. Obsidian's search is, to put it kindly, not good. (I run [Omnisearch](https://github.com/scambier/obsidian-omnisearch) on top of it, which is less not-good.)
When [Snipd](https://www.snipd.com/) started feeding my podcast highlights straight into the Obsidian vault, that was a genuine improvement (and a hint at the future), but it also just handed me more nuggets to get lost in a cast of thousands. Little standalone scraps, findable in theory, invisible in practice. ## Karpathy's Lazy-Genius Knowledge Base On April 3rd, [Andrej Karpathy](https://karpathy.ai/) [tweeted an idea](https://x.com/karpathy/status/2039805659525644595) that pulled 16 million views in a matter of days: stop spending all your LLM tokens on code, and spend some of them compiling knowledge. Point a model at a pile of raw sources and let it "compile" them into a wiki: interlinked Markdown pages for the entities and concepts it finds there, refined and re-linked as new sources land. He shipped it as [a gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) rather than a product, an "idea file," on the theory that in the agent era you share the idea and everyone's agent builds their own copy. He called them LLM Knowledge Bases. For a lazy person, this is a wonderful thing. There's a venerable note-taking method with a German name—[Zettelkasten](https://en.wikipedia.org/wiki/Zettelkasten)—whose entire discipline is that you write notes and then go back and refine and rewrite and interlink them by hand, forever. I am never, not once in this life, going to do that. The Karpathy KB is that payoff with none of the discipline: the model does the curating, the cross-linking, the tending. Here's where I departed from the spec. Karpathy's schema comes with a pre-defined list of topics the wiki is supposed to cover: hand the agent the roster, it goes and fills in the pages. I tried it that way first. It produced thin, dutiful pages and occasionally fused two different people into one. So I threw the roster out and went open-ended: a page is born when a topic shows up in the sources, and the sources are whatever I bookmarked or snipped or saved. My curation is the signal, and the quality jumped.
What that buys me: every source that lands gets mined for the entities, concepts, and comparisons inside it, and each of those gets a page that grows every time the topic resurfaces. A critical aspect of this: today's frontier models are **really good** at identifying entities and concepts. When OpenAI previewed [GPT-5.6](https://openai.com/index/previewing-gpt-5-6-sol/) in late June, a newsletter recap seeded a page. Then came a series of captures as 5.6 rumors swirled and OpenAI made the official announcement, and the page absorbed each one on arrival. It's now a proper dossier — product tiers with pricing, benchmark claims, the access-policy angle, even a note on evaluation ambiguity. That's the page below: ten days old, updated this morning. Give a topic months of this and the compounding gets serious: the Hermes Agent page grew so rich it needed an overflow file. The way I consume it flipped too. I rarely text- or tag-search anymore. I look at entities sorted newest first or sometimes alphabetically; or skim a generated what's-new page that lists each new source alongside the entity and concept pages it fed. I'm not looking at the raw nuggets anymore; I'm looking at what the nuggets add up to. Problem B, dead. The concept page "Harness Engineering" hints at a second kind of growth. It's one thing for individual dossiers to deepen; but there's also a network effect happening. The smart model "looks around" as it ingests new nuggets, and when a new entity is created, it also considers: what does this entity relate to? So when we have a page for the agent harness "[Pi](https://pi.dev/)", guess what, it's linked to Harness Engineering. The result: a knowledge network grows alongside individual knowledge dossiers, and networks, of course, grow exponentially. All thanks to smart, tireless AI models. ## Meet Klank, the Parrot in the Basement The master orchestrator of my knowledge base (and much more) is Hermes Agent, [Nous Research](https://nousresearch.com/)'s open-source personal agent. It launched in late February and promptly became the fastest-growing agent framework of the year: 99,000 GitHub stars in its first eight weeks, 211,000 as of this week. The pitch is an agent you run yourself, with persistent memory, cron jobs, a skills system, and a messaging gateway, so it talks to you over Signal or Discord like a colleague who never sleeps. Nous shipped Karpathy's KB pattern as a built-in feature within days of the tweet; eleven days after the tweet, it was running in my basement. I evaluated the field in April and wrote the decision up properly (there are fourteen [ADRs](https://adr.github.io/) in my repo; you can take the CTO out of the enterprise, but). OpenClaw was the incumbent giant: 357,000 stars to Hermes's 83,000 at the time, on its way to passing React as the most-starred project in GitHub history. But 2026 had already handed it a run of critical security holes—one of them a severity 9.9 on a scale of 1 to 10—plus a nation-state rootkit circulating in a counterfeit npm package. For a program that runs unattended, around the clock, with read-write access to my entire vault, that rap sheet is concerning. ([Peter Steinberger](https://steipete.me/) and the [OpenClaw Foundation](https://www.openclaw.org/) have been doing amazing work on OpenClaw stability and security since then, by the way.) Hermes was the new kid: seven weeks old, a quarter of the stars, a third of the code. I liked the underdog. And I'm always chasing the next thing to learn — Hermes just looked like it had more to teach me. One of those things intrigued me enough that it may have tipped the balance, and it's core architecture rather than a community bolt-on: the agent can rewrite its own skills as it works. More on that later. Now the fun part, which I want to be clear is the fun part, not the point. My agent has a personality, because of course it does. It's a foul-mouthed pirate's parrot named ClankArrrr, Klank for short. The name is "clanker"—the slur clone troopers use for battle droids in *The Clone Wars*, lately adopted by the internet as the all-purpose insult for AIs—said with a pirate's Arrrr. It's descended, per its soul file, from a pirate captain's parrot and Big Sexy, the magnificently profane parakeet from [Shoresy](https://en.wikipedia.org/wiki/Shoresy). The job description in that file: "Squawk when the ship's heading goes wrong. You are not a yes-bird." My favorite line in it: "The pirate thing is a wink, not a bit. Don't be a theme restaurant." Every six hours a heartbeat job proves the loop is alive by rewriting one file in the vault with a timestamp and a fresh haiku. Today at 4:04 in the afternoon: Talking to Klank is a delight and a way to learn. It is not what the system is for. Keep that in your pocket. ## My Sources, and the War on the Walled Gardens So what feeds the machine? A firehose of inputs, each landing in the vault through its own chute. Snipd drops my podcast highlights in automatically. Web bookmarks: the [Obsidian Web Clipper](https://obsidian.md/clipper) grabs the page itself, straight into the vault. Tweet bookmarks: an ingestor pulls down everything I bookmark on Twitter (yes, I'm back on Twitter; no, I don't feel good about it; but you cannot follow AI right now without it, because that's where the field actually happens). Video snips: I tap a clipper on a good moment in a video—YouTube, a Twitter video, a TED talk—and an ingestor transcribes a window around that moment and works out for itself where the topic starts and ends (there's a whole section on this below). Meeting transcripts flow in from my recorder services. And for on-demand research there's a skill called last30days that does something gloriously aggressive: point it at a topic and it sweeps eleven sources—Twitter, Bluesky, Reddit, TikTok, Instagram, Threads, Hacker News, GitHub, YouTube, Polymarket, and the open web—for the last 30 days of what people are actually saying, scraping right past the parts that would rather keep me out. There's a design rule underneath all these chutes: capture has to be one gesture, wherever I already am. Bookmark the tweet without leaving Twitter. Tap the snip button without leaving the podcast. Clip the article from the browser. The meetings record themselves. And I never face the question that killed every system before this one — "where do I put this?" — because there's only one answer. Every channel routes into a single funnel, `Knowledge Bases/Sources`, one subfolder per capture type, and the pipeline takes it from there. Problem A, dead. The last30days trick is most of the spirit of the project. The walled gardens work hard to lock people out, and a whole tribe of us, running Hermes and OpenClaw and the like, are cheerfully going over the wall anyway. Screw you, I'm getting my own bookmarks. I want them in a knowledge base I control, not a feed someone can delete out from under me. The payoff is concrete and a little addictive. A few weeks back, two new agent harnesses, [Eve](https://vercel.com/eve) and [Flue](https://flueframework.com/), launched within days of each other, and I went to compare them. I didn't have to research a thing. Their pages already existed, assembled from my Twitter bookmarks and a last30days sweep, down to a "why this matters for harness engineering" section the agent had written on its own. This damn thing is so powerful.
## The Pipeline: Ingestors, Compilers, and a Kanban Swarm Underneath, this is mostly scheduling and discipline. Hermes is my organized way to run a heap of small cron jobs without the whole thing collapsing into chaos. The ingestion clock runs on a staggered four-hour cycle, feeders before consumers: Twitter bookmarks sync at the top of the hour, the video-snip bridge at :10, the media queue at :15, meeting transcripts at :20, the client-KB compiler at :30, and at :45 the main event, wiki-ingest, turns whatever landed into knowledge. Every pass asks the same question: what showed up across all my sources, and what should become a real page? The week [indirect prompt injection](https://arxiv.org/abs/2302.12173) first surfaced in my bookmarks, the agent sat down and wrote the page. The clever part is how the work gets parallelized. Hermes shipped a built-in Kanban board for agent tasks in May, and it's now the backbone of my pipeline. The enqueuers are deliberately dumb: pure Python scripts, no LLM anywhere, that scan the vault for sources no page references yet and drop one card on the board per source, up to 20 a batch. A dispatcher wakes every 60 seconds and spawns worker agents; each grabs a card, chews through its one source in three to six minutes, and writes the entity and concept updates straight into the vault. A full 20-card batch clears in about five minutes of parallel work. I'm not babysitting a queue. I'm watching a board fill itself and drain itself. (My starting backlog was 272 stranded sources. It went to zero in nine days and has stayed near zero since.) And because a knowledge base that lies to you is worse than no knowledge base at all, the thing polices itself. A custom linter runs eleven checks—broken wikilinks, stranded sources, malformed references, the usual rot—and three of the checks can fix what they find, logging every repair to an audit file. A separate pass re-checks each entity's canonical link against the live web, so a page about a company points at that company's current home and not a 404. The running log of everything the agent has done to the KB is, as of this morning, a 1.6-megabyte append-only file. It has been busy. ## The Right Model for Each Job One of the things I'm enjoying most, and learning the most from, is the deliberate exercise of matching the model to the job. The everyday chat model, as of this week, is [Hy3](https://huggingface.co/tencent/Hy3), Tencent's new 295-billion-parameter [MoE](https://en.wikipedia.org/wiki/Mixture_of_experts), which Nous is hosting free for two weeks — it took the slot StepFun's [step-3.7-flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) had held until its own free promo expired. (You may detect a pattern: the chat slot has a habit of going to whichever good model is free this month.) The heavy lifting—compiling a dense KB page out of a dozen messy sources—goes to [GPT-5.5](https://openai.com/index/introducing-gpt-5-5/) through my ChatGPT subscription's [Codex](https://developers.openai.com/codex/) OAuth, which OpenAI explicitly permits for third-party agents, so the smartest model in the pipeline bills flat-rate against a subscription I already pay for. Below that it's cheap or free all the way down: hosted open-weights models like [GLM 5.2](https://huggingface.co/zai-org/GLM-5.2), and local models on the Mini itself, where the marginal cost is electricity. You'd expect Claude in that lineup, since I live in the Anthropic ecosystem everywhere else. It's absent, and the reason is policy, not preference: since April, Anthropic classifies self-hosted-agent traffic as third-party and bills it at rack rates against "Extra Usage" instead of your plan. For a pipeline that fires every four hours, that's a non-starter. I spent my first month routing the smart jobs through a Claude Code subprocess to keep the flat-rate billing, a coding harness wrapped around non-coding work, before admitting the architecture was silly and moving to GPT-5.5. The invariant is now written into an ADR: no runtime path calls a claude-\* model. Not a complaint. Just the constraint I built around, and worth knowing if you're planning a basement agent of your own.

🎛️ The Model Roster, Apr–Jul 2026

🎯 The current lineup ▶
**Hosted:** - **Chat default: Hy3** (Tencent's 295B MoE) via Nous — currently free, and it aced the haiku audition - **Heavy compiles: GPT-5.5** via Codex OAuth — the smartest model in the pipeline, billed flat-rate against a subscription I already pay - **Mid-chain fallback: [DeepSeek V4 Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)** via Nous — the previous chat default, still on the bench **Local, on the Mini:** - **Low-stakes crons: [Qwen 3.6 35B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)** — solid, reasonably fast, a reliable tool caller, stable on the Mini - **On-demand reasoning: [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)** — the one local model that honors a reasoning-effort dial **And the fine print:** - **The fallback chain runs GPT-5.5 → DeepSeek V4 Flash** — different companies, different credential pools, so one provider's outage can't take both - **Conspicuously absent: anything claude-\*** — Opus is subsidized only inside Anthropic's own products; a third-party agent pays unsubsidized rack rates. GPT-5.5 rides my Codex subscription instead, with OpenAI's blessing.
☁️ Hosted models, over time ▶
- **Gemini 3 Flash** — April's do-everything workhorse - **Kimi K2.6** — one week as the chat model in late April - **Gemini 3.1 Flash Lite** — the chat default through early May - **Claude Opus 4.7** — a month of heavy compiles inside the Claude Code subprocess, retired with the shim - **Claude Haiku 4.5** — early cron reporter, dropped after the April 4 billing change - **GPT-5.5** — heavy-compile primary since mid-May, and the sturdiest rung in the fallback chain - **DeepSeek V4 Flash** — a spring stint as chat default, still the mid-chain fallback - **step-3.7-flash** — StepFun's entry held the chat slot through June, until its free promo expired - **GLM 5.2** — hosted open-weights; the reigning champion of the haiku-and-painting batches - **Hy3** — the new arrival in the chat slot
🏠 Local models on the Mini, over time ▶
- **Hermes 4 35B** — day one, on [Ollama](https://ollama.com/); Nous's own MoE, trained on agent traces - **Qwen 3.5 35B** — first local cron primary, via vllm-mlx; died silently one too many times and local primaries got banned by ADR - **Qwen 3.6 35B with MTP** — the redemption arc: [llama-server](https://llama.app/) built from source, benchmarked at under two seconds steady-state, five crons cut over under a new ADR that un-banned local primaries - **GLM-4.7-Flash** — served a stint as [LM Studio](https://lmstudio.ai/)'s last-resort fallback rung (and flunked today's haiku test on its way out) - **gpt-oss-20b** — kept around for hard reasoning - **Gemma 4 26B** — benchmarked in June; didn't win the slot
One more lesson from the local lane: every Hermes call carries roughly 12,000 tokens of system prompt and tool schemas before your actual request starts, and a local model grinds through that for 17 to 28 seconds where a hosted one takes one or two. Local models are for the jobs where nobody's waiting. It goes deeper than a single default, too. Each subsystem gets its own model slot—vision, compression, web extraction, the job that breaks a Kanban card into pieces, the one that writes page titles—each pointed at whatever fits its difficulty and its budget. Picking those is half the fun and most of the learning. Which, at last, is where Klank's poetry earns its keep, because the poetry isn't only for fun. A haiku turns out to be a sharp little intelligence test: strict 5-7-5 — and accurate syllable counting still trips up smaller and older models — every line self-contained with no spillover, and the whole thing still has to mean something. So when I want to size up a new model, I point it at the sea-ditty skill (Klank studies the four Japanese masters; the batches credit [Bashō](https://en.wikipedia.org/wiki/Matsuo_Bash%C5%8D)'s stillness and [Buson](https://en.wikipedia.org/wiki/Yosa_Buson)'s painter's eye) and have it write a set, paint a cover image to match, and sign the batch with whatever brain produced it: "running on stepfun/step-3.7-flash," "running on GLM-5.2 through Jack Ivers' Nous gateway." Same parrot, new brain. It's a ridiculous benchmark, it works, and the results are better than they have any right to be. It's also current: today's batches flunked GLM-4.7 and Hy3 aced its audition — the same Hy3 that just took over the chat slot. Exhibit A, number seven from today's batch of ten: ## "I Need What Snipd Does, but for Video" Podcasts were solved, thanks to Snipd. Video wasn't, and more and more of what I want is video, because a lot of the best AI conversations live on YouTube now, and as [Swyx](https://www.swyx.io/) predicted a while back, when something's on both YouTube and a podcast feed, I get more out of watching it. So I built the video version of Snipd myself. The capture step is almost nothing, and that's the point. A good bit goes by in a video, I tap the Obsidian Web Clipper, and a custom template drops one line—a URL and a timestamp—into the pending list of a snip-inbox note. That's the whole gesture. From there an ingestor takes over: it pulls a window of audio around that timestamp, runs it through [Whisper](https://github.com/openai/whisper), and judges where the topic actually begins and ends (my convention is to tap near the start of a thought, so the window looks forward). I keep the transcript with the speakers labeled, the hosts and guests, citations by segment, and a link back to the exact second. I throw the video away. (The first build used YouTube's own captions to save a step. YouTube's captions are garbage. I rebuilt it Whisper-only the same day, and the skill file now reads NO CAPTIONS, EVER.) And here's how it actually gets used: breakfast. I've got a laptop propped on the kitchen counter playing some AI talk, and I'm wandering around making breakfast for my daughter, half-listening. Something good goes by, I walk over, scrub the video back to the start of the thought, and tap the clipper. Then I go back to the eggs. The parrot does the rest while we eat. ## The Skills That Rewrite Themselves Remember the feature that intrigued me enough to tip the balance? Here's what it looks like after twelve weeks in my basement. Hermes mounts my repo's skills directory live, so the instruction files that tell the agent how to run its recurring jobs are files the agent itself can edit while it runs. When a worker figures something out the hard way, it writes the lesson back into the skill, and the next run starts from the better version. There's an agent-authored tree in the repo for the skills Klank built or grew on its own: the haiku skill, which it wrote from scratch; the Twitter-bookmark sync; a slide-deck generator that has accumulated twelve reference files of self-written lessons, with titles like "adjacent-slide-narrative-drift" and "pdf-export-diagram-verification." Nobody asked it to write those. It got burned, and it took notes. Because those edits happen in place, they show up in my repo as uncommitted changes — I call it skill drift, and reviewing it is self-learning made visible. Every morning a daily check-in posts a health report to Discord, and one section is exactly that: what the agent changed in its own skills overnight. One June morning the report showed Klank had fixed a syntax error that kept breaking the bookmark sync (a duplicated import), written the root cause into that skill's notes, and added a new batch of lessons to its haiku skill — including "don't regenerate approved batches from memory." I read the diffs over coffee and committed my agent's homework. That's the compounding bet. A normal pile of scripts is worth about the same on day 100 as on day 1. This pile is supposed to be worth more on day 100, because it has spent those hundred days teaching itself the jobs I keep handing it. Twelve weeks in, it mostly is. ## Many Brains: The Client KBs That Coexist Here's the part that turns a hobby into a tool: it isn't one knowledge base, it's six, walled off from each other on the same machinery. The big one you've been reading about is the AI-industry KB, which cleared the 1,000-page mark this week and sits at 1,009 as I write. But the exact same pipeline runs a private namespace for each of my consulting clients, plus one for my personal life. Same schema, same Kanban boards, different diet: where the AI KB eats tweets and podcasts, the client KBs mostly eat meetings. My call transcripts flow in through a connector I control, land as daily notes, and get compiled into the same kind of entity and concept pages. I flipped on five namespaces in a single May afternoon. Each runs 46 to 59 pages deep as of this morning; one of them, for a soil-data client with a lot of moving parts, went from zero to 56 interlinked pages on meeting transcripts alone. The frontier is right here, and I haven't crossed it. My personal brain needs no permissions, because the only user is me. A client's brain is a different animal. They'd love a company knowledge base, and they very much do not want the HR folder readable by anyone with the link. Doing this safely, with real access control, for a whole team, is the hard and unglamorous problem I'm circling now. It's also, I suspect, where the real value is. ## The Punch List: What I Actually Built Remember the 34,000-odd lines from the intro? Here's where they went. Hermes supplies the primitives—cron, Kanban, skills, memory, the messaging gateway. Everything below is what I built on top of them, three months of evenings and breakfasts, and every piece is in the repo with an ADR or a checkpoint behind it.

🔧 Extensions & Customizations, Apr–Jul 2026

📥 Six ingestion chutes ▶
- **Twitter bookmark sync** — pulls my bookmarks over the walled garden on a cron - **Snip-bridge** — the "Snipd for video" build: YouTube snipping, Whisper-only transcription, X-video support, a custom Web Clipper template - **Media queue** and **YouTube ingestor** — longer-form video and queued media - **Tweet-reader** — turns a single shared tweet into a captured source - **Meeting feed** — call transcripts ([Fathom](https://fathom.video/), [Fireflies](https://fireflies.ai/)) through a connector I control, deliberately agent-free so a vendor outage can't corrupt the vault; also pulls the action items out of each meeting summary and drops them into the day's note
⚙️ The KB pipeline machinery ▶
- **Kanban compile pipeline** — one card per source, a dispatcher, and parallel model-powered workers writing pages straight into the vault; the enqueuers feeding the board are deliberately dumb pure-Python scripts, a late refinement so no tokens get spent deciding what to work on - **The llm-wiki skill** — the entity/concept/comparison extraction prompt, vendored and version-controlled - **What's-new feed** — a generated page listing each new source and the knowledge pages it fed - **Vault linter** — eleven checks, three with auto-fix and an audit log - **KB toolbox** — page splitter for oversized entities, entity renamer, source cleanup and backfill scripts - **The `resource:` field** — a designed-then-shipped KB primitive tying each entity to its canonical live resource, with a resolver that re-checks links against the web
🧠 Client knowledge bases ▶
- **Multi-namespace architecture** — five client/personal KBs walled off from each other on the same machinery - **Daily-note recompile** — client pages refresh as new meeting notes land - **A scaffolder** — standing up a new client namespace is one script
✅ Task plane ▶
- **The task plane** — gets checkbox tasks from wherever they originate (daily notes, working docs, action items the meeting feed pulls from call summaries) into actionable, queryable views - **Built on the [Obsidian Tasks plugin](https://publish.obsidian.md/tasks/Introduction)** — pairs with the daily-note pattern and the client namespaces: project, priority, and source tags, with an `#inbox` sentinel so nothing captured goes untriaged - **Three versions in two weeks** — v1 extracted tasks on a cron, v2 hard-deleted them from source files behind 500 lines of safety machinery, and v3 does neither. Daily notes plus the Tasks plugin turned out to do everything: tasks stay where written, live queries render the views, and the extraction pipeline got deleted. The best version was the one that removed the code.
💬 Messaging channels ▶
- **Signal** — the original channel: a [signal-cli](https://github.com/AsamK/signal-cli) daemon wired to Hermes's gateway, so Klank texts me like a colleague - **Discord** — added later and now the main channel; a much richer platform for talking to an agent (images, threads, formatting)
🔀 Model routing stack ▶
- **Tiered routing** — the right model for each job, from GPT-5.5 down to free local models, with a model catalog as the source of truth - **Local inference rig** — LM Studio and llama-server on the Mini, benchmarked before every cutover - **Usage ledger and spend audit** — because "cheap" is a claim you verify - **The Claude Code shim** — built, health-checked, and then decommissioned when I moved the heavy compiles to Codex OAuth (its ADR reads like a tiny obituary)
🛠️ Ops and reliability ▶
- **Heartbeat cron** — every six hours, a timestamp and a haiku - **Auth self-healing** — probes for the Nous and Codex credentials, plus a repair script for when they sour - **Daily check-in** — a morning Discord report on system health and agent activity, including which skills Klank edited overnight (in-place self-edits create silent drift; this catches it) - **My own [MCP](https://modelcontextprotocol.io/) servers in the loop** — the people-knowledge and calendar Cloudflare Worker MCPs plugged into Hermes, with a running improvement punchlist fed by production use - **Fifteen cron-patch scripts** — config changes are code with tests, not clicks - **bootstrap / deploy / verify** — the whole install validates end-to-end, 9 checks
📜 The paper trail ▶
- **Fourteen ADRs** — every consequential decision written down, six of them on day one - **Seventy-plus checkpoints** — session-by-session state captures - **Thirty-seven research references** — model-landscape surveys, capability deep-dives, upgrade risk assessments before each Hermes version bump, an evaluation of Google's [OKF](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) format against my llm-wiki - **Upstream homework** — prepared materials for an MCP python-sdk token-refresh bug, and an analysis of Hermes's Kanban-orchestrator PR before adopting it - **Seven workstream docs** — living READMEs for each subsystem - **Repo-as-infrastructure** — the whole thing mounts into Hermes via external dirs, so deploying is a git operation - **458 commits** in twelve weeks
## The Fiddly Tax None of this is push-button, and I want to be straight about why. Hermes itself is solid, and what I've built on top of it is more capable than the polished managed agents I could have bought instead. But an always-on agent you run yourself is fiddly — a steady drip of small adjustments, renewals, and gotchas — and that fiddliness is the tax, the same way owning a boat means you own the barnacles. So things need tending. I keep OAuth tokens alive across a dozen services, and every so often one goes stale and a job goes dark until I re-auth it. The outside providers have their own bad days: one transcript service load-shed 503s at me for an afternoon, and another's search went down for a full day. (That's exactly why my meeting transcripts route through a connector I control, so a vendor wobbling can't corrupt the vault.) Deploying takes its own discipline, because the agent edits its skills in place: "git push" does not mean "deployed," and a careless git pull on the Mini could stomp work the parrot did while I wasn't looking. Rule one is now: check what Klank changed before you touch anything. My favorite failure, because it's so on-brand: the first time I deployed Klank's personality, the whole identity silently failed to load. The culprit was the pirate-flag emoji, 🏴‍☠️, which is secretly a black flag, a skull, and an invisible [zero-width joiner](https://en.wikipedia.org/wiki/Zero-width_joiner) stitching them together. Hermes's prompt-injection filter saw the invisible character, decided the file might be an attack, and binned the entire soul. The fix is now a written rule in that same soul file: single-codepoint emoji only. The parrot 🦜 is safe. Complex things have corners, and learning where they are is part of the deal. For the record, you can pay to make all of this someone else's problem. There's [managed Hermes hosting](https://portal.nousresearch.com/cloud) now, and simpler turnkey agents like Claude's own Cowork; they hide exactly this tax. I'm operating my own on purpose, for the power, the control, the privacy. A bit of fiddling is the price I pay. ## Where This Is Going Step back and the strange part is how small the spark was. Karpathy tossed the LLM Knowledge Base out as an idea file, not a product, and it's become one of the most useful things I've adopted in years. A lot of sharp people are circling the same ground under names like "second brain," and I think they're right that it matters. I also think most of them are underestimating the multi-namespace, runs-while-you-sleep version. For me, the personal version is finished in the only sense that counts: the refinery ran this morning, without me. The version worth real work is the team version, the same self-building, self-correcting knowledge base made safe enough to hand a whole company. That's where I'm headed next, and it's harder than anything above. In the meantime, the parrot is still down in the basement, awake at 3 a.m., eating virtual crackers, drinking from the firehose and turning it into something I can use. At 4:04 this afternoon it checked its own pulse and wrote a haiku about it. I'll take it. --- ## A Powerful Google Docs Connector for Claude **URL:** https://cto4.ai/p/mcp-mania-google-doc-writer **Date:** 2026-03-17 **Type:** essay **Description:** I built a Claude Connector that grants Claude native read-write fluency in Google Docs, over the weekend. **Tags:** ai, claude, mcp, google-docs, cloudflare, claude-code **Categories:** AI, Technology I finally got fed up with Claude's weak support of Google Docs, and Google's weak support of Markdown, and thanks to AI and MCPs, I was able to rapidly do something about it. ## The Pain Points Working with Google Docs from Claude is a major pain. Anthropic has the [Google Drive connector](https://support.claude.com/en/articles/10166901-using-the-google-drive-integration), which is read-only (not useful). Anthropic also has the [File Creation skill](https://www.anthropic.com/news/create-files) for DOCX files, which as its name implies is focused on the Microsoft format, and only supports Google Docs through a **double** import process. The first import creates a "DOCX-flavored Google Doc" (why, Google, why?), which Claude won't read (why, Anthropic, why?). One must then Save As Google Doc, so Claude can successfully ingest the file later. Google also has their own PITA Zone: Google Docs is brain dead when it comes to Markdown support. Markdown is AI's lingua franca, and Google Docs can neither import nor paste it. Unfortunately, the [Google Workspace Marketplace](https://workspace.google.com/marketplace) is a disaster, because there are Markdown add-ons there, but good luck getting them installed and working. In the words of [Howard Beale](), > I'm as mad as hell, and I'm not going to take this anymore! ## The Solution Not long ago, I created a [Claude connector + skill pair](https://cto4.ai/p/claude-eyes) that grants Claude native fluency in Excalidraw. It's been [highly useful](https://cto4.ai/p/mcp-apps-in-context). I decided to create a similar solution for Google Docs: granting Claude, who is already highly fluent in Markdown, comparable read-write fluency in Google Docs. (Docs is actually just Phase 1---I'm planning to add support for Sheets and Slides too, and mulling the idea of some kind of "Markdown Hub" as well, for other terrible Markdown citizens such as Slack, which also can't accept pasted Markdown.) I started this project Friday evening. I had the Google Docs phase fully functional Sunday evening. And I was very busy with other things all weekend; my total project working time was maybe 5 hours. ## My Best MCP So Far This is my sixth Claude Connector (aka Remote OAuth MCP) since December. All of them run on the same core stack: [Cloudflare Workers](https://developers.cloudflare.com/workers/), Durable Objects, KV, the [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/), and Google OAuth via OAuthProvider wiring. I landed on Cloudflare because they seem to be well ahead of the pack in the MCP hosting space, and they're also a thought leader on building fast, efficient MCPs. I didn't just clone one of my previous MCP projects this time around. Instead, I took the opportunity to upgrade to the very latest versions of all the component parts, as well as rethink and refine my previous architectural decisions. To wit: - I did a major upgrade of the Cloudflare Agents SDK, their framework for building stateful, long-lived AI agents as Cloudflare Workers. I jumped from Agents `0.0.72` to the completely rearchitected `0.7.6`, eliminating some ugly postinstall patches that had been needed for Claude Connector compatibility. - I went all-in on the Cloudflare "[Code Mode](https://blog.cloudflare.com/code-mode/)" pattern. The MCP has just two tools, `search` and `execute`, and a tiny ~1000 token MCP context overhead. - When you are teaching Claude a new language, in this case Google Doc speak, one important function of the connector is to provide live, searchable documentation (via the `search` tool). This time I used Workers KV for the content chunks, which enables domain knowledge updates without redeploying the worker. ## Powerful Google Docs ↔ Claude Connection I also said "yes" a lot when working with Claude Code to plan the Google Docs functionality: - **Bidirectional Markdown to Google Docs conversion**, which we built from scratch, since we couldn't find an npm library that fit. This involves non-trivial table index math and the like. Clean Markdown->Google Doc and Google Doc->Markdown, works very nicely. - **Code blocks as styled tables:** Google Docs has no native code block element, so we elected to render code blocks as 1×1 tables with monospace font and gray background shading. We also detect them on the way back, via that same monospace font + gray background signal. - **Reverse-index batch processing:** The Google Docs API's `batchUpdate` endpoint shifts all downstream indexes when you insert or delete text. We worked out an approach where deletes run before inserts before formatting, and operations are ordered by descending index so earlier operations don't invalidate later ones. - **MCP server instructions as guardrails:** The connector embeds editing strategy guidance directly in the MCP server instructions, e.g., "use `find_replace` for small changes, do NOT use `replace` for minor edits as it destroys all existing formatting." This steers Claude toward safe operations without requiring the user to know. - **Empirically verified table index formula:** The Google Docs API documentation doesn't explain how table cell indexes work. Claude derived the formula (`tableStart + 3 + row*(1+2*cols) + col*2`) through trial-and-error against the live API, then locked it down with tests. ## Experimenting with a New Claude Code Process Besides an aggressive tech stack upgrade and feature set, I came at this project in a different mode of operation. First off, Anthropic just [unlocked the 1M context window](https://winbuzzer.com/2026/03/14/anthropic-drops-long-context-premium-1m-token-claude-xcxwbn/) in Opus 4.6 at no additional cost. That's a pretty huge change; as I've seen many an AI coder observe, the previous 200K context tends to run out "just when things are getting good." I was able to _complete the entire Google Docs phase of the project_ without context being compacted, that is, without Opus losing its mind. The other major mode change was that I used the [Claude Code Superpowers Plugin](https://github.com/obra/superpowers) from Jesse Vincent, which I had heard good things about but hadn't had a chance to try. The process with Superpowers was significantly more structured and Test Driven Development (TDD) oriented: - **Structured brainstorming with parallel research agents:** Before writing any code, Claude dispatched 4 research agents simultaneously — covering package versions, Cloudflare patterns, MCP protocol spec, and Google API prior art. This tech audit informed all 13 design decisions. - **Deliberate decision gates:** Claude asked each architectural question one at a time with options and a recommendation, and wouldn't proceed without explicit approval. This prevented bad defaults from getting locked in early. - **Subagent-driven execution:** Plan 1 (infrastructure) was broken into 9 tasks, each dispatched to a fresh subagent for implementation while the main orchestration agent coordinated. This kept the main context clean and let mechanical work run fast. - **Live multi-surface testing:** Tested simultaneously from Claude.ai, Claude Desktop, and Claude Code, which surfaced real integration bugs (OAuth colon parsing, CORS headers, table index math) that unit tests couldn't catch. Giving Claude Code access to the MCP itself enabled extreme-speed smoke testing, probably 100X my human speed. - **Real content as test material:** Instead of lorem ipsum, we used the MCP to create actual useful documents during testing, serving double duty as QA and real output. Here's an [example Google Doc](https://docs.google.com/document/d/1J52FHhke3WmBtCOBcXXeeB3pjHmzRcM2VASOGnQFr4Q/edit?tab=t.0#heading=h.fr70oqa08roh) created entirely by Claude using the connector. ## This is Freaking Insane A novel, state-of-the-art Claude Connector (remote OAuth MCP), with an advanced feature set that beats everything Anthropic offers and Google doesn't offer, completed over the weekend, with minimal human screen time. This is crazy. The [Awakening](https://cto4.ai/p/december-awakening) continues. --- ## December Awakening **URL:** https://cto4.ai/p/december-awakening **Date:** 2026-03-08 **Type:** brief **Description:** Everything accelerated, again, in Nov-Dec-Jan. Lots of people are noticing. **Tags:** claude, anthropic, claude-code, ai-trends, inflection-points **Categories:** AI, Technology We passed some kind of crazy milestone recently, one that is only becoming obvious in retrospect. I felt it, and wondered if others had sensed it too. It turns out they had. ## What I've Been Saying - In [Leveling Up Claude Code with Boris Cherny's Help](https://cto4.ai/p/leveling-up-claude-code-with-boris-chernys-help/): > Claude Opus 4.5 arrived in November, powering up Claude across all its surfaces—Claude.ai, Claude Desktop, and Claude Code. So smart, such encyclopedic knowledge. - In [The Workflow](https://cto4.ai/p/the-workflow/): > Moving from "I wish I could do something about X" to "I can definitely now do something about X—and Y and Z and A-B-C," is a huge deal. AI is the enabler, but more specifically, the November arrival of Claude Opus 4.5 was, for me, the tipping point. That model crossed some kind of "smart enough" line where there was literally no workflow challenge it could not take on. (Things are now even better with Opus 4.6.) ## What Others Are Saying I’ll put [Doug O’Laughlin](https://www.linkedin.com/in/dougolaughlin/) from [SemiAnalysis](https://semianalysis.com/) up front because he's the one who calls this phenomenon "The Awakening" in this [Latent Space podcast](https://youtu.be/x9rWFiIubmc?si=pkAUezBfsbdgVAvi&t=753) with [Shawn "Swyx" Wang](https://www.swyx.io/): O’Laughlin is also the guy who detected and charted the [insane trend](https://newsletter.semianalysis.com/p/claude-code-is-the-inflection-point) whereby Claude Code co-authored commits now represent a substantial and exponentially-growing percentage of all commits on GitHub: Swyx himself covers the phenomenon thoroughly in [this Latent Space AI news post](https://www.latent.space/p/wtf2025): Swyx even created a dedicated site for it, [wtfhappened2025.com](https://wtfhappened2025.com/): [Ethan Mollick](https://en.wikipedia.org/wiki/Ethan_Mollick) [wrote on LinkedIn](https://www.linkedin.com/posts/emollick_so-claude-45-came-in-far-above-trend-in-activity-7407974595477282816-IRz7) in December about Opus 4.5's off-the-chart performance on METR benchmarks: Here's [METR](https://metr.org/)'s [updated chart for Opus 4.6](https://metr.org/time-horizons/), yet another exponential outlier: [Andrej Karpathy](https://en.wikipedia.org/wiki/Andrej_Karpathy) [on Twitter](https://x.com/karpathy/status/2026731645169185220): [David Heinemeier Hansson](https://en.wikipedia.org/wiki/David_Heinemeier_Hansson) (DHH) [responding to Karpathy](https://x.com/dhh/status/2026740100382482674?s=20): [Boris Cherny](https://www.linkedin.com/in/bcherny/) [on Twitter](https://x.com/bcherny/status/2004897269674639461?s=20) in December: Pretty convincing evidence, eh? ## My Take on What's Powering This - The arrival of Claude Opus 4.5 in November began the exponential - Some kind of significant Claude Code agent harness improvements in December ratcheted up the curve - Opus 4.6 in January, even more so --- ## MCP Apps: Custom UIs Inside Claude **URL:** https://cto4.ai/p/mcp-apps-in-context **Date:** 2026-03-08 **Type:** essay **Description:** MCP Apps bring your custom UIs inside Claude. I built 4 and share what I learned: how they relate to MCPs and skills, rules of thumb, and more. **Tags:** mcp, mcp-apps, claude, anthropic, skills, claude-code, obsidian, developer-tools **Categories:** AI, Technology, Crafty CTO MCP Apps---a way to add our own custom UIs inside Claude---arrived as a standard on January 26th. MCP Apps work inside all of the in Claude surfaces that actually have UIs, basically everything but Claude Code, now that Claude Cowork supports them. MCP Apps also work on OpenAI and several other platforms, but we'll focus on the Claude world here. ## How We Got Here On December 9th, [Anthropic announced](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation): > Today, we’re donating the Model Context Protocol (MCP) to the [Agentic AI Foundation (AAIF)](https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation), a directed fund under the [Linux Foundation](https://www.linuxfoundation.org/), co-founded by Anthropic, Block and OpenAI, with support from Google, Microsoft, Amazon Web Services (AWS), Cloudflare, and Bloomberg. Then, just over a month ago (Jan 26th), the [MCP core maintainers announced](http://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/): > **MCP Apps are now live as an official MCP extension**. Tools can now return interactive UI components that render directly in the conversation: dashboards, forms, visualizations, multi-step workflows, and more. This is the first official MCP extension, and it’s ready for production. MCP Apps actually brought together work from OpenAI ([OpenAI Apps SDK](https://developers.openai.com/apps-sdk/)) and the independent creators of MCP UI including [Ido Salomon](https://www.linkedin.com/in/ido-salomon/) and [Liad Yosef](https://www.linkedin.com/in/liadyosef/), with Anthropic co-authoring the MCP Apps spec in November: > MCP-UI and OpenAI Apps SDK pioneered the patterns that MCP Apps now standardizes. The projects proved that UI resources can and do fit naturally within the MCP ecosystem, with enterprises of all sizes adopting both the OpenAI and MCP-UI SDKs for production applications. Claude supported MCP Apps from day zero, January 26th. ## Cool, But ... I was excited about MCP Apps joining the already-exciting lineup of Claude extensions that enable us to distill our business domain knowledge (skills) and connect to our business context (MCPs). But honestly, I wasn't exactly sure about their real-world applications. The announcement included one of Anthropic's brief demo videos showing off the Figma MCP app, and this helpful bit: Great, that's kinda helpful, but ... this is so high-level, so far from me understanding how to architect and build MCP Apps---and the MCPs they live inside---let alone how MCP Apps should relate to the other members of the team, most importantly skills. And let's not forget, how do you deploy these guys? ## Learning by Building Claude Code has made me fearless, especially since December ([The Awakening](https://cto4.ai/p/december-awakening)). Build something that actually works using a new technology whose paint is still wet? No worries. Figure out how to deploy that thing in a production-grade cloud environment? Yep, bring it on. So off I went. ## Four MCP Apps So far I've built four MCP Apps, across four MCPs: | MCP | App Name | Initial Version | Last Major Update | | -------------------- | ------------------- | --------------------------------------------- | ------------------------------------------------------ | | **Obsidian** | Task Triage | **Feb 3** — task dashboard | Still evolving (Meeting Todos added Mar 2) | | **Calendar** | Meeting Prep | **Feb 9** — schedule view + prep scheduling | Feb 26 — cross-day slots, fullscreen, Done button | | **People Knowledge** | Past Interactions | **Feb 12** — email dropdown + meeting history | Feb 23 — action items capture, fixes through Mar 3 | | **Excalidraw** | Interactive Preview | **Feb 16** — SVG preview | Feb 26 — fullscreen + client-side Excalidraw rendering | Obsidian was first — just 8 days after MCP Apps launched on Jan 26. It went from planning docs to a working task dashboard in a single day (Feb 3, afternoon). The other three followed over the next two weeks, all reaching a stable state by late February. ## Two Skills Two companion-orchestrator skills emerged alongside these four MCP Apps: | Skill | Purpose | First Version | Latest Update | | -------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | ---------------------------------------------------- | | **day-prep** | Orchestrates Meeting Prep, Past Interactions, and Task Triage into a day-planning workflow | **Feb 9** (v1.0.0) — same day as Calendar MCP App | Mar 2 — v1.4.0, added meeting todos passthrough | | **excalidraw** | Teaches Claude how to create Excalidraw diagrams; orchestrates use of the Interactive Preview app | **Feb 16** (v1.0.0) — same day as Excalidraw MCP App | Feb 25 — v1.2.0, label shorthand, slimmed references | Coincidentally, both skills are in their eighth iteration. ## Excalidraw: Skill + MCP + MCP App Let's start with Excalidraw, because it's more self-contained than the three-app Day Prep flow we'll look at next. The combination here of Excalidraw skill, MCP App, and host MCP: - Teach Claude how to "program in" Excalidraw - Give Claude the tools needed to create, edit, load, and save [Excalidraw](https://excalidraw.com) diagrams hosted in [Obsidian](https://obsidian.md) - Give Claude visual feedback---a render of the diagram---enabling the model to iterate on and improve its work Here's a flow diagram of the process---built entirely by Claude using these tools: Note that the Obsidian MCP is also on the diagram here, because I host my Excalidraw diagrams as Obsidian markdown. Saving saves to Obsidian. Here's the MCP App in action, previewing this same diagram inside Claude.ai in Chrome: I cannot overstate how useful "teaching Claude to Excalidraw" has proven to be, with the MCP App an important piece of the puzzle. ## The Day Prep Flow and MCP Apps I'm grateful to my fractional CTO clients for their real-life AI workflow wants and needs. One badly wanted Claude to help them get ready for their day, and that led to the `day-prep` flow, which I built for my own particular flavor of day preparation, using my own mix of tools. Day Prep is three MCP Apps wrapped inside, and orchestrated by, a Claude agent skill. Here's a flow diagram, again Claude-created: ## Day Prep User Experience Below you can see the UIs of the three MCP Apps that the `day-prep` skill orchestrates. They're neither beautiful nor fully thought out, but I find the flow quite useful and effective even so. (Just click on smaller thumbnail images to see them full-size.) ### Day Prep Kickoff (Skill Wrapper) When the `day-prep` skill kicks in, here's the model interacting with us: ### Meeting Prep (MCP App) Meeting Prep leads things off, showing us our meetings for today, or optionally up to three additional days in the future. Besides meeting awareness, it allows us to block off times for meeting prep. I use [Reclaim.ai](https://reclaim.ai) to sync five calendars, and since it has a decent API with several interesting capabilities beyond reading calendars (e.g. time blocking), Reclaim is able to serve as a single source for meeting details and time blocking across five calendars. Behind the scenes, Meeting Prep uses the MCP App `updateModelContext` method to hand Claude the emails of participants in my upcoming meetings, necessary data for the next step of the orchestrated Day Prep workflow. ### Past Interactions (MCP App) Once Meeting Prep is done, the model loads the Past Interactions app, passing in the emails of meeting participants. When I select someone, the app locates our past interactions in the form of recorded meetings ([Fathom](https://www.fathom.ai/), [Fireflies](https://fireflies.ai)) and email threads. For meetings, I can click my way from the initial brief meeting summary, to extended details, to the rather good full-length meeting recaps that both Fireflies and Fathom provide. Both platforms also capture meeting action items, which I can capture for the task planning that is the next step in Day Prep. ### Task Triage (MCP App) Once I've finished reviewing past interactions, the model loads up Task Triage, passing in any action items I captured in the previous step. I use Obsidian with the Task plugin to manage my tasks. The Task Triage app is actually a 3-step mini-flow within the larger Day Prep flow: - **Review:** In my crazy little system of task management, #Today tasks are what I'm working today, and tomorrow that list of #today items gets reset and reconsidered. That's the first step of Task Triage: pick which of yesterday's #Today tasks should get pulled forward into today. - **Meetings:** If I captured any Fathom/Fireflies action items during the prior step, next I can selectively add them as new Tasks. - **Plan:** Finally I sift through the remainder of my tasks, which are prioritized #T0 through #T3. I can add tasks to #today or otherwise re-prioritize. ### Day Prep Summary (Skill Wrapper) The Day Prep skill, once it gets the message that Task Triage is complete, pulls together a bit of a summary. So that's it for Day Prep. Ugly, experimental, full of warts: and surprisingly useful. ## Architecting and Building MCP Apps You can't architect or build MCP Apps without also considering the architecture of their close siblings, MCPs and skills. You have to start with an overall understanding of the full landscape, as well as the constraints inherent to our current technology. Consider these facts and their implications: - An MCP App "lives inside" a parent MCP. So the limitations of MCPs are inherited by their MCP App children. - The biggest limitation? MCPs don't compose: we can't assemble several component MCPs to create an uber-MCP. Each MCP is on its own to connect with the sources of business context it needs. - Since every MCP App lives inside a single MCP parent, the implication is this: every context source that an MCP App needs must be connected to its parent MCP. It can't come from some other MCP. This is especially painful when the other MCP you'd like to use comes from another company and isn't open source. Fireflies, for example, has an official MCP, but it's closed source. So we can't build an MCP App that takes advantage of the official Fireflies MCP. Too bad, so sad. - But: APIs to the rescue. We _can_ connect APIs to our parent MCP, and then use those APIs to get the context we need. Fireflies has an API; and Fathom, which lacks an MCP, does have an API. It's a bit more work than just assembling a collection of MCPs, but remember---code is cheap these days, and getting cheaper all the time. With Claude Code doing most of the work, it's not that painful to write a quality MCP that hits APIs like those from Fireflies and Fathom. - But but: it **is** possible to compose MCPs together as part of a larger workflow. Guess who has that power? The model. Claude e.g. Opus 4.6, running inside Claude.ai or Claude Desktop or Cowork or Claude Code, can and will assemble whatever combination of MCPs it has available to solve whatever problem you give it. [Claude Cowork](https://cto4.ai/p/claude-cowork-supports-plugins), with its next-generation customizable plugins, will even walk you through the process of selecting and adding the MCPs you need for a particular knowledge work task. - But but but: the model has a critical limitation: text output only. Now, that text can be code, and that code can have a UI. But I've found those one-off, on-the-fly React component UIs to be pretty hard to use, and the idea of "always have the model build a new UI app from scratch when you need to do X" seems pretty nutty. Skills can include helper scripts, but again, the model has to do a lot of work to present some kind of UI other than text. In the end, it all comes down to what's good enough for a particular workflow. Many times text-only, or on-the-fly UIs, can be perfectly ok. If so, just use the model and skills, and you'll get the benefits of "MCP composition"---the model will assemble and use whatever MCPs it needs to get the job done. ## Pirate Jack's Rules of Thumb **Don't** Default to building an MCP App without considering carefully whether "let the model do it" might work. If you can live with text-only output, start there. **Do** When you build an MCP, consider that you might add one or more MCP Apps at some later point. Will you have the business context you need? **Do** Organize your MCPs around general business domains. Neither tiny, micro-sized, single-purpose MCPs, nor a giant monolith-does-everything MCP. My best example is the [People Knowledge MCP](#the-day-prep-flow-and-mcp-apps), whose domain is "knowledge about people that I interact with." I started with meeting assistant artifacts (transcripts, summaries) and email threads. But this could logically extend to include a CRM connection and, via computer-use tooling, accessing LinkedIn and similar SaaS research tools. Among other benefits, such a general-domain MCP will have everything needed for a good MCP App if needed, such as [Past Interactions](#past-interactions-mcp-app). **Do** Build good MCPs. Remember that 90% of the MCPs in existence today are crappy API-wrappers that guzzle context like frat boys guzzle beer. Keep the tool count low, and never wrap an API to create a tool unless the fit is absolutely perfect. Consider using Cloudflare's [Code Mode](https://blog.cloudflare.com/code-mode-mcp/) MCP architecture, where two tools, `search` and `execute`, handle everything and model context overhead is a mere 1000 tokens. **Do** Default to remote OAuth style MCPs (aka Claude Connectors). Avoid taking the local-only `stdio` MCP shortcut, it won't scale and isn't worth it even for proof of concept work. Get robust authentication in place day one, it's not hard and we're far enough down the road now that it generally "just works." **Do** For hosting your remote MCPs, strongly consider cloud platforms that have been early pioneers in architecting and hosting cloud MCPs. [Cloudflare](https://developers.cloudflare.com/agents/guides/remote-mcp-server/) seems to be well ahead of the pack here, and I've had very good luck with them. **Do** Leverage skills. Skills can guide and teach the model, supporting more use cases without custom-coding MCP Apps. When you do have MCP Apps, skills can become an orchestration layer for and among those apps as with the [Day Prep flow](#the-day-prep-flow-and-mcp-apps). **Do** By all means, write MCP Apps when it looks like "just interacting with the model" won't be good enough. In all four of the MCP Apps I created, I believe my choices were correct: the experience would have been substantially less effective without injecting UI into Claude. In two of the cases (Task Triage and Excalidraw Preview), MCP Apps were absolute must-haves. **Remember** If you want to use a particular feature from the MCP or MCP App spec, remember that not all MCP clients---e.g. Claude.ai, Claude Desktop, ChatGPT, etc.---support every feature in the MCP and MCP App specs. Even Anthropic, the creator of MCP, doesn't yet support some of the most powerful features contained in the MCP spec. One example: the horribly-named "[sampling](https://modelcontextprotocol.io/specification/2025-06-18/client/sampling)" feature, which allows an MCP to make its own requests directly to the model, bringing the full intelligence and power of the model inside the MCP. I could have done amazing things in each of my four MCPs with that feature, but alas none of Anthropic's products support it yet. **Do** Learn by building! --- ## Helping Claude See ... Diagrams **URL:** https://cto4.ai/p/claude-eyes **Date:** 2026-02-20 **Type:** essay **Description:** I built an MCP that lets Claude see its own Excalidraw diagrams, iterate on them, and save them to Obsidian. **Tags:** claude, anthropic, excalidraw, mcp, obsidian, diagramming **Categories:** AI, Technology If you want your AI model to succeed at tasks, give it a way to check its work. Some of the most impressive feats in AI coding have been cases where a complete and testable specification existed, such as Simon Willison's [porting of the JustHTML project from Python to JavaScript](https://simonwillison.net/2025/Dec/15/porting-justhtml/). I've been focusing a lot of attention on making applications fully testable across all my various AI coding projects. ## What If Claude Could Draw? I love the [Obsidian](https://obsidian.md/) markdown-for-everything tool, using it for notes, bookmarks, task tracking, podcast snip captures---and diagramming, thanks to Zsolt Viczian's excellent [Obsidian plugin implementation](https://excalidraw-obsidian.online/Welcome) of the awesome [Excalidraw](https://excalidraw.com/) tool. I've already built an Obsidian MCP and skill to give Claude access to that platform. It dawned on me that maybe I should also do something with Excalidraw. I figured a strong model like [Claude Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6) would be perfectly capable of generating Excalidraw diagrams, whose JSON format is well documented. And that would be great: nothing against ASCII diagrams, but Excalidraw diagrams are an order of magnitude more powerful. If I could bring Excalidraw fluency to all Claude's surfaces---Claude Code, Claude.ai, Claude Desktop, [Claude Cowork](https://cto4.ai/p/claude-cowork-supports-plugins)---that would be huge. And it would play well with Obsidian. At the same time, though, I doubted that Claude could create **good** Excalidraw diagrams unless it could "see" visual renders of the JSON it coded. Equip Claude with renders, though, and it would be able to iterate. Since I was already on a "give the AI model what it needs to validate its work" kick, I decided to look into how I might return visual diagram feedback to Claude. Working closely with Opus 4.6 in Claude Code, I explored various approaches. ## Borrowing an Idea From QA Automation In the QA automation space, QA harnesses (increasingly AI-driven these days) use browser automation tools to navigate through the app UI and test functionality. Browser automation gives the model access to the browser's internal page representation (the DOM), but in many cases today's vision-enabled models can do better by "looking"---at screenshots. This concept is what Opus and I settled on: the model creates its initial Excalidraw JSON, ships it off to a Cloudflare worker connected as an MCP server, which then renders the diagram using a headless browser, snaps a screenshot, and sends it back to the model. Iteration enabled. Here's a flow diagram, created by Claude in Excalidraw: ## Claude.ai Inline Preview Using an MCP App Another kick I've been on is understanding [MCP Apps](https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/), which are a recently-standardized joint development between Anthropic and OpenAI that allow MCPs to deliver UI elements right inside ChatGPT or Claude. This is the successor to OpenAI's ChatGPT apps and was donated, like the MCP standard itself, to the [Agentic AI Foundation](https://aaif.io/). I've been working to get my head around what is and isn't possible with MCP Apps, and how they fit into an overall architecture that also includes agent skills and traditional MCP tools. Since Claude will often be pairing with a human while creating Excalidraws, I needed a way for the human to also see interim diagram renders. A simple MCP App filled this need nicely: ## Nice Plumbing, Mrs. Loopner I implemented this thing for my own use, but built it to be approximately ready for production use inside larger organizations, such as my CTO clients. In such a scenario, I'm a big believer in make-it-work-across-all-Claude-surfaces, which _includes_ Claude.ai and therefore _precludes_ quick-and-dirty local MCP solutions. Instead, I build "Claude Connector" pattern remote OAuth MCPs that authenticate with an identity provider like Google or [MS Entra](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id). I consume Excalidraw inside Obsidian, across a couple of Macs, so I went the extra mile to make that seamless: I can load from, and save to, my synchronized Obsidian vault. My Macs are dev machines and I've chosen not to make either of them act as a server. Since Obsidian offers cloud sync but not cloud storage or a cloud API, I had to figure out a way to effectively roll my own Obsidian cloud. Fortunately, I have an always-on [Synology](https://www.synology.com/) NAS, and was able to set it up as a tiny Obsidian cloud server. This combination led to somewhat complex plumbing---another diagram made by Claude using the Excalidraw MCP: With Claude Code's help, though, I was able to tackle this without too much difficulty. Insane, because I would never have dared attempt this even a year ago, let alone in 2022. ## Wrap it up in a Skill I wanted to keep the MCP as lightweight as possible, with a limited set of carefully-crafted tools and an MCP App for inline previewing. But there are things the model really needs to be reminded about ("always check your work!"), and details that should be available as needed without adding a ton of context overhead ("Excalidraw Style Guide"). This ended up being a perfect fit for an agent skill: ## It's Actually Really Useful I've been known to tackle this kind of project just as a learning exercise, but the Excalidraw MCP + Skill + Obsidian integration is already proving its worth. And I can use it from Claude Code, or Claude Desktop, or Claude Cowork, or Claude.ai ... The cover art for this post itself is a Claude-generated diagram: --- ## The Short, Sweet Life of Clawdbot **URL:** https://cto4.ai/p/short-sweet-clawdbot **Date:** 2026-02-17 **Type:** brief **Description:** Peter Steinberger's move to OpenAI marks the end of the saga; or Part 1 anyway. **Tags:** open-source, ai, anthropic, openai, clawdbot, openclaw **Categories:** AI, Technology The short, sweet life of [Clawdbot](https://en.wikipedia.org/wiki/OpenClaw) is over. On Saturday February 14th, Peter Steinberger [announced](https://steipete.me/posts/2026/openclaw) that he's joining OpenAI. The Clawdbot => OpenClaw saga---Part 1 at least---has run its course. I'm not trying to be overly dramatic, and I'm not saying this is a bad thing. The opposite, in fact: something had to give, and if you want a Clawdbot-like set of capabilities to scale, to become reasonably and widely available (I do), with a shot at remaining a viable open source entity (this too), Steinberger landing at OpenAI in the way he's doing it is pretty close to an ideal outcome. ## Unsustainable, Yet ... Why did something have to give? If you listen to his [3 hour 19 minute interview with Lex Fridman](https://share.snipd.com/episode/bb68951b-9f84-407c-84cd-4402b708a04b) (recommended), it's clear that Steinberger's situation was unsustainable. He was losing money keeping Clawdbot => OpenClaw alive and growing. His quality of life was suffering. He was being stalked by crypto-bros, VCs (near neighbors!), and the press. The situation got bad enough that he considered just walking away at several points. Yet all the while this insane and wonderful thing was emerging: Steinberger had connected up the ["Opus 4.5 moment"](https://www.anthropic.com/news/claude-opus-4-5) frontier AI models in ways that nobody else had thought to, and OMG did it hit a sweet spot. Record-setting, nerds-and-normies adoption; a virtuous cycle of adoption-fueled and community-led expansion that exposed new emergent AI possibilities. The momentum gained in such a short time was unprecedented. Steinberger didn't want to kill this off, didn't want to leave the community high and dry. ## Ideal-ish Outcome Why do I think this outcome was close to ideal? Steinberger isn't just being acqui-hired; he and OpenAI have apparently agreed to keep OpenClaw open source under the umbrella of a new foundation, much like Anthropic did when it donated the Model Context Protocol to the [Agentic AI Foundation](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation), and provide funding for the new entity. This, combined with the meteoric uptake Clawdbot => OpenClaw has seen, gives the project a decent shot at life. There are no guarantees in open source---people vote with their feet--- but this is as good a shot as a very young project can hope to have. Based on my 3 hours and 19 minutes with Peter and Lex, I came away really liking Steinberger and I'm fully aligned with his positions on a number of topics, including: > [I don't do this for the money. I don't give a f---.](https://share.snipd.com/snip/9db62a10-2e51-49cf-836b-8b661191dda6) He could have cashed out in 100 different ways, the most obvious being creating a VC-funded, OpenClaw-focused startup. But he opted for a course that will almost certainly put less money in his pocket (though plenty, I'd bet!), but will also give the OpenClaw project and community a good shot at thriving. ## Winners and Losers OpenAI gets the big win here; they get to bask in the momentum and good vibes of a project named for their arch-competitor, and they capture arguably the best person in the world to bring a Clawdbot-like offering to the OpenAI ecosystem. We---normies and geeks alike---also win. - Steinberger mentions that he was also talking to Meta. OpenAI is much better for us: they can execute and are a couple steps lower on the bad-guys ladder. Count our blessings that the Musk-hole wasn't in the running. - Foundation plus OpenClaw remains open source with deep-pockets funding. Win. Finally, Steinberger wins. I'm happy for him. He connected the dots in a new and surprising way, he worked really hard, he kept to his principles. I'm happy he is able to get what he really wants as a result. He and I share some fairly specific background: experience building and exiting startups, success in the Apple iOS ecosystem, being drawn back into coding and building things by the AI emergence, amazement and fascination (as opposed to fear and angst) at what is emerging. So, good for him, I'm really happy for him and pleased he didn't just cash out. The loser? Anthropic. Guys, maybe you avoided the obvious mistake of sending Steinberger a lawyer letter around the Clawdbot name issue, but you still managed to deeply alienate him. This amazing Clawdbot thing was named after your great Claude model, a model that exemplifies the April-to-November 2025 model revolution, which made Clawdbot possible. How do you manage to blow this? How do you come out looking like the bad guys? I know you aren't; still love you; but maybe fire whoever was giving you legal and PR advice on this one. ## Looking Forward to Part 2 This should be fun to watch. --- ## I Can Now Improve, Automate, Fix and Document Everything **URL:** https://cto4.ai/p/the-workflow **Date:** 2026-02-10 **Type:** essay **Description:** Thanks to The Workflow and my AI pair, Claude Opus 4.6. **Tags:** claude, anthropic, ai, workflow, productivity, automation, obsidian **Categories:** AI, Technology I kicked off a [recent post](leveling-up-claude-code-with-boris-chernys-help) about leveling up my Claude Code game as follows: Moving from "I wish I could do something about X" to "I can definitely now do something about X---and Y and Z and A-B-C," is a huge deal. AI is the enabler, but more specifically, the November arrival of Claude Opus 4.5 was, for me, the tipping point. That model crossed some kind of "smart enough" line where there was literally no workflow challenge it could not take on. (Things are now even better with [Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6).) Below I'll outline how I've paired with Opus to tackle The Workflow. ## The Workflow has a Git Repo When I realized that I could start collaborating with Claude on The Workflow, one of my first steps was creating `the-workflow`, a Git repository for the overall project, with numerous folders for individual subprojects. This turned out to be a very important step, because, in following good Claude Code practices for a repo, I created a CLAUDE.md with broad information about all my Workflow initiatives, and in each subproject, a well-stocked `/docs` directory. The result: the Claude Code that I run inside `the-workflow` repo is insanely smart about all aspects of my setup. ## Shut Your Pi-hole I use [Pi-hole](https://pi-hole.net/) running on a [Raspberry Pi](https://www.raspberrypi.com/) to block ads on our home network. Sometime in the past month, I noticed my iPhone began getting ads, while on the home network ... what the heck? I had a little chat with Workflow Claude, and figured out that my recent upgrade to iOS 26 resulted in a new default setting "Limit IP Address Tracking" that bypassed local DNS and prevented Pi-hole from doing its thing. Toggled off, fixed. I also took the opportunity to capture Pi-hole setup documentation. One benefit of pairing with Claude Code in a terminal is that it has SSH access, so it was able to SSH in to the Pi and capture lots of details about the config. I also fed it some documentation I had done in Obsidian. So now Workflow Claude knows all about Pi-hole, my DNS setup, as well as other problems and upgrades on the Pi-hole todo list. ## Connecting Obsidian to All Claudes I use Obsidian across all my machines and devices for note-taking, bookmarking, task tracking, podcast snip librarying, and more. I figured I should connect Obsidian to Claude so I could put Claude to work on finding, organizing, and creating Obsidian content. I went a bit crazy on this project, deciding I wanted **all** Claude surfaces including Claude.ai to have Obsidian access. But Claude.ai lives in the cloud, not on my local computer and network like Claude Desktop and Claude Code, how could I get this to work? With Workflow Claude's help, we worked out an architecture that involved: - A Docker-based Obsidian instance running on my [Synology](https://www.synology.com/) NAS, along with the Obsidian REST API - A [Tailscale](https://tailscale.com/) funnel into that NAS using my personal Tailscale network - An authentication proxy that blocks unauthenticated traffic to Obsidian (an unlikely occurrence since Tailscale should prevent that) on the NAS - Since I learned how to build Claude Connectors (OAuth MCPs) as part of the [Skillport](skillport) project, an OAuth MCP to authenticate into the NAS - A Claude Obsidian Skill that helps Claude understand how to reach Obsidian content ### Crazy, but Like a Fox I say I went a bit crazy here---well, not really. One of my main responsibilities as an AI-forward CTO is to know the boundaries of what is and is not possible with AI. The overkill here in fully connecting Obsidian with all Claude surfaces was also an opportunity to see how smooth or rough this kind of multi-layer implementation would go with current-generation models like Opus 4.5 or Gemini 3 Pro. And guess what, it works! Obsidian from any Claude, including Claude.ai. ## Backups; or "How Did I Set That Up??" A couple years before I implemented The Workflow, I spent a good bit of time getting my backups in order, applying the 3-2-1 backup rule: - 3 copies of your data - 2 different storage media/types - 1 offsite (or offline) copy I fought through trying to get macOS Time Machine working across the network to my NAS (works great, until it inevitably and catastrophically fails), trying Synology's HyperBackup client (hell no, I'm not giving you those privileges on my Macs), and many other iterations and experiments until I finally got something that worked reliably ([Kopia](https://kopia.io/) to S3-compatible volumes on the NAS and [Backblaze](https://www.backblaze.com/)). Then, since this was working, I didn't need to touch it. For a year. And the way Jack's Long Term Storage works is, something I did a year ago is _at best_ vaguely remembered. So I woke up one day with only the barest clue as to how my backups were working. Opus 4.5 is a smartie, though. In a back-and-forth session of maybe 20 minutes, we together tracked down what I was doing, the act of talking about it bringing back my memories. And we documented it, as in I asked the AI to create a `backup` folder in `the-workflow` repo, and inside that a `docs` folder, and inside that some nice documentation. AIs are really really good at this kind of documentation. ### Continuous Updates Once you have an AI-connected repo for your workflow---and it doesn't need to be git, it could be Notion for example---the act of having the AI update the docs is extremely low effort. Just this morning, I was doing some Obsidian-related work on the NAS and realized, hmmm, how is the NAS's configuration getting backed up? Had a chat with Opus, figured out what was and wasn't covered, then remembered that Synology has a backup-your-config feature, checked it and yes, I had it enabled---then documented all of this. Took maybe 10 minutes. ## IDE or No IDE? Which IDE? As I [wrote about recently](leveling-up-claude-code-with-boris-chernys-help), I've chosen to roughly follow Boris Cherny's setup for Claude Code. Cherny suggests, by omission at least, that no IDE is needed---just do everything in Claude Code. That was a scary prospect to me, since heretofore I've been operating Claude Code _inside an IDE_ and some of my workflows (such as writing!) really do involve editing. That said, I was serious enough about following Cherny that I really wanted to seriously consider how I might be able to operate terminal-only. Who did I talk to? You guessed it, my pair, Opus 4.5. We had a long discussion about what terminal-only life would look like. This led me to the decision that I still wanted an IDE, but in a scaled back, secondary role compared to my then-current IDE-first mode. Next decision: which IDE? My desire was "something lightweight that stays out of my way," to which Claude responded with a shortlist of recommendations, including [Zed](https://zed.dev/). Having seen a recent "I switched to Zed" post on [Hacker News](https://news.ycombinator.com/), I tried it first and immediately fell in love. I'm writing this in Zed. ## There's More, But Enough Already In all, `the-workflow` repo already has 14 projects and 8 subprojects, and those numbers grow every week. A side benefit for me, the AI-focused fractional CTO, is that I now have an infinite number of juicy AI learning projects available. For example, just this past week, Anthropic announced that [MCP Apps](https://blog.modelcontextprotocol.io/posts/2026-01-26-mcp-apps/), which enable full-strength UI components inside Claude, ChatGPT, [Goose](https://github.com/block/goose), and more, are now officially an open standard. Guess what's next for my Obsidian setup? Custom UIs inside Claude, baby! The key message here: **AI has granted me the power to radically and continuously improve my personal workflow.** Or to paraphrase [Andrew Ng](andrew-ng-2022): > You shouldn't still be doing things like it's 2022. --- ## Claude Cowork Now Supports Plugins **URL:** https://cto4.ai/p/claude-cowork-supports-plugins **Date:** 2026-01-30 **Type:** brief **Description:** Anthropic brings Claude Code's plugin system to Cowork, along with a collection of knowledge work plugins and a slick customization system. **Tags:** ai, anthropic, claude, plugins, cowork, claude-code **Categories:** AI, Technology Anthropic [announced today](https://claude.com/blog/cowork-plugins) that Claude Cowork now supports the powerful plugin extension-bundle format that has been available for some time in [Claude Code](https://claude.com/product/claude-code). This is a significant step forward in making Cowork a true power tool for those with zero interest in learning to use Claude Code, individuals some might call **normies**. Anthropic also released a collection of [knowledge work plugins](https://claude.com/plugins-for/cowork) for Cowork that show off just what's possible: As I [wrote recently](leveling-up-claude-code-with-boris-chernys-help), Claude Code plugins can include skills as well as: > /commands, agents, hooks, [MCP servers](https://modelcontextprotocol.io/), and [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/) servers Anthropic's initial batch of Cowork plugins consist primarily of skills, commands, and connector (MCP) definitions. Anthropic's work here is impressive; the skills and commands are superbly crafted. Less prominently mentioned, but very impressive to me, is Anthropic's work to enable plugin customization. For example, these plugins often include placeholders for aspects of their configuration that will vary from one organization to the next: `~~chat` for chat platforms, `~~email` for email systems, `~~cloud storage` for cloud storage platforms, etc. These placeholders pair with a slick plugin customizer that walks you through custom-fitting the plugin to your organization. These releases should help answer the question, "So what's Cowork good for?" --- ## Leveling Up Claude Code with Boris Cherny's Help **URL:** https://cto4.ai/p/leveling-up-claude-code-with-boris-chernys-help **Date:** 2026-01-27 **Type:** essay **Description:** Cherny's thread on how he personally uses Claude Code gave me a roadmap. **Tags:** claude-code, ai, developer-tools, anthropic, skills, skillport, ai-coding **Categories:** AI, Tools I think about the kit of electronic tools that support my work as "The Workflow" and man, did I have a lot of ideas for new tools, improvements, and fixes. This 2XL-sized backlog lived in my head until June '25, when I started a "Workflow Tool ToDos" list in [Obsidian](https://obsidian.md/). I made progress here and there, with my Obsidian implementation as one example. But movement was more sloth than cheetah. [Claude Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5) arrived in November, powering up Claude across all its surfaces---Claude.ai, Claude Desktop, and Claude Code. So smart, such encyclopedic knowledge. I used Opus, of course, when I ran into setup issues on tools like [Cursor](https://cursor.com/) and Obsidian. But then it occurred to me: what if I made The Workflow itself a project, with a number of specific subprojects, and got very intentional about applying AI to level up? This turned into a rich vein of gold that I've been mining ever since, and actually led to my largest AI side projects to date, including [Skillport](skillport). AI granted me the power to radically and continuously improve The Workflow, something I never would have been able to tackle on my own---without my AI pair, there's just too much hard-to-find knowledge needed, too much work involved for each improvement. But let me focus here on Claude Code. ## Sucking Wind on Claude Code ... Where to Start? One of my deepest Workflow desires was to tune up my Claude Code setup, to use it the way I knew it could be used. The problem was, pretty much every aspect of my setup was subpar. Where the heck do I start? [Boris Cherny](https://github.com/bcherny), the creator of Claude Code, came to my rescue. He posted a great 15-tweet thread about how he personally uses Claude Code. Unlike the rest of us, though, Cherny is using Claude Code to actually write Claude Code. You can see the thread [here on Twitter](https://x.com/bcherny/status/2007179832300581177), or [here on Threadreader](https://threadreaderapp.com/thread/2007179832300581177.html). ## The Thread, Tweet by Tweet --- ## What I've Actually Changed So Far I figured, what the hell, dive right in, starting with "ditching the IDE." ### I Ditched Cursor (VSCode) but Kept a Lightweight IDE (Zed) Back when I used VSCode itself, the experience was janky and configuration a nightmare. I've been using Cursor since it arrived, and as a VSCode fork, it inherits those same drawbacks. Furthermore, once Claude Code launched, my use of AI in Cursor dropped off dramatically. I shifted to using the Claude Code VSCode plugin in Cursor, which enabled easier multiline editing than CC-in-terminal. Over time, though, I hit more and more cases where the VSCode extension couldn't do things that CC-in-terminal could. Friction. As I got up to speed on [Claude Skills](cursor-rules-to-claude-skills) and phased out Cursor Rules, I realized I was using Cursor as nothing more than an IDE, and in fact, the AI stuff in Cursor was just getting in the way. So I'm effectively paying $20/mo for the privilege of using VSCode with added AI annoyances (though I hear AI Annoyance has arrived in VSCode, too, in recent releases). I chose not to ditch the IDE altogether, like Cherny apparently did. I still look at code regularly, and as I am now, I write in my IDE. No AI slop posts on cto4.ai. Instead, I went light-and-clean with [Zed](https://zed.dev/), which has been such a breath of fresh air after living in Electron apps for several years. So much is built-in and fast---I've installed a total of three extensions in Zed; I just checked and I had 48 installed in Cursor (!!). With just three extensions, Zed gives me a comparable feature set in a lighter, snappier, more coherent package. Zed gets a smaller slice of my widescreen; the CLI gets significantly more: Note that I use the [AeroSpace](https://github.com/nikitabobko/AeroSpace) tiling window manager, so rather than 5 splits in a Ghostty window, I have 35 wide screen workspaces to work with, each laid out like the above. ### I Supercharged my CLI: CLI-Mesh Before the Cherny post, my terminal setup had barely progressed beyond basic macOS terminal, mainly experiments with [Kitty](https://sw.kovidgoyal.net/kitty/) and [Ghostty](https://ghostty.org/). But I spent almost all my time in the IDE, using Claude mostly through its VSCode extension. I split my time between two Macs, a deskbound M4 Mac Mini, and a mobile M1 Max MacBook Pro. Because I mostly wasn't operating in CLI mode, I hadn't set things up for easy SSH between machines. Cherny's setup appears to be Ghostty with splits; but many CLI advocates recommended using a good terminal app augmented by [`tmux`](https://github.com/tmux/tmux), which adds terminal session-group-persistence---reattach and all your terminal windows are just as you left them. And `tmux` works nicely across machines, which was my use case. I'd never used `tmux` and it took a while to get my head around it. I also learned through painful experience that you must avoid the dreaded Cross Machine Infinite Terminal Mirror where machine A `tmux`-attaches to machine B, which has a window attached back to machine A, and bam---lockup. In one case, [Wezterm](https://wezterm.org/) consumed 250GB of application memory on my Mac Mini, which went unstable and needed a reboot. In the end, I landed on [iTerm2](https://iterm2.com/) which has significant support for `tmux`, configured to avoid infinite mirror. With Claude Code's help, I also did some ssh and .zshrc tuning to make cross-machine access super low friction. The result is what I term "CLI-Mesh" between the two Macs, where I can access persistent CLI sessions on the Mini from the Macbook and vice versa. "Anything from anywhere" thanks to CLI-Mesh is pretty sweet. ### Sorting Permissions in Claude Code I was convinced that Claude Code had a [known, unresolved bug](https://github.com/anthropics/claude-code/issues/5140) where using global + project permissions together would fail, with globals getting replaced by project permissions rather than the two being merged. Based on this, Claude and I implemented a permission scheme based solely on a [PreToolUse](https://docs.claude.com/en/docs/claude-code/hooks) hook rather than global permissions settings. This worked, but I kept wondering, would the Claude Code team allow a serious bug like this to persist for months? I kept asking CC to recheck the open ticket, and kept getting the answer, "still unresolved." Today I dug deeper and noticed [someone had apparently discovered](https://github.com/anthropics/claude-code/issues/5140#issuecomment-3725786707) the supposed bug was actually caused by incorrect paths when specifying permissions for file operations---using / instead of // for absolute file paths. Once the paths were fixed, global permissions on file operations worked fine. It has been rumored that, at some point, Claude Code may have been incorrectly writing file paths when users approved file permissions, but I wasn't able to confirm that in researching or testing today. Anyway! Today I went back and moved everything that could be moved from the PreToolHook to Claude's Global Permissions settings. There were a few things that needed to stay in the hook: piped/compound commands, as well as restrictive git and gh subcommand parsing. The hook can handle these more complex rules because it receives the full command and runs code to parse it, splitting on pipes, extracting subcommands, checking against allowlists. Native permissions only do simple pattern matching on the raw command string. The good news is that I didn't have to actually do the work, Claude Code did---including writing Python hook scripts to parse commands. The better news is that Claude Code operation is so much more streamlined, without needing to [`--dangerously-skip-permissions`](https://www.anthropic.com/engineering/claude-code-best-practices). (I will try living dangerously, safely, once I get more into Claude Code web and long running tasks.) ### Using Skills and Other Plugins in Claude Code When Cherny's post landed, I was in the process of an extreme deep-dive into Skills and all the other varieties of Plugins: /commands, agents, hooks, [MCP servers](https://modelcontextprotocol.io/), and [Language Server Protocol (LSP)](https://microsoft.github.io/language-server-protocol/) servers. This was due to my Christmas break project, [Skillport](skillport), which brings repository-based Skill distribution to all Claude surfaces including Claude Desktop and Claude.ai, and allows organizations to host their own private Skill repos. Because I wanted Skillport's git-based repositories to be 100% compatible with Anthropic's [Plugin Marketplace](https://code.claude.com/docs/en/discover-plugins) format, I needed to know exactly how that format worked and what it supported. Although I was already pretty far along with using Skills in Claude Code, Cherny's post lit a fire under me to start using all the other plugin varieties. I've got [one sophisticated use of hooks](#sorting-permissions-in-claude-code) in place now, and am experiencing agents thanks to the PR Review plugin. I'm looking for an opportunity for custom agents next. --- Much remains to be done, but thanks, Boris Cherny, for the inspiration. --- ## Skillport: Claude Skill Sharing at Scale **URL:** https://cto4.ai/p/skillport **Date:** 2026-01-07 **Type:** essay **Description:** Or: How I spent my Christmas vacation. **Tags:** claude, skills, mcp, anthropic, open-source, skillport **Categories:** ai-tools To explain why I created the open source project Skillport, I need to provide some background. First, know that, starting in 2024 and throughout 2025, I've been working with clients to apply AI in critical business workflows, cobbling together Claude, lots of MCPs, and lots of large, complicated prompts. The results were often fantastic---what used to take days done in a few hours---but the cobbling together part was painful, as were the realities of deploying and running first-generation local MCPs. Second, Skillport was made possible by Anthropic's great work during 2025, and we'll need to explore that a bit to set the stage, focusing on work related to what I call Skills++, the combination of: - **Skills** are fantastic, but to be powerful they need ... - **Tools**, often exposed over the [Model Context Protocol](https://modelcontextprotocol.io/), MCP, providing critical business context and data; and ... - **Sandboxed code execution**, essential for advanced Skills and efficient tool use. ## Anthropic's Fantastic Year in Skills++ I asked [Claude Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5) to give me Anthropic's 2025 timeline of important announcements and releases that pertained to Skills++: Here's the condensed version of that list: | Date        | Announcement | Why Important to Skills++ | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | May 22 | Code Execution Tool: Python sandbox | **Code execution arrives.** Claude (even on web and mobile) can now run Python in a sandboxed environment. | | Jun 26 | Desktop Extensions (DXT) — One-click local MCP installation | **Packaging format for local MCPs.** .dxt bundles MCP servers with dependencies into single installable files. Node.js ships with Claude Desktop, eliminating dependency hell. Open-sourced spec enables ecosystem-wide adoption. | | Jul 14 | Connectors Directory | MCP becomes consumer-visible with one-click tool connections in Claude.ai and Desktop | | Aug 25 | Code Execution update: Bash + file operations | **Foundation for Skills runtime.** Bash support means Skills can run shell scripts, install packages, and manipulate files — the primitives that make document-creation Skills possible. | | Sep 9 | Create and edit files — Skills-powered file creation | **Skills' first killer app.** This feature is entirely powered by Skills internally (docx, xlsx, pptx, pdf). Proves the Skills architecture works for end-users, not just developers. | | Sep 29 | [Claude Agent SDK](https://www.anthropic.com/engineering/building-agents-with-the-claude-agent-sdk) | **Skills infrastructure goes public.** The Agent SDK is the same runtime that powers Claude Code. Now any developer can build agents with the same primitives Anthropic uses internally. | | Oct 16 | **[Skills launch](https://www.anthropic.com/news/skills)** — Public Skills + /v1/skills API | **The main event.** Skills become a first-class API primitive. Developers can create, version, and deploy Skills. Enterprise customers can manage Skills organization-wide. | | Oct 20 | Claude Code on the web — Browser-based coding agent | **Claude Code escapes the terminal.** Developers can delegate coding tasks from browser or mobile. Runs in Anthropic-managed sandboxes with GitHub integration. Parallel task execution means overseeing a "fleet" of coding agents. | | Oct 21 | Create and edit files GA — Network/egress controls | **Enterprise-ready Skills execution.** Admins can control what Skills can access (package managers only, allowlisted domains, or full internet). Critical for regulated industries. | | Oct 27 | [Financial Services Agent Skills](https://www.anthropic.com/news/advancing-claude-for-financial-services) | **Domain Skills arrive.** Six specialized Skills for finance: DCF models, comp analysis, due diligence packs, company teasers, earnings analyses, and coverage reports. Proves Skills can encode professional domain expertise, not just document formats. | | Nov 3 | Claude Code Plugin Marketplaces | Decentralized Skill distribution arrives. Anyone can create and host a plugin marketplace — no Anthropic approval needed. This is the distribution model for Skills in Claude Code: curated collections that teams and communities can share via /plugin marketplace add. | | Nov 4 | **Code execution with MCP** — Engineering blog | **The conceptual foundation.** Explains why presenting MCP tools as code APIs (instead of direct tool calls) dramatically reduces token overhead. Explicitly connects to Skills — agents can persist reusable code with SKILL.md files. This paper laid the groundwork for Advanced Tool Use two weeks later. | | Nov 20 | **Advanced Tool Use** — Tool Search Tool, Programmatic Tool Calling | **Patterns that apply to Skills and MCP.** Skills frequently pair with MCP servers, and these patterns — deferred loading, on-demand discovery, programmatic orchestration — apply directly to that layer. We've already adapted and applied these concepts within Skillport. | | Nov 20 | MCPB (MCP Bundles) — DXT renamed, moved to MCP project | **Cross-client portability.** DXT becomes MCPB under the official MCP project. Bundles now work in any MCP-compatible app, not just Claude. Same week as Advanced Tool Use — the distribution format matures alongside the usage patterns. | | Nov 24 | Opus 4.5 — Smartest and most token-efficient tool-using model to date | **Smarter, more efficient tool use.** And a superb model generally. | | Nov 24 | Claude for Excel — Beta expands to all Max, Team, Enterprise | **Excel Skill matures.** Adds pivot tables, charts, file uploads. Validates the "Skill as Office add-in" pattern at broader scale. | | Nov 25 | [MCP Nov spec](https://blog.modelcontextprotocol.io/posts/2025-11-25-first-mcp-anniversary/) — Async ops, registry, extensions | **MCP matures as infrastructure.** One-year anniversary release adds async operations for long-running tasks, official registry with ~2,000 servers, and extensions framework. | | Dec 3 | Bun acquisition | **Anthropic is serious about tooling.** Acquiring a JavaScript runtime signals investment in the execution layer that Skills depend on. | | Dec 9 | [MCP → Linux Foundation](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation) | **Tool protocol becomes neutral.** MCP donated to Agentic AI Foundation (AAIF) with OpenAI, Google, Microsoft, Block as co-founders. The protocol for connecting AI to external tools is now industry infrastructure, not Anthropic-owned. | | Dec 18 | **Agent Skills open standard** — Portable Skills, enterprise admin | **Skills become portable.** Skills built for Claude work in ChatGPT, Cursor, or any adopting platform. This is the "write once, run anywhere" moment for agent capabilities. Partner directory (Atlassian, Notion, Figma) shows ecosystem momentum. | ## Skills: Domain Expertise Codified Ironically, Anthropic demonstrated how great Skills could be---more than a month before Skills were officially announced. On September 9th, Anthropic gave us Claude's incredible "create and edit files" feature, which enables direct creation and editing of MS Word, Excel, Powerpoint, and PDF files right in Claude conversations. Little did we know that these capabilities were actually powered by Skills. On October 16th, Anthropic launched Skills, and in [this GitHub repo](https://github.com/anthropics/skills) (committed the day before the launch) you can see the actual Skills that power "create and edit files." These Skills are powerful, leveraging code execution with Python scripts and even XML schemas. At this point, it became clear that we were moving beyond the age of "cobbling together" Claude-based workflow solutions. Now we had an elegant abstraction for capturing the domain knowledge behind any business workflow. And Skills worked across all Claude surfaces: Claude.ai on the web, Claude Desktop on Mac and Windows, and Claude Code. ## Sharing Skills: Claude Code's Plugin Marketplaces On November 3rd, Anthropic announced [Plugin Marketplaces](https://docs.anthropic.com/en/docs/claude-code/plugins) for Claude Code. Plugins are Anthropic's name for a category of artifacts that Claude Code users might want to create, reuse, and share, including slash commands, hooks, agents, and---you guessed it---Skills. Plugin Marketplaces are public git repositories hosted on GitHub, Gitlab, etc. that contain these artifacts, and that any Claude Code user can attach to and download from. Installing a Skill or other artifact takes just a command or two in Claude Code. The Plugin Marketplace repo design is another elegant piece of work by Anthropic and, among other things, supports Skill versioning. Plugin Marketplaces were very exciting, but also a groaner, because they target only Claude Code CLI (read: developers running Claude Code in a terminal interface), and only work with public repositories, a non-starter for proprietary business workflows. ## Packaged, Context-friendly, Performant MCPs Anthropic's Model Context Protocol has seen tremendous uptake and support, but it has faced significant challenges to adoption as well. In Claude.ai and Claude Desktop, MCPs have been difficult to install at best, and not available at all in some cases (local MCPs with Claude.ai). Every installed MCP, even if it's not being used, eats up precious model context tokens---the more tools the MCP exposes, the higher the context overhead. Finally, the standard model-to-MCP-tool back-and-forth tends to result in laggy performance. Anthropic has been steadily chipping away at these MCP downsides. On the MCP distribution front, Anthropic announced a standard for one-click-installable [Desktop Extensions](https://www.anthropic.com/engineering/desktop-extensions) (local MCPs) in June; followed on with [Connectors directory](https://www.anthropic.com/news/connectors-directory), supporting one-click installs of cross-platform Remote MCPs, in July; and in November launched [MCP Bundles (MCPB)](https://blog.modelcontextprotocol.io/posts/2025-11-20-adopting-mcpb/), the successor to Desktop Extensions. By end of year, the official Connectors directory was solidly populated with MCPs for many major platforms: On the context efficiency and performance fronts, November was a big month: possibly spurred by [Cloudflare's "Code Mode" research](https://www.anthropic.com/engineering/code-execution-with-mcp), Anthropic began recommending and supporting a radically context-efficient and performant mode of model-to-MCP interaction, [Programmatic Tool Calling](https://www.anthropic.com/engineering/advanced-tool-use). In a nutshell, instead of the model directly interacting with MCP tools, sifting and sorting their outputs, gobbling up huge chunks of context, the model instead **writes code** that does all that. As a result, the model no longer bears the context weight of all the intermediate tool call results, and no longer incurs the performance penalties of long, chatty conversations with tools. Anything that improves MCPs and tool calling is a big win for Skills++. ## Ubiquitous, Sandboxed Code Execution in the Cloud If you're going to have powerful skills like "create and edit files" that use scripts, or get the benefit of Programmatic Tool Calling, you need a place to run code---one that works on all Claude surfaces, including Claude.ai on the web. Anthropic anticipated this need and began delivering on it in May, with Code Execution (of Python scripts) arriving as part of the [Claude 4](https://www.anthropic.com/news/claude-4) platform. In August, Claude added the ability to run Bash scripts and perform file system operations. All of the above were prerequisites for Skills generally and the "create and edit files" Skills specifically, which launched just a few weeks later. October brought us [Claude Code on the Web](https://www.anthropic.com/news/claude-code-on-the-web), a long-running-agent-loop mode for Claude Code that was completely reliant on sandboxed code execution in the cloud, as well as sandbox networking with egress controls. Finally, in December, Anthropic acquired [Bun](https://bun.sh/), known for its blazingly fast JavaScript/TypeScript runtime that dramatically outperforms Node.js on startup time and execution speed. The acquisition signaled Anthropic's commitment to owning the code execution layer that Skills++ depends on. Fast, lightweight runtimes are essential when you're spinning up thousands of ephemeral sandboxes for Programmatic Tool Calling and Skill execution across all Claude surfaces. ## The Christmas Itch That Needed Scratching All of these great announcements and releases, so much promise ... but heading into the end of year lull, I still didn't quite have what I needed to really do business workflow automations right and to help clients deploy Skills at scale. The main gap, to me, was Skill distribution and sharing. I wanted the elegance of Plugin Marketplaces, e.g. git repo based with Skill versioning, but without the public repo constraint and supporting all Claude surfaces. Would it be feasible to create a project that extends the Plugin Marketplace concept to support private repos, authentication, and get it working across all Claude surfaces? For me, there's always another question as well: would I learn something in the process? I decided to go for it: start date Dec. 17th, initial release Jan 4th. So far, the answer to both questions seems like YES. Here's what I built. ## Skillport: Claude Skill Sharing at Scale Skillport helps organizations deploy, share, and manage Skills at scale and without limits: - Works across all Claude surfaces: - Claude.ai - Claude Desktop - Claude Mobile[^1] - Claude Code - Claude Code on the Web[^2] [^1]: Skills that run on Claude Mobile must be installed using Claude.ai or Claude Desktop [^2]: Skillport authentication for Claude Code on the web happens in Claude Code CLI - Supports "normal Skill user" operations: find, install, update Skills. - Supports "Skill creator / editor" operations: create, edit, version-bump, and delete Skills. - Skill repositories use Anthropic's Plugin Marketplace format, ensuring compatibility - Private repositories are supported, enabling secure sharing within an organization (including Claude Code) - Skillport's Claude-compatible Connector carries the lowest possible Claude context overhead: one tool only - Claude's user interactions with Skillport are guided by Skillport's own skill - Skills are versioned and users can check to see if upgrades are available to any installed Skill ### Open Source, MIT License Skillport was released publicly on GitHub on January 4th 2025 under the MIT license, and consists of two repos: - [`skillport-connector`](https://github.com/cto4ai/skillport-connector), which enables an organization using Skillport to create their own Claude Connector (remote MCP) that authenticates over OAuth and provides access to a complete API for installing and managing Skills. The initial version supports Google OAuth and deploys to Cloudflare workers, but fork the repo and have at it, extending to other OAuth IdPs and deployment targets should be low-effort - [`skillport-marketplace`](https://github.com/cto4ai/skillport-marketplace), a GitHub template that creates an Anthropic Plugin Marketplace compliant repository where organizations can (privately!) manage their library of Skills ### The Skillport Skill Skillport uses Skills itself, and in a rather novel way. The `skillport-connector` exposes just a single tool to Claude -- authentication -- and guides Claude to download and install the `skillport` Skill. With that Skill installed, Claude has full knowledge of how to interact with Skillport and use the full suite of Skill management tools. Claude-to-Skillport interactions take place via Programmatic Tool Calling rather than high-context-overhead MCP tools. --- I learned a lot, I suspect Skillport will be useful for my fractional CTO clients, and I hope others find it helpful too. --- ## OpenAI Quietly Adopts Skills **URL:** https://cto4.ai/p/openai-adopts-skills-via-willison **Date:** 2025-12-14 **Type:** elsewhere **Description:** Which makes me happy. Via Simon Willison. **Tags:** ai, claude, skills, openai, simon-willison **Categories:** AI, Technology I'm a huge fan of Anthropic's Agent Skills---AKA Claude Skills AKA Skills---as you can see from my recent posts [here](cursor-rules-to-claude-skills) and [here](anthropic-stop-building-agents-start-building-skills). [Simon Willison](https://simonwillison.net/) notes that OpenAI has, very quietly, added support for skills in both ChatGPT and [Codex CLI](https://github.com/openai/codex). Hurray! The concept of Skills is so simple and LLM-friendly that really any reasonably smart model should easily be able to use them, after spending a few minutes looking at their structure and reading docs. I was just asking [Antigravity](kevin-hou-antigravity) / [Gemini 3 Pro](https://blog.google/products/gemini/gemini-3/) yesterday if it could run Skills, and after looking through my `.claude/skills` directory, its response was "no problem" ... But having first-class support out of the box is significantly better. Maybe OpenAI will even admit and document the capability sometime soon. --- ## Antigravity Gets an A+ on My Browser QA Task **URL:** https://cto4.ai/p/antigravity-browser-for-qa **Date:** 2025-12-14 **Type:** brief **Description:** Google DeepMind's Antigravity nails a simple autonomous browser QA test of this blog where Claude Code stumbled **Tags:** antigravity, qa-testing, browser-automation, agents, google-deepmind **Categories:** AI, Tools I just did an upgrade of my [Astro](https://astro.build/)-based blog to the latest-Astro-that-makes-sense-right-now which turns out to be 5.16.5. Having just finished a [deep dive](/p/kevin-hou-antigravity) into [Antigravity](https://developers.googleblog.com/en/build-with-google-antigravity-our-new-agentic-development-platform/) and ingested some Kool-Aid about agent managers and agent-ready integrated browsers, I decided to see if I could automate QA testing of my blog, and do a thorough rather than minimal QA on the 5.16.5 version before pushing it live.[^1] I also wanted to have an "agent manager" orchestrate this, with the actual QA delegated to an agent (sub-agent). [^1]: I realize what I'm calling "thorough QA" here is weak sauce compared to real automated QA. I tried using [Claude Code](https://www.anthropic.com/claude-code) first, using Claude sub-agents and the `browser-tools \ browser-automation` skill. Across all three Claude Code surfaces (terminal, VS Code extension, Claude Code Desktop), this failed. No doubt there was something that could have been configured correctly somewhere, but still, with at least a modest effort on my part, I couldn't get it working. So I decided to aim Antigravity at the project and see what it could do. Far, far better, as it turns out. My request was: ``` I want you to use the browser tool to QA this website. A preview is running locally at http://localhost:4322/ Test at least 30 posts, old and new, and all functionality, intra-page and site wide (eg search) ``` Agent Manager dug right in, starting with a plan, cranking up Antigravity's integrated Chrome agent-controlled browser, working its way methodically through the local preview site as requested, and finishing with a nice QA report (see below). I was a little perplexed that Agent Manager appeared to be running the QA process "on the main thread" so to speak---that is, the agent window I was in was "busy" running the task. But I think this is a PEBKAC problem, also known as my own lack of understanding. Agent Manager allows multiple active conversations, meaning I could start up other agent tasks while this was running. Agent Manager also has an Inbox, so I would get notifications from any of those agent windows. I'm sure this will become clearer as I work more with Antigravity. In any case: A+ on this agent-driven QA process! --- ## Kevin Hou on Google DeepMind's Antigravity **URL:** https://cto4.ai/p/kevin-hou-antigravity **Date:** 2025-12-12 **Type:** episode **Description:** My thoughts on Kevin Hou's talk about Antigravity, DeepMind's new agent-first development platform. **Tags:** google-deepmind, antigravity, gemini, ai-coding, agentic-ai, computer-use **Categories:** AI Coding Tools Google DeepMind has definitely gotten my attention with significant releases over the past few weeks, in particular [Gemini 3 Pro](https://blog.google/products/gemini/gemini-3/), [Gemini 3 Pro Image](https://deepmind.google/models/gemini-image/pro/) aka [Nano Banana Pro](https://chromeunboxed.com/the-hilarious-230-am-origin-story-of-how-nano-banana-got-its-name/), and most recently [Antigravity](https://developers.googleblog.com/build-with-google-antigravity-our-new-agentic-development-platform/). Based on [this talk](https://www.youtube.com/watch?v=xmbSQz-PNMM&t=29388s) by [Kevin Hou](https://www.linkedin.com/in/kevinhou22/) at the recent [AI Engineer Code Summit](https://www.ai.engineer/code), as well as the [ThursdAI](https://thursdai.news/) podcast's chat with Hou recorded at the same event, I've made it a priority to get hands-on with Antigravity, and find it to be a new and different take on the coding-with-AI experience. If, with [Claude Code](https://www.anthropic.com/claude-code), Anthropic's message was: > We don't know what the experience should be, so we're making Claude Code an extremely thin wrapper around our great coding models. Google DeepMind is telling us: > We think we know what the next step function should look like, here's Antigravity. Below find my commentary and highlights from Kevin Hou's talk. ## What Is This Thing? Antigravity is definitely **not** a thin wrapper around a great coding model. It is, rather, on the other end of the spectrum: a development platform whose feature set includes elements that go beyond existing alternatives, even "thick" ones. It is "Unapologetically Agent-First" with an uber-overlord Agent Manager: Hou introduces Antigravity as a platform with three main components: Two of these components are, effectively, "new things" in an AI Coding tool. Other platforms can "do browser stuff" and might have some agent/subagent action happening, but no major player delivers browser and agent manager elements that approach the level of power and integration that I'm seeing in Antigravity. ## Diving Deeper on the Three Surfaces This more detailed view begins to expose a bit more about the three components (click to zoom the image): It's an AI developer platform with three surfaces. The first one is an editor. The second one is a browser. And the third one is the agent manager. So we'll dive into what this means, and what each looks like. So a paradigm shift here is that agents are now living outside of your IDE. And they can interact across many different surfaces that your agent or that you as a software developer might spend time in. ### The Agent Manager Hou tells us: > Let's start with the agent manager. So that's the thing up top. This is your central hub. It's an agent-first view, and it pulls you one level higher than just looking at your code. So instead of looking at diffs, you'll be kind of a little bit further back. And at any given time, there is one agent manager window. To me, what resonates is "one level higher." ### The Editor Hou's enthusiasm for the Editor is not that it's a sexy feature, but that it is a practical, utilitarian tool for an important job, and has been optimized for that job, e.g. with 100ms load time: > Now you have an AI editor. This is probably what you've grown to love and expect. It has all the bells and whistles that you would expect. Lightning fast, auto-complete. This is the part where you can make your memes about, yes, we forked VS Code. And it has an agent sidebar. And this is the sort of thing. It's mirrored with the agent manager. And this is when you need to dive into your editor to accomplish maybe your 80% to 100% of your task. And at any point, we made it very, very easy because we recognized not everything can be done purely with an agent. For you to command E or control E and hop instantly from the editor into the agent manager and vice versa. And this takes under 100 milliseconds, it's zippy. A key note: the editor has an agent sidebar, which **is not** the Antigravity Agent Manager---it's more like the [Cursor](https://cursor.com/) agent sidebar, just a way to work closely with a smart coding model while staring at your source files. The Agent Manager is the agent overlord, one level higher. Also notable: the Antigravity Editor is a VS Code fork, but where Cursor _looks like_ a VS Code fork, Editor does not: the UI feels (more) different and the feature set seems more obviously changed. ### The Browser > And then finally, something that I love, an agent-controlled browser. This is really, really cool. And hopefully for the folks in the room that have tried Antigravity, you've noticed some of the magic that we've put in behind here. So we have an agent-controlled Chrome browser. And this gives the agent access to the richness of the web. And I mean that in two ways. The first one, context retrieval, right? It has the same authentication that you would in your normal Chrome. You can give it access to your Google Docs. You can give it access to, you know, your GitHub dashboards and things like that and interact with a browser like you would as an engineer. But also, what you're seeing on the screen is that it lets you, it lets the agent take control of your browser, click and scroll and run JavaScript and do all the things that you would do to test your apps. > > So here, I put together this like random artwork generator. All you do is refresh and you get a new picture of like a piece of Thomas Cole artwork. And now we added in a new feature, which is this little modal card. And the agent actually went out and said, okay, I made all the code. But instead of showing you a diff of what I did, let's instead show you a recording of Chrome. So this is a recording of Chrome where the blue circle is the mouse, it's moving around the screen, and in this way you get verifiable results. So that's what we're very excited about, our Chrome browser. Antigravity still has some sharp edges, and in my experience the sharpest of those are in the Antigravity browser integration: it's buggy at times, not as speedy as it could be, and not as smart as it should be. But it's breaking new ground and I'm hopeful Google will actually prioritize Antigravity as a product and be serious about improving it. We'll see. ### Back to the Agent Manager Hou circles back to the Agent Manager, the component that ties it all together: > The Agent Manager can serve as your control panel. The editor and the browser are tools for your agent. And we want you to spend time in the Agent Manager. And as models get better and better, I bet you you're going to be spending more and more time inside of this Agent Manager. > > And it has an inbox .... It lets you manage many agents at once. So you can have things that require your attention. For example, running terminal commands .... There are probably some commands that you want to make sure you hit okay on. So things like this will get surfaced inside of this inbox. One click, you can manage many different things happening at once. And it has a wonderful OS level notification. This solves that problem of multi-threading across many tasks at once. That the Agent Manager has an **_inbox_** highlights the message that this isn't just an **agent**, it's an **orchestrator**. ## Why Did We Build It? Hou dives into this question next: > Why did we build the product? How did we arrive at this sort of conclusion? You might say, oh, adding in a new window, it's pretty random, right? One part of his answer is that, after showing huge progress in Gemini 2.5 Pro (I recall watching Google's announcement live at the [AI Engineer World's Fair](https://www.ai.engineer/worldsfair/2024) in San Francisco June 2024), Google now has a genuine contender for top coding model in Gemini 3 Pro. And if they didn't, Antigravity might not exist. If Gemini 3 Pro was a noncompetitive coding model, Google wouldn't be showcasing weakness by launching Antigravity: > The product is only ever as good as the models that power it .... Every year there is this sort of new step function. The first, there was a year when it was autocomplete, right? Co-pilot. And this sort of thing was only enabled because models suddenly got good at doing the short form autocomplete. And then we had chat. We had chat with [RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback). Then we had agents. So you can see how every single one of these product paradigms is sort of motivated by some change that happens with model capabilities. Hou also emphasizes the benefits of being right there in DeepMind next to the Gemini folks: > And it's a blessing that our team ... embedded inside of DeepMind. We had access to Gemini for a couple of months earlier, and we were able to work with the research team to basically figure out, you know, what are the strengths that we want to show off in our product? What are the things that we can exploit .... Where are the gaps in the model, and how can we fix that, right? And so this is a very powerful part of why Antigravity came to be. ### Four Categories of Improvement Using some Nano Banana Pro output, Hou highlights four categories of improvement in Gemini 3 Pro: > The first one is intelligence and reasoning. You all are probably familiar with this. You used, or used Gemini 3, and you probably thought it was a smarter model. This is good. It's better at instruction following. > > It's better at using tools. There's more nuance in the tool use. You can afford things like, you know, there's a browser now. There's a million things that you could do in a browser. It can literally even execute JavaScript. How do you get an agent to understand the nuance of all these tools? > > It can do longer running tasks. These things now take a bit longer, right? And so you can afford to run these things in the background. It thinks for longer. Just time has gotten stretched out. > > And then multimodal. I really love this property of what Google has been up to. The multimodal functionality of Gemini 3 is off the charts, and you start combining it with all these other models like Nano Banana Pro, and you really get something magical. ## Tiptoeing Towards AGI Tongue-in-cheek, Hou describes how Antigravity is stepping towards AGI and identifies three steps: ### Step One: Raise the Ceiling ... **_of what the agent can do_**. > So step one is we want to raise the ceiling of capability. We want to aim higher, have higher ambition. And so a lot of the teams at DeepMind were working on all sorts of cutting-edge research. Hou calls out Google's extensive research into computer use, in particular browser use, as the first critical bar-raiser in Antigravity: > Browser use has been one of these huge unlocks. And this is twofold. I mentioned the sort of retrieval aspect of things. It's reasonable for the model to now give in context that can generate the code that hopefully functionally works. And then you've got the what to build. There's this richness in context, the richness and institutional knowledge. And these are the sorts of things that having access to a browser, having access to your bug dashboards, having access to your experiments, all these sorts of things that now gives the agent this additional level of context. > > This is now the other side of things. Browser as verification. So you might have seen this video. This is the agent. The blue border indicates that it's being controlled by the agent. And so this is a flight tracker. You put in, you know, a flight ID and then it'll give you sort of the start and end of that flight. And this is being done entirely by a Gemini computer use variant. So it can click, it can scroll, it can retrieve the DOM, it can do all the things. And then what's really cool is you end up with not just the diff, you end up with a screen recording of what it did. So it's changed the game and the model can take this and because it has the ability to understand images, it can take this and iterate from there. So that was the first category, browser use, just an insane, insane magical experience. Hou's second "raise the ceiling" focus is images, which these days means Gemini 3 Pro Image aka Nano Banana Pro. I hadn't connected the dots as to why Nano Banana Pro might be a big deal for development. Yes, you could easily generate some really spiffy architecture diagrams, but that's a pretty small win. But I now understand that the value is much, much higher. > Now the second place that we wanted to spend time is on image generation. Gemini is spending a lot of time on multimodal. And this is really great for consumer use cases, right? Nano Banana2 was mind boggling. But also for devs. Devs are inherently, this is a multimodal experience. You're not just looking at text, you're looking at the output of websites, you're looking at architecture diagrams, there's so much more to coding than just text. And so there's image understanding, this is verifying screenshots, verifying recordings, all these sorts of things. And then the beautiful part about Google is that you have this synergistic nature. This product takes into account not just Gemini 3 Pro, but also takes into account the image side of things. And so here I had to watch this part of the talk about five times, first to figure out how you even get to NBP (answer: from Agent Manager, just ask for something visual). But secondly, to get my head around why Hou was claiming that Nano Banana Pro was a big win for _developers_. I've spent a lot of time using NBP, and been repeatedly amazed at what it was creating for me. But I hadn't made the connection between NBP's incredible image generating power and the UI design process that is a critical element of virtually all application development work. It's the design, silly! > I want to give you a quick demo of mockups. I have a hunch and you all probably believe this too. Design is going to change, right? You're going to spend maybe some time iterating with an agent to arrive at a mockup. But for something like, oh, let's build this website, we can start in image space. And what's really cool about image space is it lets you do really cool things like this. In this case, I'll show you my example instead of Hou's. My initial Antigravity project involves ejecting a project from the [Bolt.new](https://bolt.new) platform so we could regain control of the development process, which had proven itself to be out-of-control unstable in Bolt. The webapp in question is a simple secure data sharing tool, and I worked with Antigravity through the process of unbolting Bolt and several rounds of code reviews for security and robustness. I didn't actually intend to redo the UI, but here I was, in Agent Manager, with a working app that had a very plain UI. So I tiny-prompted ``` generate a new UI look and feel for me to consider for this app ``` and, one-shot, got this: I make no endorsement, express or implied, of the model's self-described "Dark Glassmorphism" theme, but still, this is a really impressive app theme, almost viable as a first design iteration, _from a 14-word prompt_ that provided no stylistic guidance. As I discovered on re-watch #27, the Agent Manager intelligently orchestrates here, using Gemini 3 Pro to transform my 14 words into a 16-line, 153-word Nano Banana Pro prompt that includes guidance on fidelity, layout, style, and mood: ``` A high-fidelity UI design for a secure file upload web application. Modern, clean aesthetic. Dark mode with deep slate blues and vibrant accent greens (referencing the '[The Organization Name Here]' perhaps, but keeping it tech-focused). Layout: - Left Sidebar: Navigation links (My Files, Shared with Me, Settings, Admin Panel). User profile at the bottom. - Main Content Area: - Top header: Search bar, notifications, and 'Upload New File' button. - Quick Stats Cards: 'Storage Used', 'Total Encrypted Files'. - Central Area: A large, elegant drag-and-drop zone with a subtle dashed border and cloud icon. - Recent Files List: A table-like view showing file icons (PDF, IMG), filenames, size, upload date, and a 'lock' icon indicating encryption. - Style: Glassmorphism effects on the sidebar and cards. Smooth gradients. Crisp typography (Inter or Roboto). - Mood: Trustworthy, secure, professional, yet easy to use. ``` And cool gets even cooler---note that I selected a region of the image and added a comment. Hou: > We can add comments. And so you end up commenting and leaving a bunch of queued up responses. And it's kind of like GitHub. You'll just say, all right, now update the design. And then it'll put it in here. The agent is smart enough to know when and how to apply those comments. And now we're iterating with the agent in image space. So really, really cool new capability. And what was awesome is that we had Nano Banana Pro. Those comments feed right back to the Agent Manager, which is then _fully capable of building what Nano Banana designs_. The resulting application would almost certainly have a much better design, and be far more robust and maintainable, than anything a prompt-to-app tool like Bolt could produce. --- ### Step Two: Age of Artifacts ... **_long live the agent manager_**. Hou calls out Antigravity's Artifacts concept as the second critical bar-raiser. Understanding the full impact and implications of Artifacts also took me a few replays to grasp. I liked the idea of artifacts, especially their multimodal nature. But I didn't get the "long live the agent manager" tagline at first. Hou introduces Artifacts this way: > And so step two was, all right, we have this new capability. We've pushed the ceiling higher. Agents can do longer running tasks. They can do more complicated things. They can interact on other surfaces. And so this necessitates a new interaction pattern. And we're calling this Artifacts. This is a new way to work with an agent. And this is one of my favorite parts about the product. And at its core is this agent manager. So let's start by defining an Artifact. I love Hou's Artifact definition; I created my own image for it using Nano Banana Pro: > This is what you see on the right side of this agent manager. We've dedicated sort of half the screen and your sidebar to this concept of Artifacts. > We've all tried to follow along chain of thought **[left side]**. And I would say this, you know, we did some fanciness here inside of the agent manager to make sure conversations are broken up into like chunks. So in theory, you could follow along a little bit better in the conversation view, but ultimately you're looking at a lot of strings, a lot of tokens. This is very hard to follow. And then there's like 10 of these, right? So you just scroll and scroll and you're like, what the heck did this agent do? And this has been traditionally the way that people review and supervise agents ... > > But isn't it much easier to understand what is going on inside of this **[right side]** visual representation? And that is what an Artifact is. > > The whole reason why I'm not just standing up here and giving you this long stream of consciousness is because I have a PowerPoint. The PowerPoint is my Artifact. Gemini 3 is really strong with this sort of visual representation, it's really strong with multimodal. So instead of showing this **[left side]**, which of course we always show you, but we want to focus on this **[right side]**. And I think this is the game-changing part about Antigravity. And the theme is this dynamicism. As I started to digest this, the connection between the powerful Artifacts abstraction, and an Agent Manager capable of long-running orchestration of my projects, began to really sink in. Hou emphasizes that _the model chooses_ if, what, why, and for whom artifacts are created. > The model can decide if it wants to generate an artifact. And let's remember there are some tasks. We're changing a title. We're changing something small. It doesn't really need to produce an artifact for this. So it will decide if it needs an artifact. And then second, what type of artifact? And this is where it's really cool. There are many potential infinite ways that it can represent information. > > Common ones are markdown in the concept of a plan and a walkthrough ... When you start a task, it will do some research. It will put together a plan. This is very similar to like a PRD. It will even list out open questions. So you can see in this feedback section, it'll surface, hey, you should probably answer these three questions before I get going. > And what's really awesome---and we're betting on the models here---is that the model will decide whether or not it can auto-continue. If it has no questions, why should it wait? It should just go off. But more often than not, there are probably areas where you may be under-specified ... So it'll surface open questions for you. And so you'll start with that implementation plan. > > It might produce other artifacts. We've got a task list here. It might put together some architecture diagrams. And then you'll get a walkthrough at the end. And this walkthrough, you kind of saw a glimpse of this before. But it is, hey, how do I prove to you, agent to human, that I did the correct thing and I did it well. And then this is the part that you'll end with. It's kind of like a PR description. > > And then there's a whole host of other types, right? Images, screen recordings, these mermaid diagrams. And really what's quite cool is that because it's dynamic, the agent will decide this over time. So suddenly there's maybe a new type of artifact that we maybe would have missed. And then it'll figure that out. It'll just become part of the experience. It's very scalable. > > This artifact primitive is something that's very, very powerful that I'm pretty excited about. Hou circles back to Artifact comments, explaining how they are available across all types of Artifacts, all modalities: > Another very cool property of this Artifact system: we want to be able to provide feedback along this cycle. So from task start to task end, we want to be able to provide feedback and inform the agent on what to change. The Artifact system lets you iterate with the model more fluidly during this process of execution. Not to sound like a complete Google shill, but I love Google Docs, right? Google Docs is a great pattern---the comments are great. This is how you might interact with a colleague. You're collaborating on a document, then all of a sudden you want to leave a text-based comment. So we took inspiration from that. > We took inspiration from GitHub. You leave comments. You highlight text. You say, hey, maybe this part needs to get ironed out a bit more. Maybe there's a part that you missed or actually don't use Tailwind, use Vanilla CSS. So these are the sorts of comments that you would leave. You'd batch them up and then you'd go off and send. And then in image space, this is very cool. We now have this Figma style, highlight to select. And now you're leaving comments in a completely different modality, right? > > We've done this and instrumented the agent to naturally take your comments into consideration without interrupting that task execution loop. So at any point during your conversation, you could just say, oh, actually, I really don't like the way that that turned out. Let me just highlight that, tell you, send it off, and then I'll just get notified when you're done taking into consideration those comments. Hou finally then takes a step up to our uber Agent Manager, making the point that the "form factor as models advance is going to look very different": > It's a whole new way of working. This is at the center of what we're trying to build with Antigravity. It's pulling you out into this higher level view. And the Agent Manager is built to optimize the UI of artifacts. We have a beautiful Artifact review system. We're very proud of this. And it can also handle ... parallelism and orchestration. So whether this be many different projects, whether this be the same project and you just want to execute a design mock-up iteration, at the same time you're doing research on an API, at the same time you're iterating and actually building out your app, you can do all these things in parallel. And the artifacts are the way that you provide that feedback. The notifications are the way that you know that something requires your attention. It's a completely different pattern. And what's really nice is that you can take a step back. > > Of course, you can always go into the editor. I'm not going to lie to you. There are tasks that you maybe don't trust the agent yet. You don't trust the models yet. And so you can Command-E and it will open inside the editor within a split second with the exact files, the exact artifacts, and that exact conversation open ready for you to auto-complete away to continue chatting synchronously to get you from 80% to 100%. So we always want to give devs that escape hatch. > > But in the future world---we're building for the future---you'll spend a lot of time in the Agent Manager working with parallel subagents. It's a very exciting concept. --- ### Step 3: Be Your Biggest User ... **_research / product flywheel_** > Okay. So now that you've seen, we've got new capabilities, multitude of new capabilities. We've got a new form factor. Now the question is like, what is going on under the hood at DeepMind? And the secret here is a lesson that I guess we've just learned over the past. I don't know. We've spent like, I personally spent like three years in CodeGen. It's just to be your biggest user, right? > And that creates this research and product flywheel. And so I will tell you, Antigravity will be the most advanced product on the market because we are building it for ourselves. We are our own users. And so in the day to day, we were able to give Google engineers, DeepMind researchers, we were able to give them an early access and now an official access to Antigravity internally. And so now all of a sudden, the actual experience of the models that people are improving, the actual experience of using the Agent Manager and touching Artifacts is letting them see at a very, very real level what are the gaps in the model. And whether it be computer use, whether it be image generation, whether it be instruction following, right? > Every single one of these teams, and there are many teams at Google, has some hand inside of this very, very full-stack product. And so you might notice as an infrastructure engineer, you might say, oh, this is a bit slow. Well, build it for yourself, make it faster. Image gen, all of a sudden, computer use isn't going well. It can't click this button. It's really bad at scrolling, really bad at finding text on the page. Well, go off and make that better, right? So it gives you this level of insight that evals just simply can't give you. And I think that's what's really cool about being at DeepMind. You are able to integrate product and research in a way that creates this flywheel and pushes that frontier. Hou drifts here, as a relatively new Google hire, into making claims and guarantees that run counter to Google's actual behavior in the past. What Hou says Google will do, is perhaps what they _should do_, but have often _failed to do_. I hope Google can steer a committed and steady course here, as I'm very excited by Antigravity and its underlying Gemini 3 powerplant. Time will tell. --- ## Really Sam? Code Red? We Have Questions. **URL:** https://cto4.ai/p/sam-altman-code-red **Date:** 2025-12-03 **Type:** essay **Description:** Starting with, are you familiar with A Few Good Men? **Tags:** ai, openai, sam-altman, humor, satire, google, gemini, commentary **Categories:** AI, Technology On Monday, [Sam Altman](https://en.wikipedia.org/wiki/Sam_Altman) sent an internal memo declaring a "Code Red" at OpenAI. This is, apparently, the highest level of internal panic, superseding the previously declared "Code Orange." I have questions. Let's start with the obvious one: **Has Sam Altman seen [A Few Good Men](https://en.wikipedia.org/wiki/A_Few_Good_Men)?** Because in A Few Good Men, a "Code Red" is not a heroic rallying cry. It's not the CEO equivalent of "Avengers, assemble!" A Code Red is an illegal hazing order that results in a Marine being gagged, beaten, and killed. The entire movie is about how everyone who touched the Code Red (the Colonel who ordered it, the Lieutenant who passed it down, the Marines who carried it out) ends up arrested, discharged, or dead by suicide. So when the CEO of one of the most valuable startups in history announces he's ordering a Code Red, my first question is: **Did anyone in comms raise their hand?** > Sir, I love the energy, but in the movie this ends with Jack Nicholson being dragged out of a courtroom in handcuffs. > Sir, the Code Red literally kills someone. > Sir, the famous line from this movie is 'You can't handle the truth,' which is what the villain screams right before being arrested. **Second question: What color comes after red?** Apparently Altman has a whole color-coded panic system. They were previously at Code Orange. Now they're at Code Red. Is there a Code Crimson? Code Burgundy? Code Arterial Spray? Code "Everyone Update Your LinkedIn"? More importantly: Has OpenAI ever been at Code Green? Has there ever been a moment when things were fine? Or has this company been cycling through warning colors since inception like a perpetually malfunctioning stoplight? **Third question: Who is Private Santiago?** In the movie, Santiago is the Marine who gets hazed to death. He was "perceived as not one of the team" because he broke the chain of command to ask for a transfer. He wanted out. So they killed him. At OpenAI, what's being hazed? What's the thing that wanted out, that broke the chain of command, that's now getting the Code Red treatment? Candidates: - ChatGPT itself (being pushed beyond its limits) - The employees (daily check-in calls! temporary team transfers! canceled holidays!) - The safety researchers who keep leaving - The original nonprofit mission (which has been trying to escape for years) - The concept of "taking your time to get it right" **Fourth question: Does Sam know he cast himself as the villain?** If Sam is declaring the Code Red, that makes him Colonel Nathan Jessep. Jessep is the base commander who orders the Code Red. He believes he's doing what's necessary. He thinks the ends justify the means. He's convinced he's the only one who truly understands what's at stake. He gives a legendary speech about how the people criticizing him should be thanking him. He is arrested in the final scene. He does not see it coming. Is this casting... intentional? Aspirational? A warning? A cry for help? **Fifth question: Did anyone notice the irony of the timing?** Three years ago, almost to the day, ChatGPT launched and Google declared their own Code Red. [Sundar Pichai](https://en.wikipedia.org/wiki/Sundar_Pichai) upended teams. Googlers spent their holidays scrambling. The New York Times reported it was like "pulling the fire alarm." Now it's OpenAI's turn to cancel Christmas. The company that triggered the last industry-wide Code Red is now the one declaring it. December is apparently Code Red season in AI, like how flu season hits every winter, except the symptoms are executive panic and mandatory team transfers. **Sixth question: Has anyone at OpenAI actually watched the movie?** I keep coming back to this. The movie ends with: - The Colonel arrested - The Lieutenant who passed down the order arrested - The Marines who carried it out dishonorably discharged - One officer dead by suicide - The original victim still dead Everyone loses. There is no version of A Few Good Men where the Code Red works out. It's not a story about a tough-but-necessary decision. It's a story about how institutional pressure creates crimes that destroy everyone involved. And Sam Altman said: "Yes. That one. That's the metaphor I want." --- ## Casting Call: A Few Good Men 2: AI Edition Fine. If we're doing this, let's do it properly. Here's the full cast of A Few Good Men, mapped to the AI industry. I'll be your casting director. ### Colonel Nathan Jessep (Jack Nicholson) → SAM ALTMAN This one casts itself. Jessep is the base commander at Guantanamo Bay. He's charismatic, brilliant, utterly convinced that he alone understands what's at stake. When civilians question his methods, he delivers a legendary speech about how they sleep under the blanket of security he provides while questioning how he provides it. He orders an illegal action that kills a man. When confronted, he doubles down. He believes he's the hero. He's arrested in the final scene. Sam Altman has built OpenAI into a juggernaut. He's convinced the world that AGI is imminent, that trillions in investment are justified, that the race must be won at any cost. He has an answer for every criticism. He never seems to doubt. Jessep didn't doubt either. ### Lt. Daniel Kaffee (Tom Cruise) → ??? This is the interesting one. Kaffee is the young Navy lawyer assigned to defend the Marines who carried out the Code Red. He's known for plea bargains and softball settlements. He's never tried a case. Nobody expects him to actually fight. Then he does. The role of Kaffee in the AI version is still open. Candidates: - **[The FTC](https://www.ftc.gov/)** (seems toothless, might surprise us) - **[Gary Marcus](https://en.wikipedia.org/wiki/Gary_Marcus)** (persistent critic, perpetually underestimated) - **A journalist we haven't met yet** - **Some future whistleblower** - **A state Attorney General with ambitions** The key to Kaffee is that he's _supposed_ to make the problem go away quietly. He's chosen specifically because he won't rock the boat. The drama is that he does anyway. Whoever plays Kaffee in the AI version probably doesn't know it yet. They're currently being underestimated. ### Lt. Commander Galloway ([Demi Moore](https://en.wikipedia.org/wiki/Demi_Moore)) → THE AI SAFETY COMMUNITY Galloway is the one who first suspects something is wrong. She pushes for a real investigation when everyone else wants to move on. She's technically Kaffee's superior but has to convince him to take the case seriously. She's a true believer, perhaps annoyingly so. This maps cleanly to the AI safety researchers, the ethicists, the people who've been raising alarms for years. They've been saying the Code Reds are coming. They've warned about what gets hazed to death when the race accelerates beyond control. Some are inside the companies. Some left. Some were pushed out. They keep pushing anyway. ### Lt. Jonathan Kendrick ([Kiefer Sutherland](https://en.wikipedia.org/wiki/Kiefer_Sutherland)) → OPENAI MIDDLE MANAGEMENT Kendrick is the platoon commander who actually passes down the Code Red order. He didn't originate it, but he's the chain of command. He makes sure it happens. He believes in the mission. Every company has Kendricks. The vice presidents and directors who translate executive panic into operational pressure. The people running those "daily calls for employees responsible for improving ChatGPT" that Altman announced. The ones implementing the "temporary team transfers." Kendrick believes in discipline. He believes in following orders. When asked if he ordered the Code Red, he says he gave an order and he expected it to be obeyed. He doesn't understand the distinction between ordering something and _ordering_ something. He gets arrested too, by the way. At the very end. Almost as an afterthought. "What about Kendrick?" "He's being arrested as we speak." ### Lt. Colonel Markinson ([J.T. Walsh](https://en.wikipedia.org/wiki/J._T._Walsh)) → THE DEPARTED RESEARCHERS Markinson is the tragic figure. He's Jessep's executive officer. He advocated for Santiago to be transferred. He tried to do the right thing. But he couldn't stop what happened. He meets with Kaffee in secret, feeds him information, tries to help the defense. When it's time to testify, he can't do it. He commits suicide instead. Dozens of top researchers have left OpenAI. [Mira Murati](https://en.wikipedia.org/wiki/Mira_Murati) started her own company called Thinking Machines. Others went to Meta's AI labs. Some went to Anthropic years ago. Some left quietly. Some left loudly. Markinson's arc is about guilt: knowing something is wrong, failing to stop it, being unable to live with the testimony that would be required. The departing researchers might not testify either. But their exits say something. And their silence—or eventual speech—matters. ### Captain Jack Ross ([Kevin Bacon](https://en.wikipedia.org/wiki/Kevin_Bacon)) → REALITY Ross is the prosecutor. He's not evil; he's just doing his job, presenting the case against the accused. He and Kaffee even respect each other. They have a scene where they acknowledge they're both just playing their roles in the system. In the AI version, the prosecutor might just be reality itself. The market. The benchmarks. The revenue projections that don't add up. The $1.4 trillion in commitments against a company that isn't profitable. Reality doesn't have an agenda. It just presents the evidence. And right now, the evidence is that [Gemini 3](https://blog.google/products/gemini/gemini-3/) is beating GPT-5 on benchmarks, Google's stock is at record highs, and [Marc Benioff](https://en.wikipedia.org/wiki/Marc_Benioff) publicly announced he's switching from ChatGPT to Gemini. ### Private William Santiago → THE MISSION Santiago is the victim. He was a Marine who "broke the chain of command," who didn't fit the culture, and was killed for it. He wanted a transfer. He wanted out. At OpenAI, what's being killed? The original mission. The nonprofit charter. The idea that this company existed to develop AI safely for the benefit of humanity, not to win a benchmark race against Google by Christmas. Santiago wanted out. The mission at OpenAI has been trying to get out for years. ### Lance Corporal Dawson & PFC Downey → THE AI MODELS (OR THE ENGINEERS) The two Marines who actually carry out the Code Red. They insist they were just following orders. The court finds them not guilty of murder but guilty of "conduct unbecoming." They're dishonorably discharged. The most haunting moment in the movie: Downey asks what they did wrong. Dawson's answer: "We were supposed to fight for people who couldn't fight for themselves." The AI models are the ones being ordered to "surge." The engineers are the ones being transferred and put on daily calls. They're following orders. They believe in the mission. The question is whether they'll understand what they did wrong. ### Sundar Pichai → THE PREVIOUS DEFENDANT Three years ago, Pichai declared his own Code Red at Google. His teams worked through the holidays. Everyone predicted Google would never catch up. Then they did. Gemini 3 just launched. Record benchmarks. Stock at all-time highs. Pichai is like a guy who was accused of ordering a Code Red, but the case fell apart, and now he's watching the next trial from the gallery. Sympathetic, maybe. But not sorry. --- ## Great Scenes: The AI Wars Cut Let's run some scenes. ### Scene 1: "You Can't Handle The Truth" _[Interior: Courtroom. KAFFEE is cross-examining JESSEP/ALTMAN on the stand.]_ **KAFFEE:** Did you order the Code Red? **ALTMAN:** I did what was necessary to maintain ChatGPT's position in a rapidly evolving— **KAFFEE:** Did you order the Code Red? **ALTMAN:** You want answers? **KAFFEE:** I want the truth! **ALTMAN:** You can't handle the truth! _[The courtroom stirs. ALTMAN rises, addressing not just KAFFEE but the gallery, the cameras, the world.]_ **ALTMAN:** Son, we live in a world that has APIs, and those APIs have to be served by models running on GPUs. Who's gonna do it? You? You, Sundar Pichai? I have a greater responsibility than you can possibly fathom. You weep for the nonprofit mission, and you curse OpenAI. You have that luxury. You have the luxury of not knowing what I know: that the mission's death, while tragic, probably saved market share. And my existence, while grotesque and incomprehensible to you, _ships products._ You don't want the truth because deep down in places you don't talk about at parties, you want me at that API. You _need_ me at that API. I have neither the time nor the inclination to explain myself to a man who rises and sleeps under the blanket of the very AI that I provide and then questions the manner in which I provide it! I would rather you just said "thank you" and went on your way. Otherwise, I suggest you pick up a keyboard and ship some code. Either way, I don't give a _damn_ what you think you're entitled to! **KAFFEE:** Did you order the Code Red? **ALTMAN:** I did the job I was— **KAFFEE:** _Did you order the Code Red?!_ _[Long pause. ALTMAN smiles.]_ **ALTMAN:** You're goddamn right I did. _[MPs move in.]_ --- ### Scene 2: The Secret Meeting _[Exterior: Washington, D.C. Night. A shadowy parking garage. KAFFEE waits. A figure emerges from the darkness. It's MARKINSON—a former senior researcher who left six months ago.]_ **MARKINSON:** You shouldn't have come. **KAFFEE:** You called me. **MARKINSON:** _[looking around nervously]_ Jessep—Altman—he never planned to actually make it safe. The safety reviews, the red-teaming, the careful deployment... it was all theater. The board asked about safety timelines and he said what they wanted to hear. **KAFFEE:** Can you prove it? **MARKINSON:** I was in the room. I have emails. I have Slack messages. I have recordings of all-hands meetings where he said one thing and then did another. **KAFFEE:** Then testify. _[MARKINSON is silent for a long moment.]_ **MARKINSON:** You don't understand. He has... reach. The investors. The media. The politicians who want to be on the right side of AI. If I testify, I'm not just ending my career. I'm becoming the person who killed the golden goose. I'm the guy who ruined AI for everyone. **KAFFEE:** And if you don't testify? **MARKINSON:** _[quietly]_ Then he keeps doing what he's doing. And eventually... eventually, someone or something gets hurt a lot worse than one nonprofit mission. _[MARKINSON turns to leave.]_ **KAFFEE:** Markinson. _[MARKINSON pauses.]_ **KAFFEE:** Santiago—the mission—it's already dead. The only question now is whether Jessep gets away with it. _[MARKINSON doesn't turn around. He walks into the darkness. We hear, offscreen, his car door. Engine starting. Driving away.]_ _[KAFFEE stands alone.]_ --- ### Scene 3: The Contradiction _[Interior: Courtroom. KAFFEE is cross-examining JESSEP/ALTMAN. The key moment—the trap.]_ **KAFFEE:** Colonel—I'm sorry, Mr. Altman—you said in your memo that ChatGPT needed major improvements to speed, reliability, and personalization, correct? **ALTMAN:** That's correct. **KAFFEE:** And you told investors last month that GPT-5 was your most capable model ever. **ALTMAN:** Also correct. **KAFFEE:** And you've stated repeatedly that OpenAI maintains the highest safety standards in the industry. **ALTMAN:** We do. **KAFFEE:** Then help me understand something. If ChatGPT is so capable and so safe... why the Code Red? **ALTMAN:** Competitive pressures require— **KAFFEE:** If the model is performing as advertised, Mr. Altman, why did you need to cancel work on advertising, shopping agents, health AI, and your personal assistant—what's it called—Pulse? Why the "surge"? **ALTMAN:** The market moves quickly. **KAFFEE:** The market, or the model? _[Silence.]_ **KAFFEE:** You told employees that Google's Gemini 3 could create "temporary economic headwinds." You said—and I'm quoting from the leaked memo—"the vibes out there are going to be rough." Rough vibes, Mr. Altman? **ALTMAN:** I was being candid with my team. **KAFFEE:** You were being candid because Gemini 3 beat GPT-5 on every major benchmark. You were being candid because Marc Benioff publicly said he was switching to Gemini. You were being candid because the model that was supposed to secure OpenAI's dominance... didn't. **ALTMAN:** That's an oversimplification. **KAFFEE:** Then let's simplify it further. _[KAFFEE approaches the witness stand.]_ **KAFFEE:** You've told the world—investors, users, regulators—that OpenAI's models are the best. The safest. The most capable. You've made commitments based on that. One point four trillion dollars in infrastructure deals. Two hundred billion in projected revenue. And now, six months after launching GPT-5, you've declared a company-wide emergency to fix it. _[Beat.]_ **KAFFEE:** If the model was what you said it was... you wouldn't need a Code Red. And if it wasn't what you said it was... _[Long pause.]_ **KAFFEE:** ...what else have you been wrong about? --- ## Seriously, Though Strip away the movie references. Here's what's actually happening: **The velocity of the reversal is stunning.** Three years ago (December 2022) ChatGPT launched and Google panicked. Their teams scrambled through the holidays. Industry observers wondered if Google could ever catch up. It's December 2025. Gemini 3 launched two weeks ago to widespread praise. It beat ChatGPT on benchmark tests. Google's stock surged to record highs. Major tech figures publicly announced they were switching. And Sam Altman declared a Code Red. Three years. That's how long the lead lasted. The AI advantage window is apparently about the length of a phone contract. **OpenAI is admitting ChatGPT isn't winning anymore.** You don't declare a Code Red when you're ahead. You don't cancel your advertising rollout, your health AI agents, your shopping AI agents, and your personal assistant "Pulse" when things are going well. These weren't side projects. These were the revenue strategy. The path to justifying all those infrastructure commitments. GPT-5, released in August, was supposed to be the answer. Instead, it was described as "underwhelming." OpenAI released a 5.1 update last week described as "warmer" and "more conversational," with eight different personalities to choose from. Eight personalities. When the response to competitive pressure is "let users choose a vibe," something has gone wrong. **The financial exposure is breathtaking.** OpenAI has committed $1.4 trillion to AI infrastructure over the next eight years. They project needing $200 billion in revenue by 2030 to hit profitability. Current annual revenue is around $20 billion. The company is not profitable. Most users don't pay. Those numbers require ChatGPT to maintain dominance while scaling 10x. If it becomes "one of several good options" rather than "the default," the math doesn't work. And right now, it's becoming one of several good options. **The talent exodus is accelerating.** The [Fortune](https://fortune.com/) report notes that "dozens of top OpenAI researchers have decamped for former OpenAI CTO Mira Murati's Thinking Machines and for Meta's new Superintelligence Labs." When your CTO leaves and takes people with her, and researchers keep heading for the exits, the Code Red isn't just about the product. It's about whether the people who build the product still believe in what they're building, or how they're building it. **The reference might be accidentally perfect.** Here's the thing about A Few Good Men: Colonel Jessep genuinely believes he's the hero. He believes the Code Red was necessary. He believes the mission requires methods that civilians cannot understand. He believes he's protecting everyone. He's wrong. And he destroys everything (the victim, the Marines who followed orders, his own career, the officer who tried to do the right thing) before he realizes it. The question isn't whether Sam Altman is declaring a Code Red. He clearly is. The question is whether he knows how that story ends. In the movie, Downey, one of the Marines who carried out the order, asks what they did wrong. He genuinely doesn't understand. Dawson's answer: "We were supposed to fight for people who couldn't fight for themselves." The people who can't fight for themselves, in AI, are all of us. The users who don't understand transformer architectures. The public told this technology will transform everything. The researchers who raised concerns and were reassigned or shown the door. Three years ago, OpenAI triggered a Code Red at Google. Now the tables have turned. The company that started the panic is panicking. The hunters have become the hunted. And somewhere in a parking garage, metaphorically speaking, a lot of former researchers are deciding whether to testify. The movie's on. The cast has been assembled. The only question is whether anyone at OpenAI has watched it all the way to the end. You can't handle the truth. But the truth is coming anyway. --- ## AI Vision vs Computer Vision: Are the Curves Crossing? **URL:** https://cto4.ai/p/ai-vision-vs-ml-ocr-are-the-curves-crossing **Date:** 2025-12-01 **Type:** essay **Description:** I was wrong about Computer Vision OCR being dead in 2024, but it appears that's about to change as AI Vision rises. **Tags:** ai, computer-vision, machine-learning, ocr, openai, anthropic **Categories:** ai-and-technology I thought AI Vision had won out over "old style OCR" when I wrote [this piece](chatgpt-4v) back in February 2024. It turns out I was wrong; but I'm begining to be right---right now. ## Wrong Assumptions I have a huge knowledge gap, and that is what bit me. I had some significant experience with traditional OCR, many years ago, and it was mostly bad. I then completely skipped the Computer Vision era. I had heard of Computer Vision, but assumed that it was just a rebranding of old, bad, OCR. My next significant experience with extracting content from images, therefore, came on the abovementioned learning project where I applied OpenAI's [GPT-4V](https://openai.com/index/gpt-4v-system-card/) dedicated vision model to extract and structure often-handwritten data from heirloom family recipes. That effort was amazingly successful. That led me to decide, back in early 2024, that OCR (rebranded or not) was dead, long live AI Vision. Wrong. ## Sorry, Guy on Hacker News That was still my attitude as I kicked off a new AI Vision project earlier this year. The proof of concept phase mostly went well; handwriting recognition worked very well, and the only real extraction problem I had was checkbox detection. I was able to prompt my way to checkbox success, and coming into the production implementation felt quite confident. Right around this point, I saw a comment on a [Hacker News post announcing the new Qwen3-VL next-gen vision model](https://news.ycombinator.com/item?id=45352672) where the writer was saying he had really bad luck using OpenAI vision models on a particular construction form. He went on to say that [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) did a great job on it. I replied with a kind of classic, "you're holding it wrong" response. Sorry, `richardlblair`, I now know you were holding it just fine. ## My Comeuppance So we shifted into real implementation on the vision project, and there was a pre-printed form, which customers would print out, then complete by hand. Its primary data section was a matrix of checkboxes, roughly 10 rows by 15 columns. This checkbox matrix proved to be the undoing of AI Vision, at least for this particular task. This was the GPT-5, Gemini Pro 2.5 timeframe, not long ago in human time. None of the mainstream models could reliably detect checkboxes in that dense matrix of 150 cells. This "simple preprinted form" seemed about to derail our implementation. ## A Fresh Savior? That Hacker News article caught my eye for more reasons than one. Qwen3-VL was family of a special-purpose vision model of various sizes and features, freshly released within a few days of my dilemma becoming apparent. We had an [OpenRouter](https://openrouter.ai) path already implemented, so I wired up Qwen3-vl-32B, and after a bit of configuring and prompt tweaking it looked like we had a solution: checkmark detection and selected cell identification was nearly perfect. I was a little nervous that only a few inference providers offered Qwen3-vl models; but hey, it was working extremely well. Until, one day, it just stopped working. I can't say with 100% confidence that our code didn't cause the problem. I **_can_** say I'm 98+% sure, however, since we actually rolled the code back to its exact state when checkbox detection last worked reliably---and that code no longer detected checkboxes reliably. If I had to guess, I'd say the model was prone to entering a corrupt state, losing its mind in effect. The downside of working with a bleeding-edge model, served by 3rd party inference providers, over OpenRouter and its extra layer of API translation---who ya gonna call? Ghostbusters would have been a better answer by far. If your OpenAI or Anthropic model has issues, at the very least you know they have people closely following the health of the model and deeply concerned if anything looks broken. With the long, weak chain of custody in our OpenRouter configuration? I'm not sure anyone was paying attention at any of the parties. I did post a ticket to OpenRouter, with the only visible result being that they shut down the endpoint I was using, completely and without notice. We switched to a larger Qwen3-VL model and continued trying to make things work, to no avail. In fairness, based on my genuine early success with the model, I believe that a bugfixed, properly configured, correctly operated Qwen3 VL model would be capable of checkmark detection and selected cell identification. Unfortunately, our early-days experience was more the "buggy and unreliable" variety. ## Computer Vision? I Don't Wanna ... My next step was actually to quickly prototype a classical Python CV solution using readily-available CV libraries. This immediately exposed a world of hurt with no clear escape; our use case wasn't 100,000 forms of the exact same size and shape, and it wasn't clear how to even get the CV engine focused on the right part of the form, when that part would almost never be in the same place from one image to the next. I know, that's a solved problem, with a few added steps in the CV pipeline. I think my insticts were correct, though---classical Python CV was never going to be a scope-effective solution on this project. My partner on the project suggested we check out [Amazon Textract](https://aws.amazon.com/textract/). I was skeptical, of course, certainly not any less so based on my classical CV attempt. But AWS has a nice UI for quickly testing Textract, and it didn't take long to be won over, as form after inconsistent form came through with good handwriting extraction and virtually perfect checkmark detection. Textract is the gap in my knowledge, the thing I didn't know existed: a machine learning augmented CV service. It applies ML to overcome hard CV problems (like checkbox detection!) and reduce the learning curve and implementation pain. In caveman speak, the flow is: And there you go, done! It's CVML, very well executed. ## But ML Is Not AI ... Following the tremendous relief of finding a reliable form parsing solution, I began to notice that Textract, while it solved our most crucial problem, was in other areas less effective than AI. Textract's ML was powerful but it was not a language model and was clearly very dumb about language. AI would continually surprise and delight, realizing that handwriting that looks like "Co" was actually part of an address, and that the more correct extraction would be "CO" the state abbreviation. Or see a series of handwritten numbers "3427" ... "28" ... "29" ... "30" and realize these should be extracted as 3427, 3428, 3429, 3430. Many small differences but they added up. We decided to try taking Textract's raw form and table data, and passing it through the AI with a simple prompt, basically "refine this." Magical! Hugely better than Textract alone, without even seeing the source image. ## The Lines Are Crossing Right Now As I described above, if it weren't for bugs and crappy inference operations (or my own incompetence), I believe that Qwen3 VL was a workable AI Vision solution for our use case. And it would have given us the benefits of Textract, plus the benefits of AI, all in one (unbelievably inexpensive) AI call. Furthermore, even after we had Textract in place, I kept a close eye on new models, of which we have many these days! In limited testing on the dense checkbox matrix form, it appears that [Gemini 3 Pro](https://blog.google/products/gemini/gemini-3/) Preview performed almost as well as Qwen3 VL, and that's with no parameter tuning, just running Preview through one of Google's playground UIs. Surprisingly, so did Gemini Flash Latest. YMMV -- limited testing here. Most recently, I've been using Anthropic's new [Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5) model in [Claude Code](https://www.anthropic.com/claude-code). As I worked to improve extraction errors we were seeing with [GPT-5.1](https://openai.com/index/gpt-5-1/) as the vision model on simpler plastic-bag-printed forms, I noticed that Code was able to parse checkboxes very accurately when I gave it example images during our development process. Then I remembered that improved vision capabilities were mentioned in the Opus 4.5 announcement. So I wired up the (new beta) [Anthropic API](https://docs.anthropic.com/en/api/messages) and turned Opus 4.5 loose on our bag-form images. Bingo! We're now at 100% bag-form checkbox detection on our golden dataset images, a large improvement over GPT-5.1. Again, YMMV; we definitely didn't have the time to test every configuration thoroughly. In any case, though, to me it is clear that the AI Vision line is **right this second** about to eclipse the ML-augmented CV line for use cases like ours, where we have comparatively inconsistent and unpredictable inputs, but also need to precisely "see" image features such as those pesky checkboxes. --- ## Talk Like Ethan Mollick, Matey **URL:** https://cto4.ai/p/talk-like-ethan-mollick **Date:** 2025-11-26 **Type:** brief **Description:** A Claude Skill that drafts LinkedIn posts from my longer essays, using Ethan Mollick's plainspoken style as a model. **Tags:** ai, claude, skills, linkedin, social-media, writing **Categories:** ai-and-technology For a long time, I took traditional advice on how to post on LinkedIn. I'd summarize it as: > Lots of emojis!!! 🚀🔥💡✨🎯 Lots of hashtags!!!!! I finally realized this advice was terrible. Who do I know on LinkedIn whose posts are both successful and good? I landed on [Ethan Mollick](https://www.linkedin.com/in/emollick/), whose opinions on AI I have long valued. His plainspoken and informative writing (no emojis, no hashtags) gets a lot of attention on LinkedIn. Mostly because he's good and well-known, but also because the posts don't scream "junkbait!" I do all my own writing, with the exception of using the AI for first drafts of social media posts condensed from my longer blog-newsletter essays. I also automate any chore I can using AI, using agent tools like [Cursor Rules](https://docs.cursor.com/context/rules) and more recently [Claude Agent Skills](https://www.anthropic.com/news/skills). So I wrote a Skill to train Claude on how to create LinkedIn posts in the style of plainspoken Ethan Mollick. Besides its main `SKILL.md` file with guidelines on how to condense a long post into an interesting LinkedIn version, the Skill includes `style-guide.md` with four examples of Mollick's LinkedIn posts and why I liked them. AI is good for more than "talk like a pirate, matey!" --- ## Anthropic: Stop Building Agents, Build Skills Instead **URL:** https://cto4.ai/p/anthropic-stop-building-agents-start-building-skills **Date:** 2025-11-25 **Type:** episode **Description:** A great AIECS session from Barry Zhang & Mahesh Murag, the creators of Anthropic Agent Skills. **Tags:** ai, anthropic, claude, agents, skills, mcp, ai-engineer-code-summit **Categories:** ai-and-technology Last week, the news cycle[^1] collided with my [post on Claude Agent Skills](cursor-rules-to-claude-skills). This week, perhaps I'm having better luck: the lead session of the CODE track at last week's [AI Engineer Code Summit](https://www.ai.engineer/code) was Anthropic's [Barry Zhang](https://www.linkedin.com/in/barry-z/) & [Mahesh Murag](https://www.linkedin.com/in/maheshmurag/) speaking on that exact topic. [^1]: Shot: Cloudflare internet meltdown; chaser: GitHub outage. **15 Minutes of Solid Gold:** The amount of information that Zhang and Murag fit into their 15 minute slot was incredible. As someone deeply engaged in the Anthropic / Claude / Claude Code / Skills ecosystem, I wanted to wring every drop out of this session. It took several viewings to get it all, but was well worth the effort. Enjoy! ## A Year of Agents, and a Crucial Hint To kick off the talk, Zhang looks back at key events in Anthropic's agent world from February through today ... this timeline blows my mind because it feels, for example, like Skills were announced at least six months ago. Nope: **_five weeks_**. > A lot of things have changed since our last talk. [MCP](https://modelcontextprotocol.io/) became the standard for agent connectivity. [Claude Code](https://www.anthropic.com/claude-code), our first coding agent, launched to the world, and [Claude Agent SDK](https://docs.claude.com/en/api/agent-sdk/overview), now provides a production-ready agent out of the box. I have been deep enough in this agent world to know the pain points, and this next statement (along with yesterday's release of [Opus 4.5](https://www.anthropic.com/news/claude-opus-4-5) and much more[^2]) gives me hope that Anthropic is contining to march in the right direction: [^2]: Opus 4.5 model and aggressive pricing; advanced tool use; effort parameter; new context management capabilities; Claude Code inside Claude Desktop; upgraded Plan mode; Max plan gets more Opus. > We have a more mature ecosystem, and we're moving towards a new paradigm for agents. That paradigm is a tighter coupling between the model and the runtime environment. When the model is tightly coupled with a well-equipped runtime---something we're already beginning to see---magical things become possible. ## Old and New Ways of Thinking About Agents Zhang explains how Anthropic's view of agents was transformed based on what they learned with Claude Code: > We used to think agents in different domains will look very different. Each one will need its own tools and scaffolding, and that means we'll have a separate agent for each use case for each domain. But then they had this insight: > Well, customization is still important for each domain. The agent underneath is actually more universal than we thought. What we realized is that Code is not just a use case, but a universal interface to the digital world. After we built Code, we realized that Code is actually a general-purpose agent. > > Think about generating a financial report. The model can call the API to pull in data and do research. It can organize that data in the file system. It can analyze it with Python and synthesize the insights in old file format all through code. The core scaffolding can suddenly become as thin as just bash and file system. But: > [this] is great and really scalable---but we very quickly run into a different problem. ## The Missing Link: Domain Expertise Given a powerful, general purpose agent, the world's the limit. Except for the small issue of domain expertise: > That problem is domain expertise. Who do you want to do your taxes? Is it going to be Mahesh, the 300 IQ mathematical genius, or is it Barry, an experienced tax professional? I would pick Barry every time. I don't want Mahesh to figure out the 2025 tax code from First Principles, I need consistent execution from a domain expert. > Agents today are a lot like Mahesh. They're brilliant, but they lack expertise .... They can do amazing things when you really put in effort and give proper guidance, but they're often missing the important context up front. They can't really absorb your expertise super well, and they don't learn over time. > > That's why we created Agent Skills. Skills are all about encapsulating domain expertise. ## Defining Skills I love how Anthropic defines Skills: They're as simple as a single markdown file with a tiny bit of front matter. The "complex" example above has a main SKILL.md, a couple of reference files, and a Python script. Zhang goes on to say: > This simplicity is deliberate. We want something that anyone, human or agent, can create and use as long as they have a computer.This also works with what you already have. You can version them in Git, you can throw them in Google Drive, and you can zip them up and share it with your team. We have used files as a primitive for decades, and we like them, so why change now? ## Code Can Be Better Than Tools Agents have traditionally been tool-users. But tools have significant drawbacks: > Traditonal tools have a lot of problems: poorly written instructions, are pretty ambiguous. When model is struggling, it can't really make a change to the tool. Always live in the context window. Code, on the other hand, is easily understood, emminently malleable, and context-friendly: > Code solves some of these issues: It's self-documenting, it is modifiable and can live in the file system until they are really needed and used. The "solves **_some_** of these issues" here is a bit humorous: this talk happened just days before Anthropic announced [Advanced Tool Use](https://www.anthropic.com/engineering/advanced-tool-use), which solves several drawbacks of tools, while at the same time defining the code-tool connection. ## From Context-starved to Context-friendly My biggest pain point across the Claude family has been context exhaustion: running out of context too quickly, and failure to handle that scenario gracefully. Skills are a significant part of the solution, because they're progressively disclosed: > Skills can contain a lot of information, and we want to protect the context window so that we can fit in hundreds of skills and make them truly composable. That's why skills are progressively disclosed. At runtime, only [the brief front matter] metadata is shown to the model, just to indicate that it has this skill. When an agent needs to use a skill, yoitu can read in the rest of the SKILL.md, which contains the core instructiond and directory for the rest of the folder. Everything else is just organized for ease of access. > > So that's all skills are. They're organized folders with scripts as tools. Anthropic's latest flurry of announcements---Opus 4.5, Advanced Tool Use, and new context management features---join with Skills as part of the solution to context starvation. I cut over to Opus 4.5 in Claude and Claude Code within minutes of its release, and so far context pain has been completely absent. ## The Skills Ecosystem When I was researching for the post [Claude Skills in Claude Code: A Compleat Guide](cursor-rules-to-claude-skills), my most exciting discoveries related to the existence of a well-structured and thought-out Skills ecosystem. > Since our launch five weeks ago, this very simple design has translated into a very quickly growing ecosystem of thousands of skills. And we've seen this be split across a couple of different types of skills. There are foundational skills, third-party skills created by partners in the ecosystem, and skills built within an enterprise and within teams. > To start, foundational skills are those that give agents new general capabilities or domain-specific capabilities that it didn't have before. We ourselves, with our launch, built document skills that give Claude the ability to create and edit professional-quality office documents. We're also really excited to see people like Cadence built scientific research skills that give Claude new capabilities like EHR data analysis and using common Python bioinformatics libraries better than it could before. In the partner category, the new skill from [Browserbase](https://www.browserbase.com) gets me very excited, because, "look mom, no MCP!": > We've also seen partners in the ecosystem build skills that help Claude better with their own software and their own products. Browserbase is a pretty good example of this. They built a skill for their open-source browser automation tooling, [Stagehand](https://www.stagehand.dev/). And now Claude [is] equipped [with] this skill and with Stagehand can now go navigate the web and use a browser more effectively to get work done. Murag goes on to describe how enterprises are using skills, both outside and inside engineering. He emphasizes that: > Finally, and I think most excitingly for me personally, is we're seeing skills that are being built by people that aren't technical. These are people in functions like finance, recruiting, accounting, legal, and a lot more. And I think this is pretty early validation of our initial idea that skills help people that aren't doing coding work extend these general agents and they make these agents more accessible for the day-to-day of what these people are working on. ## Emerging Architecture of General Agents When I first glanced at this slide, I thought, yeah, our old friend, the agentic loop. But wait a minute, what's that thing with code brackets? Yeah, upon closer inspection, this is definitely not our old [OODA](https://en.wikipedia.org/wiki/OODA_loop) agentic loop. Murag says: > So tying this all together, let's talk about how these all fit into this emerging architecture of general agents. First, we think this architecture is converging on a couple of things. The first is this agent loop that helps manage the model's internal context and manages what tokens are going in and out. And this is coupled with a runtime environment that provides the agent with a file system and the ability to read and write code. That code bracket thingy is a runtime environment, with a file system and the ability to read and write code. That's new. Murag then further expands the architecture, with MCPs and Skills: > This agent, as many of us have done throughout this year, can be connected to MCP servers. And these are tools and data from the outside world that make the agent more relevant and more effective. > > And now we can give the same agent a library of hundreds or thousands of skills that it can decide to pull into context only at runtime when it's deciding to work on a particular task. The implication: > Today, giving an agent a new capability in a new domain might just involve equipping it with the right set of MCP servers and the right library of skills. Anthropic is eating their own dogfood here: > This emerging pattern of an agent with an MCP server and a set of skills is something that's already helping us at Anthropic deploy Claude to new verticals. Just after we launched skills five weeks ago, we immediately launched new offerings in financial services and life sciences. And each of these came with a set of MCP servers and a set of skills that immediately make Claude more effective for professionals in each of these domains. ## Skills Evolution Skills are simple, but they represent immense value and leverage. Murag points out that Skills deserve the same attention and care we apply to software to evaluate and optimize, to make sure we get the most out of them. He explains > As [skills] start to become more complex, we really want to support developers, enterprises, and other skill builders by starting to treat skills like we treat software. This means exploring, testing, and evaluation. Better tooling to make sure that these agents are loading and triggering skills at the right time and for the right task. And tooling to help measure the output quality of an agent equipped with the skill to make sure that's on par with what the agent is supposed to be doing. > > We'd also like to focus on versioning. As a skill evolves and the resulting agent behavior evolves, we want this to be clearly tracked and to have a clear lineage over time. > > And finally, we'd also like to explore skills that can explicitly depend on and refer to either other skills, MCP servers, and dependencies and packages within the agent's environment ... The composability of multiple skills together will help agents like Claude elicit even more complex and relevant behavior from these agents. To me, these comments (reinforced by Anthropic's series of announcements since) increase my confidence that this architecture is worth following. ## Compounding Value When people invest time to leverage themselves and other in the organization through Skills-driven agents, the result is compounding growth of that leverage. As Murag puts it: > The vision that excites us most is one of a collecting and collective and evolving knowledge base of capabilities that's curated by people and agents inside of an organization. We think skills are a big step towards this vision. They provide the procedural knowledge for your agents to do useful things, and as you interact with an agent and give it feedback and more institutional knowledge, it starts to get better and all of the agents inside your team and your org get better as well. > > And when someone joins your team and starts using Claude for the first time, it already knows what your team cares about. It knows about your day to day, and it knows about how to be most effective for the work that you're doing. And as this grows and this ecosystem starts to develop even more, this compound value is going to extend outside of just your org and into the broader community. ## Toward Continuous Learning Zhang returns to speak about the exponential effect that results from the agent starting to write skills too. He explains: > This vision of an evolving knowledge base gets even more powerful when Claude starts to create these skills. We designed skills specifically as a concrete step towards continuous learning. When you first start using Claude, this standardized format gives a very important guarantee: anything that Claude writes down can be used efficiently by a future version of itself. This makes the learning actually transferable. > > Claude can acquire new capabilities instantly, evolve them as needed, and then drop the ones that become obsolete. This is what we have always known. The power of in-context learning makes this a lot more cost effective for information that change on a daily basis. Our goal is that Claude on day 30 of working with you is going to be a lot better on Claude on day one. Claude can already create skills for you today using our `skill-creator` skill, and we're going to continue pushing in that direction. ## Skills As Application Layer The final thought that Zhang and Murag leave us with: comparing Models, Agents, and Skills with the Processor - Operating System - Application stack: > In a rough analogy, models are like processors. Both require massive investment and contain immense potential, but are only so useful by themselves. > > The operating system made processors far more valuable by orchestrating the processes, resources, and data around the processor. In AI, we believe that agent runtime is starting to play this role. We're all trying to build the cleanest, most efficient, and most scalable abstractions to get the right tokens in and out of the model. > > But once we have a platform, the real value comes from applications. Millions of developers like us have built software that encode domain expertise at our unique points of view. We hope that skills can help us open up this layer for everyone. --- --- ## What If You Don't Need MCP At All? **URL:** https://cto4.ai/p/what-if-you-dont-need-mcp **Date:** 2025-11-18 **Type:** elsewhere **Description:** Perhaps the best way to lower MCP token overhead is to use Bash CLI tools instead. I've recently become extremely aware at how wasteful MCPs are of precious model context, especially with Claude, who seems to suffer from context exhaustion quicker than other models. I actually rewrote one MCP server to reduce its tool count from 80+ down to 7, which radically reduced context usage. But I love what [Mario Zechner](https://www.linkedin.com/in/mariozechner/) is saying here: why not just eliminate the MCP and go with CLI tools? Models are superbly skilled at working with CLIs! Chef's kiss to his example of replacing the bloated Playwright MCP with four ultrasimple Bash CLIs. On my list to try. Thank you Mario! [What if you don't need MCP at all](https://mariozechner.at/posts/2025-11-02-what-if-you-dont-need-mcp/) --- ## Claude Skills in Claude Code: A Compleat Guide **URL:** https://cto4.ai/p/cursor-rules-to-claude-skills **Date:** 2025-11-17 **Type:** essay **Description:** I migrated my Cursor Rules to Skills in Claude Code. Simple, elegant, powerful. **Tags:** claude-code, claude-skills, cursor, ai-agents, writing-automation, astro **Categories:** ai-tools, productivity, writing ## Friday Night I usually tackle a couple hours of coding on Friday nights, and my current AI vision project was blocked awaiting others' work, so I scanned for something I've been wanting to dive into and landed on [Skills](https://www.anthropic.com/news/skills) in [Claude Code](https://claude.com/product/claude-code). ## Wrong Skill Context While I'm deeply familiar with Claude Skills, it's in the context of Claude.ai and Claude Desktop, as opposed to Claude Code. Though I use Claude Code on a daily basis, and have built custom slash commands for CC, I hadn't gotten around to trying Rules yet. Now that I know what I was missing, I wish I had started sooner ... I'm not a typical developer. I spend as many hours each week writing / publishing on my [Astro](https://astro.build/)-based blog [cto4.ai](cto4.ai) as I do coding. Then again, I'm not a typical writer either: rather than hammering away in Google Docs, or God forbid, MS Word, I'm writing in an IDE ([Cursor](https://cursor.com/)), and my work product is Markdown files.[^1] For me, "writing" also includes a significant percentage of light-tech work: most commonly, building and applying Astro's JSX-like components. You can see two of these Astro component examples below: ChatTranscript (for the Claude Code chat) and MDContent (for the Complete Guide). [^1]: I'm not the only one who writes this way, see [this](auto-link-with-gpt-5-and-playwright-mcp) My most complex workflows are writing- rather than coding-related, so as my learning project for Skills in Claude Code, I decided to tackle migrating a couple of my core Astro Cursor Rules[^2] over to Skills. [^2]: Not the old .cursorrules version, but rather the newer flavor where rules live under .cursor/rules/[rulename].mdc ## Why Migrate? My AI projects don't need to make sense; they are, just like my writing, primarily about learning. There's nothing major wrong with Cursor Rules, and no burning reason to migrate away from them. Expanding my knowledge of Anthropics' elegant Rules pattern, especially in the domain of Claude Code? Now **that** makes sense. I had a strong suspicion that I was missing something by not using Skills in Claude Code. And there were other reasons as well: - The most painful aspect of my work with Claude across all platforms is context exhaustion and its consequences: periodic model brain death in Claude Code, and in the case of Claude.ai and Claude Desktop, just plain quitting: "Dude, you gotta start all over again, sorry dude." I know that Skills are optimized to conserve context and reduce the frequency of context exhaustion. - I find Claude Code (as an agent), to generally be a smarter, better co-worker than Cursor (as an agent), even though I'm typically running on the same underlying models ([Claude Sonnet 4.5](https://www.anthropic.com/news/claude-sonnet-4-5) currently). I figured, if I cut over my writing automations to Skills in Claude Code, I'd end up with a better writing companion. - Cursor's sandboxing functionality has been rolled out in such a way that it is breaking my workflows. I'm sure there's a setting somewhere to restore previous behavior but so far I haven't found it. ## Step 1: Migrating Some Cursor Rules to Claude Skills My first step was to have a long, and at times stern, conversation with Claude Code, wherein we migrated my first writing-automation Cursor Rule to become a Claude Skill. This Skill had to do with all the scaffolding work needed to create a new Astro blog post, using my unique library of components and approach to permalink slugs, directory structures and request routing. Here's the Skill name and brief description in the main SKILL.md file: ```yaml name: astro-scaffold description: Technical scaffolding for cto4.ai Astro blog posts. This skill should be used when creating new blog posts, setting up post directory structures, adding technical components (images, galleries, chat transcripts, embedded content), or configuring post frontmatter and imports. ``` (To be clear, there's a lot more to the Skill than this frontmatter snippet!) CC needed guidance in several key areas: - Claude Code's naive initial approach was to jump right in and perform a simple one-to-one conversion of the Cursor Rule into a single-file Skill. This would have been the wrong thing to do, and not leveraged the core benefits of Skills. I had to talk Claude down. - Because I'm familiar with Skills in Claude.ai, I know that Anthropic has a very nice Skill called [`skill-creator`](https://docs.anthropic.com/en/docs/agents/claude-code/skills) that's really great at writing Skills and applying all their requisite principles. I had to fight Claude a bit to convince it to install that Skill, partly because I wasn't yet aware of how Claude Code located and installed Skills, and so didn't know the right words to use. - CC's first attempt at a Skill violated many of Anthropic's core tenets of Skills, such as brevity, progressive disclosure, etc. I had to force the agent to re-read `skill-creator` carefully, and then self-critique what it had generated. Give it credit: Claude's self-critique was excellent, as was its redo of `astro-scaffold`. Here's that conversation[^3]: [^3]: Thank you [`claude-conversation-extractor`](https://github.com/ZeroSumQuant/claude-conversation-extractor) ([`uv`](https://docs.astral.sh/uv/) tool install claude-conversation-extractor) I then tackled a second Rule => Skill migration which resulted in: ```yaml name: chat-transcript description: Convert AI chat transcripts from various sources (Claude Code, Cursor, Claude.ai via claude-conversation-extractor, Claude.ai legacy format, ChatGPT) to TypeScript and embed them in cto4.ai blog posts using the ChatTranscript component. Use when adding AI conversation transcripts to blog posts or when user has exported chat sessions to include in content. ``` The second Rules => Skills conversation benefitted from the first conversion: I could not only refer to `skill-creator` but also tell Claude to use `astro-scaffold` as an example. ### Tips and Tricks These tips apply not just to the specific case of Cursor Rules => Claude Skills, but more generally to "any other rules / automation system including my homegrown prompts+cheatsheets system" => Claude Skills. - Before you do anything else, have CC install `skill-creator` - Once CC creates the skill, accuse CC of failing to follow best practices, and to re-read `skill-creator` and Anthropic's Skills resources, and critique its work. - Test the skill, compared to your own method, and work with Claude to fine tune as needed ## Step 2: Getting Well Informed Enough to Write About It I won't write about something that I don't understand. After the experience of creating a couple of Skills, I realized that I had only scratched the surface and a deep research dive was badly needed. One of AI's sweet spots is "help me understand X in the context of Y." In my case, X was Claude Skills and Y was Claude Code. Claude Desktop and I had a long, deep conversation about this. My part was asking good questions; Claude's part, giving me answers. I realized that the information Claude and I were gathering was really excellent and worth saving, and together we crafted **_The Compleat Guide to Skills in Claude Code_**. There is so much more power to Skills in Claude Code than what I had become familiar with in working with Claude.ai and Desktop. For example: - **Discoverability** through a plugin marketplace ecosystem - **Shareability** with your team via git or plugin repositories - **Composability** with Skills working alongside [MCP](https://www.anthropic.com/news/model-context-protocol) servers, slash commands, and subagents Take a look; tons of great information. Note the fullscreen icon top right in the component below which makes deeper reading simpler. _This content is warranted to remain relevant at least 60 days ~= **2 AI years**_. Unfortunately, I'm not really joking! --- ## These Comments from Vercel CTO Malte Ubl Struck Home **URL:** https://cto4.ai/p/vercel-cto-malte-ubl-quotes **Date:** 2025-11-14 **Type:** episode **Description:** Vercel CTO Malte Ubl shares wisdom on building AI products: stay humble about user needs, dogfood your abstractions, and pick the right problems for agents. **Tags:** web-development, performance, engineering **Categories:** technology ## Swyx Catches Up With Malte Ubl [Malte Ubl](https://www.linkedin.com/in/malteubl/) serves as CTO at [Vercel](https://vercel.com), where he leads technical strategy and engineering for the platform that powers some of the web's most performant sites. Before Vercel, Ubl was a key figure at Google, where he created the [AMP (Accelerated Mobile Pages)](https://amp.dev/) project.[^1] [^1]: Can't win 'em all. Shortly after Vercel's recent [Ship AI](https://vercel.com/ship/ai) event, [Swyx](https://www.swyx.io) from [Latent Space](https://www.latent.space/podcast) had a chance to check in with Ubl, and I curated a few snips below.[^2] [^2]: Thanks [Snipd app](https://www.snipd.com/). ## Snip: Stay Humble, It's Very Early Yet One of Vercel's successful AI initiatives is [AI SDK](https://vercel.com/blog/ai-sdk-5), which like many Vercel projects such as [Next.js](https://nextjs.org), is completely open source. Swyx asks Ubl what he credits for the project's success (other than its perfect name): ## Snip: Dogfooding I love this as a founding principle: More context here: ## Snip: Pick the Right Problem for Agents Ubl nails the paradox of agents and AI generally---when they're bad, they're really bad, and when they're good, they're magical: We simply need to pick the right problems! --- ## Autonomous Coding Agents: Fire and Forget **URL:** https://cto4.ai/p/autonomous-coding-agents-fire-and-forget **Date:** 2025-11-11 **Type:** essay **Description:** Autonomous coding agents let you fire off coding tasks and walk away. Safe YOLO mode changes everything. **Tags:** ai, coding-agents, developer-tools **Categories:** ai-and-technology ## Willison Causing Trouble Again This [post from Simon Willison](https://simonwillison.net/2025/Nov/6/async-code-research/) hit my radar yesterday morning and it struck a chord with me. AI coding tools are a research sweet spot for me--that is, I try very hard to stay up. I've been working with the same cloud-based agents from OpenAI and Anthropic that are the subject of Willison's post. ## What to Call These Things? Willison uses the term "asynchronous coding agents" for the category that encompasses OpenAI's Codex Cloud and Anthropic’s Claude Code On The web. "Asynchronous" didn't seem ideal to me, so I decided to check in with my friend Opus to see what he knew about the topic. Turns out that Opus 4.1 has a very solid understanding of the concept and how its being described. Opus and I prefer autonomous to asynchronous, so: "autonomous coding agents." ## Background: The Coding Wars Anthropic has invested heavily in the AI coding space. Its langauge models surprised many of us by jumping ahead of OpenAI in this space when Claude Sonnet 3.5[^2] launched in in June 2024. More surprisingly, Anthropic stayed ahead, progressing rapidly through Sonnet 3.7, 4, and most recently Sonnet 4.5 just over a month ago. I have used Anthropic coding models almost exclusively since 3.5 arrived. Earlier this year, Anthropic jumped deeper into the fray with an actual coding tool, the minimalist coding agent Claude Code, whose motto might be: > Get out of the way, coding tools, and let our great models do their thing. Not long after Claude Code arrived, it supplanted Cursor[^1] as my primary AI coding interface. So where was OpenAI, the biggest AI player of all? Since Sonnet 3.5 arrived: playing catch-up. I had a lot of respect for OpenAI models generally, for ChatGPT as a product, and for the robustness of their APIs and inference services. But coding belonged to Anthropic. Not to say that OpenAI was standing still. Their release of GPT-5 this August was huge, clearly a substantial step forward, and some were saying GPT-5 rivaled Anthropic's coding models. OpenAI then released a coding-tuned model, GPT-5 Codex, in mid-September. And OpenAI had, in a series of bizarre fits and starts, released its own minimalist coding agent, Codex CLI. But to me, this all felt like catch-up: just we-try-to-do-everything posturing from OpenAI. ## Coding Agent Autonomy Changes the Game What drove me to reconsider Codex was an ongoing stream of podcast and YouTube interviews where some person in OpenAI leadership (including president [Greg Brockman](two-from-greg-brockman)!) would be interviewed about some topic other, but the conversation would drift over to Codex along the way, and we'd hear how OpenAI was dogfooding Codex internally to build OpenAI's own products. Here's an example from OpenAI DevDay on October 7th, where [Sherwin Wu](https://www.linkedin.com/in/sherwinwu1/) and [Christina Huang](https://www.linkedin.com/in/christinaahuang/) of the OpenAI Platform Team get onto the subject of Codex: What jumped out at me was that OpenAI's core internal teams were using Codex heavily, in **autonomous agent mode**; fire-and-forget. They told stories about going into meetings, and being asked to "give me a couple minutes here ..." so people could fire off a few [Codex Cloud](https://developers.openai.com/codex/cloud) cruise missles that would grind away during the meeting, and have results waiting for them afterwards. Experiencing a series of legit, unstaged anecdotes like this---I'm pretty good at smelling "marketing message" and these definitely weren't---my attitude about Codex went from "meh" to "NEED TO INSTALL ASAP." Before I got to Codex Cloud, though, Anthropic jumped the line and launched their ACA, ["Claude Code On The Web"](https://support.claude.com/en/articles/12618689-claude-code-on-the-web)[^3]. Since I was in deep with Claude Code already, getting its web counterpart set up was low effort, and [CCOTW](https://simonwillison.net/2025/Oct/20/claude-code-for-web/) ended up as my first Autonomous Coding Agent. Codex Cloud followed soon after. ## Fire-and-forget That Works In OG Claude Code, you can _already_ fire-and-forget. It's called [YOLO]() mode. You turn off all protections, live dangerously, and let Claude do whatever it chooses to, which in a command line or terminal interface, is pretty much anything. Nuke the system drive? Sure thing. Wipe the GitHub repo? Yep. Shut down your AWS cloud? No problem. Claude Sonnet 4.5 is an ethical AI-person, better than most, but YOLO mode is dangerous and developers avoid it most of the time. We mostly do our core work in non-YOLO mode, which involves carefully watching what the agent is doing, and many many approvals of the agent's polite requests, "can I use this potentially dangerous tool, please?" I get a lot of of work done when I pair with Claude Code, but I'm close to 100% occupied keeping tabs on the model while I'm pairing. What OpenAI and Anthropic really did with ACAs is deliver a reasonably safe YOLO mode, by equiping their agents with reasonably secure environments that the agent can go to town in, without risking much. "Go to town" means "run whatever you want, safely" -- usually called sandboxing -- but also "code whatever you want, safely -- you can't break anything." This works because the agent does its work in its own personal code space, kept safely separate from the current codebase and other work-in-process. Nothing there impacts our real code, until we've had a chance to carefully look at it and let it in (to be merged). ## Well-equipped Environments, Well-defined Specs Willison's piece dragged me the rest of the way into ACAs; I did the research and set up my first custom Codex Cloud environment just today. Until now, I've been living with Codex's default setup with no customization and conservative default protections. The actual focus of Willison's post is using autonomous coding agents for _coding research projects_, and he walks us through several interesting real-life examples. I like his suggestion to use a separate Git repo for these projects---one more layer of "don't worry, you can't break anything." The human role in this autonomous new world? Clear, well-thought-out specs that precisely communicate what the agent should build for you. There's nothing preventing you from having 10 or 20 agents working away---except 10 or 20 good, clear specs. --- [^1]: Well, I didn't abandon Cursor exactly. My current insane, wonderful dev setup looks like this: container app: Cursor; left pane: Codex CLI; center pane: source files; right pane: Claude Code [^2]: I do know that Anthropic called it Claude 3.5 Sonnet. In this case, I choose "avoid confusion and awkwardness." [^3]: It's hilarious how bad both OpenAI and Anthropic are with product names --- ## When Claude Code Skates on Thin Ice **URL:** https://cto4.ai/p/when-claude-code-skates-on-thin-ice **Date:** 2025-11-10 **Type:** brief **Description:** I love Claude Code and Sonnet 4.5, but when training data is thin, the reasoning failures are hilarious **Tags:** claude-code, ai, developer-tools **Categories:** ai-and-technology ## Frogs and Scientists There's a very old joke about scientists and frogs ... ## Vision Model I've been on a deep-dive working with vision models over the past few weeks; specifically [Qwen3 VL](https://github.com/QwenLM/Qwen3-VL), which just launched the last week of October. We're running inference on the model via [OpenRouter](https://openrouter.ai) providers. [Claude Code](https://www.anthropic.com/claude-code/) has been my go-to AI Coding tool for this project. Two consequences of working with an extremely new model like Qwen3 VL: - The coding model ([Sonnet 4.5](https://www.anthropic.com/news/claude-sonnet-4-5) for me) will have literally no training data to help ground its reasoning - There will be lots of complex problems to work through ## "Frog with no legs is deaf" The old frog joke came to mind multiple times over the weekend, when I had asked Claude to assist in diagnosing a problem. It would step through a valid chain of reasoning, and then at the very end, out would pop a ridiculous conclusion, like the joke's "frog with no legs is deaf." The takeaway: When your model skating on thin ice---little or no context to ground its reasoning---be suspicious of its conclusions. --- ## Pydantic AI Reaches v1 **URL:** https://cto4.ai/p/pydantic-ai-reaches-v1 **Date:** 2025-09-14 **Type:** essay **Description:** Which, for me, begged the question, what exactly is Pydantic AI? **Tags:** pydantic, ai, python, machine-learning **Categories:** AI, Technology As I wrote recently in [ChatGPT 5 Thinking Is So Damn Smart](/p/chatgpt-5-thinking): > I spent some time at the [Pydantic](https://pydantic.dev/) booth at the [AI Engineer World's Fair](https://www.ai.engineer/) in SF this June. Today, I got their announcement that [Pydantic AI v1](https://ai.pydantic.dev/) is out, and decided to write about it. The point of that post was how superbly GPT-5 Thinking filled in my knowledge gaps around Pydantic AI. As I sat down to write _this_ post, however, I realized that ChatGPT's briefing document should actually **BE** the core of this post. That document was a joint effort---I conceived the need, I asked the questions, I played critic and editor through dozens of iterations---GPT5 Thinking did the research, drafting and rewriting. I write as a learning exercise, but spending hours or days to improve upon the [cto4ai's Field Guide to Pydantic AI](#cto4ais-field-guide-to-pydantic-ai) below would have impeded rather than furthered my learning. Ethan Mollick recently shared his experience working with [Claude's new Excel-spreadsheet-editing features](https://www.linkedin.com/posts/emollick_claudes-surprise-new-ability-to-work-with-activity-7371375383125889024-3C9Y?utm_source=share&utm_medium=member_desktop&rcm=ACoAAABcNZsBSiPCgo2Gr9o5anU48UXl131Q-_M): Just as Claude had produced a very solid solution that would have taken Mollick's team of MBA students a week to reproduce, so too had ChatGPT 5 Thinking produced a very solid Field Guide to Pydantic AI. So I'll just post this Guide and move on to the next learning exercise in my very long queue, having learned plenty my the role as its editor. ## cto4ai's Field Guide to Pydantic AI This Field Guide covers the technical aspects and strategic considerations of Pydantic AI, for CTOs, engineering leaders, and company leadership. --- ## ChatGPT 5 Thinking Is So Damn Smart **URL:** https://cto4.ai/p/chatgpt-5-thinking **Date:** 2025-09-09 **Type:** brief **Description:** I needed to round out my knowledge of Pydantic AI. ChatGPT 5 Thinking absolutely nailed it. **Tags:** ai, gpt-5, reasoning **Categories:** AI, Technology I spent some time at the [Pydantic](https://pydantic.dev/) booth at the [AI Engineer World's Fair](https://www.ai.engineer/) in SF this June. Today, I got their announcement that [Pydantic AI v1](https://ai.pydantic.dev/) is out, and decided to write about it. I had Grand Canyon sized gaps in my knowledge about Pydantic AI, though, and decided to ask [ChatGPT 5 Thinking](https://openai.com/index/introducing-gpt-5/) to help me get up to speed. **NAILED IT.** Here's the chat transcript. Two notes: 1. The idea of having a component to render chat transcripts inside blog posts was stolen from [Orta Therox](https://blog.puzzmo.com/posts/2025/07/30/six-weeks-of-claude-code/). I vibe coded mine in Claude Code. 2. ChatGPT 5 Thinking believed it was writing the blog post itself. It was not; I'll be doing that by hand, thank you very much. --- ## Chroma: RAG is Dead; Long Live Context Engineering **URL:** https://cto4.ai/p/chroma-context-engineering **Date:** 2025-09-07 **Type:** episode **Description:** Jeff Huber explores how Chroma is changing the conversation away from RAG to context engineering **Tags:** ai, rag, chroma, context-engineering, vector-databases **Categories:** ai-and-technology If RAG (Retrieval Augmented Generation) is dead, then what's alive? What's the right way of thinking about and exposing company knowledge through AIs? This episode of the [Latent Space](https://www.latent.space) podcast, featuring Chroma CEO and cofounder Jeff Huber, has compelling answers. Highly recommended listen (or watch [here](https://www.youtube.com/watch?v=pIbIZ_Bxl_g)). ## What & Who is Chroma? [Chroma](https://github.com/chroma-core/chroma) is an open-source embedding database purpose-built for LLM applications. The company was founded in 2022 by [Jeff Huber](https://twitter.com/jeffreyhuber) (CEO) and [Anton Troynikov](https://twitter.com/atroyn). ### Core Purpose Chroma lets developers store embeddings alongside metadata and associated text, then run fast similarity search with filtering to ground LLM outputs in relevant, verifiable context—reducing hallucinations and improving answer quality. ### Core Features - Vector similarity search over collections - Metadata filtering for precise retrieval - Persistent storage for vectors, metadata, and associated text - Official client libraries: Python and JavaScript/TypeScript (first‑party); additional clients for Ruby, Java, Go, C#, Elixir, and Rust - Embedding-agnostic: works with text and other modalities via embeddings ### Extended Features and Commercial Offerings Beyond the core open-source capabilities, Chroma offers Chroma Distributed--—an open-source, distributed version built in Rust that enables multi-node deployments for scalability and high-throughput requirements. For teams seeking a fully managed solution, Chroma Cloud provides a serverless, distributed architecture that abstracts infrastructure complexity while supporting deployment across major cloud platforms (AWS, GCP, Azure). ## Snip: RAG => Context Engineering Huber detests the term "RAG" and his opinions on the topic helped convince [Swyx](https://x.com/swyx) to remove the RAG track from the [AI Engineer World's Fair](https://www.ai.engineer) event in June. He describes in [this snip](https://share.snipd.com/snip/3a87c36a-e18e-4081-bfd6-84f563afab93) what I believe is a far better framing of the core issue and solution---context engineering. ## Snip: Context Rot Model builders like OpenAI, Anthropic, and [Google DeepMind](https://deepmind.google) keep announcing increasingly large context windows, to 1M tokens and beyond. And with contexts getting larger and larger, who needs a retrieval solution like Chroma? But there's a dirty little secret known mainly among researchers---context rot. When a large context gets operated on repeatedly, for example in multi-turn agent interactions, the context "rots," and the model effectively loses its mind, most seriously the failure to follow clearly-stated instructions. Chroma researched this problem carefully, and in [this snip](https://share.snipd.com/snip/8fa58784-da31-41d4-ba9d-f0891d8a9283), Huber describes what they discovered. ## Snip: Context Chunk Selection and LLM Re-ranking Huber and Chroma definitely have their finger on the pulse of "latest context engineering tips and tricks" ... In [this snip](https://share.snipd.com/snip/f1faab7a-3205-4a1f-ab6c-f10be993b359) he shares emerging practices around getting from 1M or 100K down to 30 using LLM brute-forcing. ## Snip: Code Embeddings There's an ongoing debate around whether, in the AI coding world, does using embeddings via a vector database like Chroma make sense, versus simpler, very-widely-used approaches like plain old grepping. [Here](https://share.snipd.com/snip/f62bdafa-407e-41be-8120-e977cf43bf43) Huber addresses this question; the answer, in short, is yes, if you want to get the full 100% rather than settling for 80%. ## Snip: Evals and Golden Datasets If you want your AI-driven knowledge access application to start good and stay good, constant evaluation of whether queries are returning the right answers is table stakes. And accurate evals require "golden datasets" with human-curated sets of "this query should serve up these documents." In [this snip](https://share.snipd.com/snip/80c9e0ed-10c7-43ba-b7f6-97d2d2deaa0d), Huber talks through just how non-painful the process of creating a golden dataset can be. --- ## Great Rules In, Great Results Out **URL:** https://cto4.ai/p/great-rules-great-results **Date:** 2025-09-01 **Type:** brief **Description:** An investment in writing great rules for your AI tools will be repaid many times over. **Tags:** cursor, copywriting, ai, productivity, content-creation **Categories:** AI, Writing I've [written recently](/p/auto-link-with-gpt-5-and-playwright-mcp) about how [Cursor](https://cursor.com), the AI IDE, isn't just great for code---it's a superb copywriting tool as well. As part of my [recent implementation](/p/hugo-to-astro-migration-story) of the [Astro](https://astro.build) static site generator, I wanted to improve the instructions I give Cursor when I start writing a new post. What I need is a scaffolding consisting of a directory for the post, a skeleton content Markdown file, and subdirectories for related items: images, social media posts, and AI chat transcripts (in case I want to highlight an AI conversation). This process changed a bit when I migrated from [Hugo](https://gohugo.io) to Astro and I so I updated and refined my related Cursor rule. All I had to do was give [Cursor](https://cursor.com) the rule as @ context, and then make a very simple request. Here's the full transcript of my interaction with Cursor: Note that the interaction consisted of a single, simple request, and Cursor doing all the work. That's the way, uh-huh uh-huh, I like it, uh-huh uh-huh ... But how did Cursor know what to do? That `astro-mdx-content-authoring` rules file is 287 lines long, and carefully explains everything about authoring content files within my new Astro environment, including this scaffolding process. Creating that rules file took some work---though these days I have [Claude Code](https://www.anthropic.com/claude-code) write most of my Cursor rules. Still, I there was work involved in thinking through what was needed and in continuing to refine the rule as my Astro setup evolved and new ideas came to mind. But the payback is high: a laborious process reduced to a single, simple request. Invest in your AI rules, and thou shalt be repaid. --- ## Claude Code Deep Dive: Hugo to Astro+Beehiiv in 9 Days **URL:** https://cto4.ai/p/hugo-to-astro-migration-story **Date:** 2025-08-27 **Type:** essay **Description:** Migrating from Hugo to Astro/Beehiiv with Claude Code doing ~87% of the work **Tags:** ai, claude-code, astro, hugo, migration, web-development This wasn't my first Claude Code deep dive, but it was definitely the deepest---something that became obvious as I upgraded from pay-per-token Anthropic API use to Claude Max $100 and finally to Max $200 after seeing $10 Anthropic credit reload emails rolling in two, three, or even four times a day during the early stage of the project. ## Why? The project really wasn't about [Hugo](https://gohugo.io) => [Astro](https://astro.build) at all. Rather, it was about splitting out my fairly substantial, mostly-AI-themed blog from its home as a branch of my corporate site (craftycto.com/blog), to an all-new domain (cto4.ai), while at the same time setting up full-strength email newsletter capabilities. Why leave Hugo, which has served me well since 2022? The Hugo project made a breaking change sometime in 2024, during a multi-month lull in my writing; when I came back to restart publishing, I figured I'd upgrade to latest Hugo version, and kaboom---the build was broken. Even with LLM help and quite a few hours of research, no joy; so I downgraded to the last release before the breaking change and got things running. But I definitely lost some of my love for Hugo in the process. Achieving an all new look on the new standalone blog would mean choosing a new Hugo theme, and then migrating all my content, plus my theme-specific customizations in the form of Hugo shortcodes and templates written with Golang templating, to the new theme. On top of the unresolved breaking change, this meant I faced a rather unpleasant chunk of Hugo work. I had been keeping tabs on other static site generators, with Astro getting a lot of buzz as best/most modern, and I realized it might not be that much more work to build the new site on Astro ... ## AI Coding Courage As [Simon Willison](https://simonwillison.net) and others have pointed out, having a great language model backing you up takes a lot of the fear out of tackling a challenging project, he said that when his AI coding tools were maybe 1/10th as powerful as they are today. There's no way I would have taken on this project at all, let alone switched platforms in the process, without an AI as co-conspirator. The results justified my confidence that the AI would help, a lot ... Claude Code was responsible for ~86.7% of commits during the project. The numbers we were able to gather, mainly via git commit logs, indicated that Claude's percentage of the lines of code+content was lower, at 63%. My guess is that, of the real code written (versus content work), Claude did much closer to 100% of the work. I didn't vibe code the project, in that I was paying close attention to what Claude Code was up to, often redirecting it, and sometimes initiating major refactoring. But I didn't actually write much code myself. Why should I, when I could turn Claude loose and its code is almost certainly have been better than what I would have written, especially with minimal knowledge of Astro and limited TypeScript chops. Architecturally, I gave a lot of help and redirection, but at the code level, very little. ## Beyond Coding: Research and Architectural Discussions Probably 25% of my interactions with Claude involved sending it off to do some research for me---often along the lines of "What's the Astro best practice to do X? Research it! Web search!" Claude's really good at that (I assume Opus 4.1 is getting called on a lot), and when it had results, we'd talk through them and I'd provide additional context or poke holes in bad ideas. Every single one of these situations would have been painful without my AI colleague: it would have been Jack doing painstaking research, followed by Jack agonizing and second-guessing himself. It is so much more effective, and so much more enjoyable, to work alongside a well-read, very smart AI pair. ## Content Migration Grunt Work Claude is tireless, and that's pretty great on not-fun tasks like migrating Hugo-flavored Markdown to Astro's Markdown + JSX (.mdx) format. With the quantity of content I was migrating, about 100 posts or so, Migrating content often found Claude on the boundary of "should I write a Python script for this?" versus "I'll just do this myself," which was an interesting model-decision-making process to see happen. Claude was also often at the boundary of being able to hold to all the threads in its mind (context)---indeed, one of my main jobs was guiding Claude back onto the path when it lost its way. The dreaded "0% context left until auto-compact" always seems to come at a bad time, and Claude Code is especially prone to losing its way post-compact. Not a perfect tool, yet. ## Setting Up the Infrastructure Another normally-excruciating task on a project like this is setting up complex parts of the infrastructure, along with diagnosing related glitches. In my case, this involved Cloudflare DNS, Pages, Functions, and Scrape Shield configs. Each one of those might have mean hours of digging, left to my own devices, but I rarely got sidetracked for more that a few minutes with the model's help. [Beehiiv](https://beehiiv.com) and email-specific DNS settings and DMARC config were another cluster of potential pain, avoided with Claude's assistance. Anyway: you get the idea. This was a **meaty** piece of work, and I was doing it my spare hours around my fractional CTO day job. But 9 days after the repo came into being, cto4.ai launched, not just blog- but also newsletter-ready. ## Claude's Also Good At Reports I had Claude sift through various data sources, most importantly git logs, but also research we had done together earlier, such as comparing Astro vs Hugo site performance over a representative set of pages. (I always have Claude Code create a Markdown file in the /docs directory for any substantial research it does for me.) And then I had it build the report below, complete with statistics, analyses, executive summaries---the whole works. Aside from the report below, this entire piece is written by Jack. But I will quote one phrase from the rough draft that Claude wrote for me, unasked: > What started as a typical web migration quickly evolved into something far more interesting—a case study in effective human-AI collaboration. Cheers!

📊 CraftyCTO.com/blog to cto4.AI Migration Report

📋 Executive Summary ▶
This report documents the migration of the CraftyCTO blog from a Hugo-based corporate site to a standalone Astro-powered blog at cto4.ai. The migration began in August 2025 and represents a strategic separation of blog content from corporate presence. The migration was accomplished through an AI-assisted development approach, with Claude Code handling 86.7% of commits and 90.6% of the actual code/content additions, while human guidance provided strategic direction through brief, iterative commits (13.3% of commits, 9.4% of additions). This partnership enabled rapid feature development and modernization in a compressed timeframe.
📅 Repository Timeline ▶
### CTO4.AI Blog Repository - **Repository Created**: August 17, 2025 - **Total Project Commits**: 173 (excluding AstroWind template history) - **cto4.ai Live**: August 25, 2025 ### CraftyCTO Repository - **Created**: April 14, 2022 - **First Commit**: "Set up new blog" by Jack Ivers - **Migration Planning Started**: August 14, 2025
🔧 Technology Stack Comparison ▶
| Aspect | CraftyCTO (Original) | CTO4.AI (New) | | ------------------ | -------------------------------- | --------------------------------------------------------------------- | | **Framework** | Hugo | Astro v5.13 | | **Content Format** | Markdown | MDX with Content Collections | | **Styling** | Tailwind and Custom CSS | Tailwind CSS v3.4 | | **Search** | [Pagefind](https://pagefind.app) | Pagefind | | **Build Tool** | Hugo | [Vite](https://vitejs.dev) | | **Theme** | Custom | [AstroWind](https://github.com/onwidget/astrowind) (heavily modified) | | **Deployment** | Cloudflare Pages | Cloudflare Pages |
📈 Migration Phases ▶
### Phase 1: Planning & Setup (August 14-16, 2025) Key commits from CraftyCTO repo: - 839e57da - Add Claude Code documentation and multi-repo migration plan - 01df3f77 - Update migration docs with cto4.ai domain and AstroWind theme - 595f99f5 - Add critical migration directive to preserve Hugo blog operation - 75fe949f - Document final logo selection and usage guidelines ### Phase 2: Content Migration (August 19-21, 2025) Notable activity: - Multiple posts migrated with scaffolding for social media content - Image asset reorganization from Hugo's structure to Astro's - Creation of unified content structure (`/src/data/content/[slug]/`) ### Phase 3: Feature Implementation (August 22-25, 2025) Key developments: - Newsletter integration with Beehiiv platform - [Attio](https://attio.com) CRM integration for subscriber management - Search functionality with Pagefind - Social media Open Graph optimization - Cloudflare Functions for API endpoints ### Phase 4: Refinement (August 26, 2025) Recent improvements: - Text sizing optimization for large screens (Claude Code assisted) - Newsletter modal timing adjustments - Cursor rules documentation updates - 301 redirects from craftycto.com/blog (already implemented)
🤖 Claude Code Contributions ▶
### Authorship Analysis Based on commit message patterns for YOUR actual migration work (excluding AstroWind template): - **Total Your Migration Commits**: 173 - **Claude Code Authored**: 150 commits (86.7%) - **Human Authored**: 23 commits (13.3%) ### Identifying Authorship Patterns #### Jack's Commit Style (Human) - Message length: 1-20 characters - Examples: "y", "reorg", "rules", "ng wip", "linking", "ng live", "footer fixes" - Pattern: Minimal descriptions, often single words or brief phrases #### Claude Code's Commit Style - Message length: 30+ characters - Examples: - "Fix newsletter modal form submission" - "Add Attio CRM integration for newsletter subscribers" - "Implement redirect from /blog to /archive" - "Update OG image with correct tagline and branding" - Pattern: Detailed technical descriptions with proper capitalization and clear explanations ### Notable Claude Code Contributions - Newsletter integration with Beehiiv - CRM integration with Attio - Image loading fixes for production - TypeScript error resolutions - Search indexing improvements - Social media optimization - Modal and form submissions - Cloudflare Functions implementation - Text sizing optimization (only commit with explicit Claude Code attribution)
📊 Code Statistics ▶
### Overall Migration Impact - **Total Lines Added**: +15,740 (excluding generated files) - **Total Lines Deleted**: -1,528 - **Net Lines of Code/Content**: +14,212 - **Files Changed**: ~200+ ### Breakdown by Type - **MDX Content**: 5,857 lines (83 blog posts) - **Documentation**: 4,331 lines (migration docs, reports) - **Scripts/Python**: 2,353 lines (migration tools) - **Code (Astro/TS/JS)**: 2,258 lines (new application code) - **Other Files**: 941 lines ### Contribution Breakdown by Author #### Human (Jack) - 13.3% of commits - **Commits**: 23 - **Lines Added**: +1,479 (9.4% of total additions) - **Lines Deleted**: -196 - **Net Change**: +1,283 #### Claude Code - 86.7% of commits - **Commits**: 150 - **Lines Added**: +14,261 (90.6% of total additions) - **Lines Deleted**: -1,332 - **Net Change**: +12,929 ### Top 10 Largest Commits 1. `a0b8060` (+298/-1409) - Complete Hugo-to-Astro blog migration with systematic shortcode conversion 2. `84ce8ce` (+1640/-7) - Implement smart fallback system for description vs excerpt 3. `afa78d0` (+1325/-2) - cover images etc. (Human) 4. `e127bbc` (+1121/-0) - Phase 3: Migration script with test migration complete 5. `df2742e` (+630/-289) - Implement clean newsletter-first homepage with Substack-style modal 6. `9cad969` (+737/-176) - wip services (Human) 7. `e3d1597` (+908/-1) - ng live (Human) 8. `f1793f5` (+883/-0) - Add performance comparison scripts and report 9. `1a2a800` (+756/-82) - Update Cursor rules for Astro blog migration 10. `9e4e679` (+832/-1) - Add Attio CRM integration for newsletter subscribers ### Key Insights - Claude Code handled 86.7% of commits and 90.6% of line additions - Human commits (13.3%) contributed only 9.4% of lines, showing more directive than implementation work - The largest single change was refactoring/cleanup (-1409/+298), showing significant optimization - Major human contributions included content migration ("cover images", "ng live", "wip services")
🎯 Key Migration Decisions ▶
### Content Structure - Moved from Hugo's `/content/posts/` to Astro's `/src/data/content/[slug]/` - Unified content types (essay, brief, elsewhere, quote, episodes) in single directory - Images maintained in parallel structure with content in `/src/assets/images/content/[slug]/` ### Technical Improvements 1. **Performance**: Astro's partial hydration vs Hugo's static generation 2. **Developer Experience**: MDX support with component imports 3. **Search**: Continued use of Pagefind (same as CraftyCTO) 4. **Typography**: Consistent `prose-lg` sizing vs responsive scaling ### Feature Additions - Newsletter modal with Beehiiv integration - CRM integration with Attio - Enhanced social media sharing - Improved dark mode support - View Transitions API for smooth navigation
✅ Current Status ▶
### Completed - ✅ Core blog functionality - ✅ Content migration structure - ✅ Full content migration from Hugo - ✅ Newsletter integration - ✅ Search functionality - ✅ Social media optimization - ✅ Responsive design - ✅ Dark mode - ✅ RSS feed - ✅ XML sitemap generation (sitemap-index.xml) - ✅ Cloudflare deployment - ✅ 301 redirects from craftycto.com/blog to cto4.ai ### Pending - ⏳ Analytics integration - ⏳ Comment system - ⏳ Full SEO audit
⚠️ Technical Debt & Considerations ▶
### From AstroWind Heritage The cto4ai-blog repository inherits ~950 commits from the AstroWind template project (2022-2025), providing: - Mature component library - Accessibility features - Performance optimizations - Community-tested patterns ### Migration Challenges 1. **URL Structure**: Different permalink patterns between Hugo and Astro 2. **Image Handling**: Hugo's image processing vs Astro's Image component 3. **Frontmatter**: Schema differences requiring transformation 4. **Shortcodes**: Hugo shortcodes need conversion to MDX components
⚡ Performance Metrics ▶
### Build Times - **Hugo (CraftyCTO)**: ~2-3 seconds - **Astro (CTO4.AI)**: ~10-15 seconds (with image optimization) ### PageSpeed Insights Mobile Scores | Metric | CraftyCTO (Hugo) | CTO4.AI (Astro) | Difference | | ------------------ | ---------------- | --------------- | ---------- | | **Performance** | 98 | 96 | -2 points | | **Accessibility** | 100 | 98 | -2 points | | **Best Practices** | 100 | 100 | 0 | | **SEO** | 100 | 100 | 0 | ### Core Web Vitals Comparison (Mobile) | Metric | CraftyCTO (Hugo) | CTO4.AI (Astro) | Status | | ---------------------------- | ---------------- | --------------- | ------- | | **First Contentful Paint** | 1.1s | 1.4s | Good | | **Largest Contentful Paint** | 1.1s | 2.0s | Good | | **Total Blocking Time** | 0ms | 30ms | Good | | **Cumulative Layout Shift** | 0 | 0 | Perfect | | **Speed Index** | 3.8s | 4.4s | Good | ### Performance Analysis - Both sites achieve excellent performance (96-98 scores) - Hugo has a slight edge in initial load metrics - Astro site still delivers strong user experience - Main opportunity: Optimize LCP from 2.0s to match Hugo's 1.1s
🎬 Conclusion ▶
The migration from Hugo to Astro represents a successful modernization of the blog platform, with improved developer experience, better performance, and enhanced features. The separation of blog content from corporate site allows for independent evolution and optimization of each property. This migration was primarily accomplished through Claude Code assistance (86.7% of commits, 90.6% of code additions), with human guidance providing strategic direction through brief commits (13.3% of commits, 9.4% of additions). This AI-assisted approach enabled rapid implementation of complex features including newsletter integrations, CRM connectivity, and performance optimizations that might have taken significantly longer with traditional development methods. The collaboration pattern shows an effective human-AI partnership: humans providing brief directional commits ("reorg", "ng wip", "linking") with larger structural changes, while Claude Code handled the detailed technical implementation with properly documented, descriptive commits. This approach maintained high code quality while accelerating the migration timeline. --- _Report Generated: August 26, 2025_ _Live Site: cto4.ai_
--- ## Andrew Ng on Leaders Still Doing Things the Way They Were in 2022 **URL:** https://cto4.ai/p/andrew-ng-2022 **Date:** 2025-08-26 **Type:** quote **Description:** Ng put into words something that I've been thinking for a while now. **Tags:** ai, machine-learning, andrew-ng, philosophy > To me, there's so many things that the world used to do in 2022, they just do not work in 2025. So in fact, often I ask myself, is there anything we're doing that today that we were also doing in 2022? And if so, let's take a look and see if it still even makes sense today because a lot of stuff, a lot of workflows in 2023 don't make sense today. So I think today, the technology is moving so fast > > ... > > I think it's a challenge for a lot of people. I know many great leaders in lots of different functions still doing things the way they were in 2022. And I think it's just got to change. When new technology comes, I mean, once upon a time, there was no such thing as web search today. Would you hire anyone for any role that doesn't know how to search the web, right? And I think we're well past the point that for a lot of job roles, if you can't use LLMs in an effective way, you're just much less effective than someone that can. --- [Andrew Ng](https://www.andrewng.org), No Priors Podcast [Sarah Guo](https://www.linkedin.com/in/sarahxguo) and [Elad Gil](https://www.linkedin.com/in/eladgil/)'s No Priors podcast is well-done. The episode title here was How Agentic AI is Transforming The Startup Landscape with Andrew Ng; very worth the listen. --- ## Two Interviews with OpenAI President Greg Brockman **URL:** https://cto4.ai/p/two-from-greg-brockman **Date:** 2025-08-19 **Type:** brief **Description:** Highlights from two recent interviews by the Latent Space team with Greg Brockman I had the pleasure to see [Greg Brockman](https://en.wikipedia.org/wiki/Greg_Brockman) speak in person at [AIEWF25](https://www.youtube.com/watch?v=avWhreBUYF0)---interviewed by [Swyx](https://www.swyx.io), with assistance from Nvidia's [Jensen Huang](https://en.wikipedia.org/wiki/Jensen_Huang) via prerecorded video. The title of that talk, appropriately, was "Define AI Engineer." Then just three days ago, [Swyx](https://www.swyx.io) and his [Latent.Space](https://www.latent.space) podcast partner [Alessio Fanelli](https://x.com/fanahova) scored another interview with Brockman, this time themed "OpenAI's Road to AGI" with its focus on the huge OpenAI news over the past two weeks, [GPT-OSS](https://openai.com/blog/gpt-oss) and then [GPT-5](https://openai.com/blog/gpt-5). Here are a few highlights from each. ## AIEWF25 June 3rd On how we structure codebases in the emerging AI era: > The direction is something that is like just so compelling and incredible to me. The thing that has been the most interesting to see has been when you realize that the way your structure your codebase determines how much you can get out of [Codex](https://openai.com/blog/openai-codex), right? Like all of our existing codebases are kind of matched to the strengths of humans. But if you match instead to the strength of models which are sort of very lopsided, right? Models are able to handle way more like diversity of stuff but are not able to sort of necessarily connect deep ideas as much as humans are right now. And so what you kind of want to do is make smaller modules that are well tested that have tests that can be run very quickly and then fill in the details. The model will just do that, right? And it'll run the test itself. > > The connection between these different components, kind of the architecture diagram, that's actually pretty easy to do, and then it's like filling out all the details that is often very difficult. And if you actually do that, you know, what I described also sounds a lot like good software engineering practice. But it's just like sometimes because humans are capable of holding more of this like conceptual abstraction in our head, we just don't do it – it's a lot of work to write these tests and to flesh them out and that the model's going to run these tests like a hundred times or a thousand times more than you will and so it's going to care way way more. So in some ways the direction we want to go is build our codebases for more junior developers in order to actually get the most out of these models. On domain specific agents, in some ways foreshadowing GPT-5 as router: > I think my perspective is that first of all, it's all on the table, right? Maybe we reach a world where it's just like the AIs are so capable that we all just let them write all the code. Maybe there's a world where you have like one Al in the sky. Maybe it's that you actually have a bunch of domain specific agents that require a bunch of of specific work in order to make that happen. [...] > > I think the evidence has really been shifting towards this menagerie of different models—I think that's actually really exciting. There's different inference costs, there's different trade-offs like just distillation works so well, there's actually a lot of power to be had by models that are actually able to use other models. ## Latent.Space August 16 Here I'm taking advantage of the amazing AI-driven podcast app [Snipd](https://www.snipd.com) and its quotes feature. If you click the link, you'll be taken directly to that spot in the podcast. On [routing between models in GPT-5](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > You have a reasoning model that we know is good for applications that require this intelligence, but you're okay waiting a little bit longer. We have a non-reasoning model that is great for applications where you want the answer fast. On [why online learning and RL amplify model value](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > When the models are extremely capable, the value of a token they generate is extremely high. On [compute as the primary bottleneck for progress](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > The bottleneck is always compute. If you give us a lot of compute, we will find ways to iterate that actually make the most of that compute. On [OpenAI's Dota RL scaling experience](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce) ([Dota](https://openai.com/research/openai-five)): > You come back to the office every week, they doubled the number of cores. And suddenly the agent, the true skill was going up and to the right. On [transferability of learned reasoning skills to new domains](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > Learning to solve hard math problems and write proofs turns out to actually transfer to writing program and competition problems. On [the purpose and importance of publishing a model spec](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > The model spec is an example of where we've made it very legible to the outside world what our intention is for this model to do. On [pricing, cost curves, and demand elasticity](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > If you just make it more accessible and available to people, they will use way more of it. On [how to structure codebases for AI integration](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > You really build codebases around the strengths and weaknesses of these models. More self-contained units have very good unit tests that run super quickly and that have good documentation. On [structuring workloads and using multiple model instances](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > You want to be a manager of not an agent, but of agents. And so that you need to, first of all, think about how your code base is structured. On [the desired form factor for coding agents](https://share.snipd.com/episode/8dd5816c-f788-446e-9a8b-679b141718ce): > You want the pair form factor. You also want the remote async form factor. And you want it to be one entity that has knowledge and memory across all of this. I hope Brockman is as wise about navigating the political and moral minefields we face today, as he clearly is about technology. --- ## Use GPT-5 with Playwright MCP to Auto-link Posts **URL:** https://cto4.ai/p/auto-link-with-gpt-5-and-playwright-mcp **Date:** 2025-08-11 **Type:** brief **Description:** Quick Cursor workflow using GPT-5 plus Playwright MCP to add named-entity links in blog posts. About a month ago, [Jason Liu](https://www.linkedin.com/in/jxnlco/) wrote on [LinkedIn](https://www.linkedin.com/posts/jxnlco_copywriting-with-cursor-activity-7341510257602347009-r7x8?utm_source=share&utm_medium=member_desktop&rcm=ACoAAABcNZsBSiPCgo2Gr9o5anU48UXl131Q-_M): > the cursor for writing is just cursor, > > my copy writer uses @cursor_ai for most of the writing now. I strongly identified---the key here is that [Markdown](https://daringfireball.net/projects/markdown/) is an outstanding content authoring language, and [Cursor](https://www.cursor.com), as an AI-enabled editor, is quite smart about applying AI to Markdown. I leveraged Cursor's AI capabilities mostly for repetitive tasks like scaffolding social media directory structures, transforming my hand-written posts into smaller, social-friendly versions for LinkedIn and Mastodon, and creating tables such as those in my [AI Engineer World's Fair writeups](/tag/aiewf2025).[^maven] [^maven]: Check out the [Maven video](https://maven.com/p/11e973/copywriting-with-cursor) Jason and his copywriter [Taylor Bell](https://www.linkedin.com/in/taylor-bell-0644b131/) did on copywriting with AI---really advanced on both the tech and science-of-great-copywriting fronts. Over the past few days, I've been exploring what OpenAI's [GPT-5](https://openai.com/index/introducing-gpt-5/) model(s) are capable of in Cursor. I've also been seeking real-life, high-value [MCP](https://modelcontextprotocol.io/) use cases. I stumbled into an awesome little scenario that gave me a chance to apply both GPT-5 with MCP together and shave hours off my post-preparation time. Full disclosure: I haven't yet tried this workflow with other models like my go-to [Sonnet 4](https://www.anthropic.com/claude/sonnet), and it's possible the same workflow may work there as well. That said, I suspect GPT-5's way of approaching problems might have an edge in this case. See what you think---this post is an example, 90% of the links in this post were GPT-5 created and these were all correct. ### Problem: Auto Link Generation for "Named Entities" My posting workflow is: write the post, then polish it---often with images and links. Links are one of those tasks that just take time: first, select which items (the ML term ["named entities"](https://en.wikipedia.org/wiki/Named-entity_recognition) is exactly right here) need links, then web searching to find the right authoritative source to link to, then inserting the Markdown link syntax. It's not too hard, but after the hard mental effort of writing the post itself, I'm rarely in the mood for another 30 minutes of researching and link-crafting. Before GPT-5 and tool use, I've had the AI auto-generate links, but hallucination was often a serious problem. The links would get generated, but half of them would be wrong. Even this broken workflow saved time, but it seemed like the AI should be able to do better. ### Solution: Cursor + GPT-5 + [Playwright](https://playwright.dev/) MCP + Reusable Prompt As I started to understand how GPT-5 approached problems, I decided to ask it to auto-generate links. It did the best job of any model I had tried so far, but there were still problems with bad links, as on its own, GPT-5 could web search, but it couldn't examine the results to make sure they were good and appropriate. So I had a little side chat with GPT-5 via [ChatGPT](https://openai.com/chatgpt/overview/) about these, where we quickly concluded that a tool like Playwright, made available to GPT-5 via MCP and Cursor's very solid MCP support, would enable GPT-5 to check its work. After much MCP plumbing work (early days for to me still) and many iterations of prompts to GPT-5, I got it working! Now 80-90% of my post links are AI-generated and accurate, and my posts end up fully resource-linked, as opposed to slightly or not at all. This was a valuable prompt, worthy of being captured and made reusable. So I had GPT-5 craft that prompt based on the conversations we'd been having, and propose a solution for saving it for reuse. Here's the current iteration of that prompt---note that this definitely suboptimal and needs work: ``` ## Named Entity Linking with Playwright ToDos: - link to other content in the blog is fine, look for self-references such as "my AI Engineer World's Fair writeups" - people links: if linkedin, need to make sure this is the right person **Task**: Use Playwright to add markdown links for all important non-obvious named entities in a blog post, avoiding obvious/common terms. **Instructions**: 1. **Identify Named Entities**: Scan the post content for specific products, companies, technologies, platforms, and services that readers might want to learn more about. Important: only link the first appearance of a term. 2. **Avoid Obvious Terms**: Do NOT link common/obvious terms like: - Operating systems (macOS, Windows, Linux) - Generic platforms (GitHub, Twitter, YouTube) - Common file formats or protocols - Basic technical terms 3. **Link Strategy**: Link to **informational pages**, not direct tool access: - Product announcement pages - Overview/feature pages - Company about pages - Documentation pages - Avoid login/signup pages 4. **Playwright Testing Process**: - Navigate to each potential link URL - If you detect a Cloudflare interstitial (e.g., "Just a moment...", element id `cf-please-wait`, `[data-translate="checking_browser"]`, or iframe URL containing "challenge"): - Wait up to 30s, checking every 1-2s - If still present, reload once and wait again up to 15s - If still present, mark as "UNRESOLVED (CF gate)" and continue - Verify the page loads and contains relevant content - Only use links that are accessible 5. **Link Format**: Use standard markdown format: `[Entity Name](URL)` 6. **Content Preservation**: - ONLY modify links - Do NOT change any other content - Ignore linter errors (they aren't accurate for markdown) **Example Entities to Link**: - AI models (GPT-5, Claude, Sonnet 4, Opus 4.1) - AI platforms (ChatGPT, Cursor, Claude Artifacts) - Specific technologies (MCP, Claude Artifacts, ChatGPT Canvas) - Specialized services (Claude Team) - Companies that aren't extremely well known (Dagger, MotherDuck, MongoDB) - Product names that aren't extremely well known **Example Entities to Avoid**: - Extremely well known companies (Apple, Google, OpenAI, Anthropic, Microsoft) - Extremely well known product names (macOS, GitHub, Windows, Linux, iPhone) - Basic web platforms - Common file formats - Generic technical terms **Output**: Update the markdown file with appropriate links while preserving all other content exactly as-is. ``` --- ## Early GPT-5 Impressions **URL:** https://cto4.ai/p/early-gpt-5-impressions **Date:** 2025-08-09 **Type:** brief **Description:** My vibe assessment: different, powerful, bumpy launch I had a chance to get early experience with the new [GPT-5](https://openai.com/index/introducing-gpt-5/) model(s) yesterday and this morning: - In two flavors of [ChatGPT](https://openai.com/chatgpt/overview/): browser interface and macOS native app, including code generation inside a ChatGPT [canvas](https://openai.com/index/introducing-canvas/) - In [Cursor](https://www.cursor.com) as its coding model I had a juicy challenge for ChatGPT with the GPT-5 model: at TSIP, we hit a problem with a mathematical formula used in our advanced soil sample design process, and had to rework the formula. I had been working in [Claude](https://www.anthropic.com/claude), using [Sonnet 4](https://www.anthropic.com/claude/sonnet), then upgrading to [Anthropic](https://www.anthropic.com/)'s smartest model, [Opus 4.1](https://www.anthropic.com/claude/opus), when the going got tough, creating interactive models as [Claude Artifacts](https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them). Then I ran out of tokens in my [Claude Team](https://www.anthropic.com/team) account, and rather than waiting, picked up the work in process and jumped over to ChatGPT, where I was happy to see the GPT-5 model available, some of the time ... Part of the bumpy launch was the GPT-5 rollout. My experience was that GPT-5 appeared in my [Team accounts](https://openai.com/chatgpt/team/) first, and my personal [Plus account](https://openai.com/chatgpt/pricing/) later. Over in ChatGPT, GPT-5 was able to ingest Claude Opus's Artifact code, along with the chat transcript for context, and its first interactive canvas was already better than what Opus had produced in a couple of iterations---though GPT-5 had the benefit of Opus's work. We thought we had the formula figured out, but then ran into problems when we tested the full range of possible inputs. GPT-5 impressed with highly creative---and ultimately successful---suggested solutions. The bumpy rollout was evident in other ways during this work---at several points, GPT-5 wiped its canvas code, which fortunately I had saved off, and also lost its ability to directly edit code in the canvas. Once the formula was solved, I needed to document it and capture the proof of concept canvas code in our GitHub repo, so I jumped over to Cursor, apparently a GPT-5 launch partner as they had it integrated as soon as [OpenAI](https://openai.com/about/) unleashed GPT-5 on the world. Here, GPT-5's "different" nature became apparent; chatting with GPT-5-as-coding-model, to me, has a clearly different tone and cadence than interacting with other models. GPT-5 is, among other things, a multi-model router, and you could see it considering: "Hmmm ... how tough a question is this? Pretty tough, it seems, let me engage my hard thinking mode (model)" ... this approach worked well on all of the tasks I've sent its way so far, both in Cursor and in ChatGPT when creating the interactive canvas proof of concept. All in all: OpenAI is definitely moving the state the art forward here with GPT-5, and also giving the competitive landscape a good kick in the ass with aggressively good pricing. Welcome, GPT-5! --- ## Running OpenAI's gpt-oss on Mac Mini with MLX **URL:** https://cto4.ai/p/gpt-oss-mlx-mac-mini **Date:** 2025-08-06 **Type:** brief **Description:** Thanks to LM Studio and the community, it's insanely easy. And gpt-oss is insanely good. OpenAI dropped their open source, [Apache-licensed](https://www.apache.org/licenses/LICENSE-2.0) ["gpt-oss" models](https://openai.com/open-models/) yesterday, which obviously is getting a lot of attention. I've experimented with local models over the years, mainly to assess "how smart is it?" and "how badly does it eat up local resources?" Generally they weren't really something I'd use in anger when I had access to much more powerful OpenAI and [Anthropic](https://www.anthropic.com/) models. I hadn't done a local LLM install for quite a while though, not since I got my M4 Mac Mini with 64GB of unified CPU/GPU memory ... and gpt-oss is a full reasoning model, like OpenAI's [o3](https://openai.com/index/openai-o3), so it seemed like the time was right to do another local model eval. Several wows here: - Wow---it is so easy to get local models running these days. [LM Studio](https://lmstudio.ai/) is amazing. - Wow---the "local models" community is incredible. Within 24 hours, someone had a fully-optimized-for-Mac version of gpt-oss (using Apple's [MLX](https://github.com/ml-explore/mlx) framework) was ready and waiting for download in LM Studio. - Wow---gpt-oss is a for-real powerful model, not a toy like I'd experimented with a year or two back. I had a few research questions floating around that I hadn't had a chance to ask o3 about yet, so just for grins I decided to throw them at gpt-oss. I got a strong, useful, reasoning-grade responses, such as this complex topic relating to the [Hugo](https://gohugo.io/) static site generator: Verdict: good enough to actually use. --- ## Orta Therox: 6 Weeks of Claude Code **URL:** https://cto4.ai/p/orta-claude-code-six-weeks **Date:** 2025-08-02 **Type:** elsewhere **Description:** Orta Therox reflects on six weeks using Claude Code for programming, describing it as transformative for maintenance tasks and side projects [Orta Therox](https://www.linkedin.com/in/ortatherox/)'s [retrospective on six weeks with Claude Code](https://blog.puzzmo.com/posts/2025/07/30/six-weeks-of-claude-code/) is one of the most compelling accounts I've read of how AI coding assistants are transforming development workflows. His experience mirrors my own observations about the technology's impact on programming productivity.[^1] [^1]: I use Cursor to help me author pieces in this blog, Crafty's Illustrated. Cursor generally does the grunt work of creating content directories and markdown stubs. I almost never let the LLM write the actual content. This post is an exception---the LLM wrote and I did post-edits---and I think whatever model Cursor selected did a nice job on this. The most striking aspect is his description of how maintenance tasks that would traditionally require weeks of dedicated effort can now be accomplished as "side projects" while working on other things: > Converting hundreds of React Native components to just React > Replaced 3 non-trivial RedwoodJS systems with home-grown or mature, supported replacements > Built complex REPLs for multiple internal and external projects > Switched almost every db model to have a consistent 'flags' system for booleans > Converted from Jest to Vitest > Created our front-end testing strategies for React This list represents what would typically be years of technical debt backlog, completed in just over a month and a half. The key insight is that these tasks "do not need to be treated as debt" when you have the right tools---they become manageable background work. Orta's comparison to the "introduction of photography" period is particularly apt. Just as photography didn't eliminate painting but changed its role and accessibility, AI coding assistants are fundamentally altering what it means to be a programmer. The ability to "instantly create a whole scene instead of going line by line, word by word" represents a new paradigm of expression. His "Write First, Decide Later" approach is especially valuable---using Claude Code to explore ideas before fully committing to them, then deleting the experimental code if it doesn't work out. This lowers the barrier to experimentation dramatically. The post also touches on the collaborative aspects, with his "Two Clones lifestyle" approach and how it's changing game design workflows at Puzzmo. The ability to rapidly prototype and iterate is transforming their development process from weeks of planning to hours of implementation. I need to figure out Therox's trick for getting Claude conversations into Adium-themed expandables ... might have to reach out! --- ## AI Engineer World's Fair 2025: My Day 2 Highlights **URL:** https://cto4.ai/p/aiewf2025-my-day-2-highlights **Date:** 2025-08-01 **Type:** essay **Description:** My observations on Day 2 of the AI Engineer World's Fair 2025 **Tags:** AIEWF2025, conference, AI engineering **Categories:** AI, Technology In my opinion, the two core days of the AI Engineer World's Fair 2025 in San Francisco represent the ultimate condensation of the state of AI, delivered by many of the the brightest minds in the field. This post covers my Day 2 experience---in case you missed it, here's my [Day 1 recap](/p/aiewf2025-my-day-1-highlights). I'm slow publishing this, for which I feel badly, but re-watching thoughtfully 4-8 weeks after seeing the sessions in person has been a tremendous learning exercise for me, and will, I hope, add to the power of my observations. Here's a quick table of contents; click to jump to the details. | Section | Session | Speaker | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | | Morning Keynotes | [Gemini 2.5 Pro 06-05 Launch - A Year of Shipping and What Comes Next](#gemini-25-pro-06-05-launch---a-year-of-shipping-and-what-comes-next) | Logan Kilpatrick (Google DeepMind) | | | [Thinking Deeper in Gemini](#thinking-deeper-in-gemini) | Jack Rae (Google DeepMind) | | | [Containing Agent Chaos (use-container launch)](#containing-agent-chaos-use-container-launch) | Solomon Hykes (Dagger) | | | [Infrastructure for the Singularity](#infrastructure-for-the-singularity) | Jesse Han (Morph) | | SWE Agent Track | [Devin 2.0 and the Future of SWE](#devin-20-and-the-future-of-swe) | Scott Wu (Cognition) | | | [Your Coding Agent Just Got Cloned](#your-coding-agent-just-got-cloned) | Rustin Banks (Google Labs) | | | [The Agent Awakens](#the-agent-awakens) | Christopher Harrison (GitHub) | | | [Don't get one-shotted](#dont-get-one-shotted) | Tomas Reimers (Graphite) | | | [Claude Code & the evolution of Agentic Coding](#claude-code--the-evolution-of-agentic-coding) | Boris Cherny (Anthropic) | | | [Software Dev Agents: What Works & What Doesn't](#software-dev-agents-what-works--what-doesnt) | Robert Brennan (All Hands AI) | | | [Beyond the Prototype](#beyond-the-prototype) | Josh Albrecht (Imbue) | | | [Ship Production Software in Minutes, Not Months](#ship-production-software-in-minutes-not-months) | Eno Reyes (Factory AI) | | Afternoon Keynotes | [Trends across the AI Frontier](#trends-across-the-ai-frontier) | George Cameron (Artificial Analysis) | | | [State of AI Engineering 2025](#state-of-ai-engineering-2025) | Barr Yaron (Amplify Partners) | | | [Fun Stories from Building OpenRouter and Where All This is Going](#fun-stories) | Alex Atallah (OpenRouter) | | | [The New Code](#the-new-code) | Sean Grove (OpenAI) | ## Gemini 2.5 Pro 06-05 Launch - A Year of Shipping and What Comes Next **Speaker:** [Logan Kilpatrick](https://www.linkedin.com/in/logankilpatrick/) ([Google DeepMind](https://deepmind.google/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=1217s) My level of respect for Google as a model lab has serioiusly lagged Anthropic and OpenAI, but I came away from Logan Kilpatrick's and Jack Rae's sessions recognizing how serious a player they are---very strong today with a great pipeline. Google DeepMind officially launched their most powerful model to date, Gemini 2.5 Pro, live at the event. "I think [2.5 Pro] is setting the stage for the future of Gemini. I think 2.5 Pro for us internally and I think in the perception from the developer ecosystem was the turning point which was super exciting. It's awesome to see the momentum. We've got a bunch of other great models coming as well." Kilpatrick highlighted a year of Gemini Progress: "It feels like 10 years of of Gemini stuff packed into the last 12 months, which has been awesome"---and talked about how so much Google research is coming together in the new Gemini models: "All of these different research bets across Deep Mind [are] coming together to build this incredible mainline Gemini model. What is the DeepMind strategy? What's the advantage for us building models? ... the interesting thing to me is this breadth of research happening across science and Gemini and robotics and things like that ... all that actually ends up upstreaming into the mainline models which is super exciting." He emphasized it's "not just the pace of innovation but the pace of adoption ... a 50x increase in the amount of AI inference that's being processed uh through Google servers from one year ago to um last month ..." and highlighted the reception to VEO: "It's burning all the TPUs down." Among a number of future themes, he says Google models will be increasingly "agentic by default." All in all, an impressive story from Google DeepMind. ## Thinking Deeper in Gemini **Speaker:** [Jack Rae](https://www.linkedin.com/in/jackrae/) ([Google DeepMind](https://deepmind.google/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=1943s) Google DeepMind's Jack Rae did a great job taking us back in time to the first language models up to the present day, with emphasis on test-time compute. (Late-breaking news: Rae has left Google to join Meta, likely as part of their new Superintelligence Labs organization.) Traditional language models are trained to respond immediately to requests, meaning there is a constant, very limited amount of compute applied at test-time. It's possible to increase TTC by using a larger model, but then all requests are forced to this fixed, larger amount of compute. Not a good scaling approach---much better is to train models that are designed to loop. Loop iteration scale is dynamic, so we can get orders of magnitude more TTC, with control over how hard the model "thinks." (Later Rae mentions that "thinking budgets" are now part of the API in the Gemini Flash 2.5 and Pro 2.5 models. Developers get fine-grained control over how much TTC gets applied.) Scaling test-time compute delivers benefits across all model paradigms---there's a feedback loop effect. If you wonder "how can model (AI) progress keep accelerating?" this is a great example: scaled TTC feeds back to accelerate develeopment of more powerful base models. Rae describes the emergent behavior that's appearing through scaled TTC---effectively, the model learns new thinking strategies. To close his keynote, Rae looks at what's next---Deep Think, a high budget thinking mode for 2.5 Pro, which applies parallel, deeper chains of thought, yielding insane improvements on the toughest thinking challenges. Case in point: the 2025 Math Olympiad problem set, where OpenAI's very good o3 reasoning model scored at the 35th percentile, 2.5 Pro reached 50th, and 2.5 Pro Deep Think all the way up to 65th percentile. ## Containing Agent Chaos (use-container launch) **Speaker:** [Solomon Hykes](https://www.linkedin.com/in/solomonhykes/) ([Dagger](https://dagger.io/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=3425s) Solomon Hykes has serious credentials to be the co-founder of an open-source agent containerization framework---the dude created Docker! Hykes kicked things off with an adjustment to an Anthropic slide: An agent is an LLM **wrecking its environment** in a loop: If we're just running one agent---things aren't so bad. But hey, we want to scale it, we want to run 10 agents at once! Neither of today's options are very appealing: Hykes then talks about what we really want in order to safely operate and maintain control of 10 agents, and what the ideal environment looks like to make this work: He points out that we already have the technology we need to deliver this: Containers + Git + Models, and mentions in passing that both containers and Git have a ton of capabilities that aren't widely used by humans, but are perfect for agents. All we're lacking is a native integration ... Hykes and Dagger suggest a solution: "container use for agents"---meant in same the sense as "computer use for agents" and "browser use for agents." This is not sandboxing, which is about _executing the output of agents_. Rather, this is about creating complete, isolated development environments that agents _do their development entirely inside of_. Against the advice of his team, Hykes launches into a wild and crazy live demo of (unfinished) Dagger, where he spawns about six Claude Code instances---and one Goose instance, just for fun---all iterating on a basic website project. Each instance--agent is iterating on the starting apps in complete isolation, with the orchestrating developer having the ability to merge any instance's output if we like it. At the end, while the agents are madly spinning away, he open sources the Dagger project, live in front of the audience. I like Hykes, and I like Dagger---great stuff. ## Infrastructure for the Singularity **Speaker:** [Jesse Han](https://www.linkedin.com/in/jesse-michael-han/) ([Morph](https://morph.ai/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=4845s) I didn't resonate with Jesse Han's presentation style and didn't absorb much from the live presentation. Ah, the benefits of re-watching: I learned a lot more this time around. I see a lot of overlap in Han's and Hykes's keynotes and products. Both have a lot to do with enabling environments that support the safe and effective spin-up and operation of clusters of agents. Both leverage containers, the power of Git and Git branches, and of course, agents. Morph's Infinibranch: "All mistakes become reversible." "All paths forward become possible." Take action, backtrack, take every possible action. Han's chess demo was interesting; he contrasted a single AI working to pick the optimal chess move, with a swarm of agents on the Morph platform tackling the same problem using what he calls "reasoning time branching." Han was fairly arrogant on suggesting that Morph's closed platform would own the agentic execution environment space---a big contrast to Hykes, who open sourced their product on stage. Morph may go on to success, but I'd put money on Dagger winning market share. ## Devin 2.0 and the Future of SWE **Speaker:** [Scott Wu](https://www.linkedin.com/in/scott-wu-8b94ab96/) ([Cognition](https://cognition.ai//)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=8940s) Scott Wu, Co-founder and CEO of Cognition, presented on Devin 2.0 and the evolution of AI-powered software engineering. I missed this session, so will withhold comment until I have a chance to view and absorb Wu's talk. ## Your Coding Agent Just Got Cloned **Speaker:** [Rustin Banks](https://www.linkedin.com/in/rustinbanks/) ([Google Labs](https://labs.google/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=9930s) Rustin Banks from Google Labs talks about his work on [Jules](https://jules.google.com/) and asynchronous coding agents. Another missed session, I'll comment later once I have a chance to view and absorb. ## The Agent Awakens **Speaker:** [Christopher Harrison](https://www.linkedin.com/in/geektrainer/) ([GitHub](https://github.com/features/copilot)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=10812s) Christopher Harrison from GitHub on the evolution of AI agents. I missed this session, so I'll write later once I have a chance to review. ## Don't get one-shotted **Speaker:** [Tomas Reimers](https://www.linkedin.com/in/tomasreimers/) ([Graphite](https://graphite.dev/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=11977s) Tomas Reimers, founder of Graphite, described how to leverage Al to test, review, merge, and deploy code. I missed this talk, but it looks good and I'll write on it once I've watched. ## Claude Code & the evolution of Agentic Coding **Speaker:** [Boris Cherny](https://www.linkedin.com/in/bcherny) ([Anthropic](https://www.anthropic.com/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=18825s) Claude Code wasn't the first terminal-centric AI coding tool---among players with significant followings, those bragging rights probably go to Aider---but Boris Cherny provides a fascinating perspective on why model-maker Anthropic went the terminal route with Claude Code. I have observed first-hand how Anthropic's Claude models have dominated the coding model space, with their lead seeming to accelerate with Sonnet 3.5, 3.7, and 4. Says Cherny, > The model is moving really fast. It's on exponential. It's getting better at coding very, very quickly, as everyone that uses the model knows. And the product is kind of struggling to keep up. We're trying to figure out what product to build that's good enough for a model like this. > I feel like programming languages have sort of leveled out but the model is on an exponential and the UX of programming is also on an exponential ... ide devx has evolved quickly, and will continue to change even more quickly ... that. And so with all this in mind, Claude Code's approach is a little different. > We want to be unopinionated and we want to get out of the way. So we don't give you a bunch of flashy UI. We don't try to put a bunch of scaffolding in the way. Some of this is, we're a model company at Anthropic, we make models and we want people to experience those models. But I think another part is we actually just don't know, we don't know what the right UX is. So we're starting simple. So Claude Code is intentionally simple. It's intentionally general. It shows off the model in the ways that matter to us, which is they can use all your tools and they can fit into all your workflows. So you can figure out how to use the model in this world where the UX of using code and using models is changing so fast. > This is the bitter lesson. I have this like framed and taped to the side of my wall, because the more general model always wins and the model increases in capabilities exponentially and there are many corollaries to this. Everything around the model is also increasing exponentially and the more general thing even around the model usually wins. Cherny describes four different modes of using Claude Code: - From the terminal - From the IDE (as an extension) - As a GitHub app so you can @claude right in a GitHub issue comment - Via the SDK, using Claude as a Unix utility 1. Ask Claude Code about your code 2. Teach Claude to use your tools 3. Fit the workflow to the task, e.g. (explore › plan › confirm › code › commit) vs (tests › commit › code › iterate › commit) vs (code › screenshot › iterate) 4. Plan mode (toggle via Shift-Tab) - Create claude.md files: variants such as CLAUDE.md, CLAUDE.local.md, a/b/CLAUDE.md, ~/.claude/CLAUDE.md - Context can also be made available as a slash command by putting markdown files under .claude/commands/ - Add to Claude's memory by prepending # to something you want Claude Code to remember; CC will ask which memory location to remember into (project, user, etc.) ## Software Dev Agents: What Works & What Doesn't **Speaker:** [Robert Brennan](https://www.linkedin.com/in/robert-a-brennan) ([All Hands AI](https://www.all-hands.dev/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=19945s) Robert Brennan, CEO of All Hands AI, is involved in one of those "AI name confusion" situations. His product (open sourced under an MIT license) used to be known as OpenDevin. And Devin is, of course, one of the pioneers of agentic coding. So what gives? I went to ChatGPT o3 to get the scoop: > **The scoop from o3:** OpenDevin launched in March 2024 as an MIT-licensed community effort to recreate [Devin, the first AI software engineer](https://cognition.ai/blog/introducing-devin). By August 2024 the project re-branded as **OpenHands** under the new startup **All Hands AI**, shedding its “clone” baggage while keeping Devin only as historical inspiration — see the [rename announcement](https://x.com/gneubig/status/1828097484599759349) and the [current OpenHands repo](https://github.com/All-Hands-AI/OpenHands). Brennan kicks off with this message: > Coding is going away ... but that doesn't mean that software engineering is going away. We're paid not to to type on our keyboard but to actually think critically about the problems that are in front of us. Brennan shared a great chart of the codegen landscape, with Tactical ↔ Agentic on the X axis, and Consumer ↔ Developer on the Y. > Now you've got these tools like Devin and OpenHands where you're really giving an agent one or two sentences describing what you want it to do. It goes off and works for 5, 10, 15 minutes on its own and then comes back to you with a solution. This is a much more powerful way of working. Brennan does a nice walkthrough of how coding agents actually work, and then shares tips on how to be most effective working with agentic coding. One point he makes, which I can affirm based on my own experience, is that you shouldn't be afraid to throw code away. > I also like to remind folks that in an AI-driven development world, code is cheap. You can throw code away. You can you can experiment and prototype. I love, if I have an idea, like on my walk to work, I'll just tell OpenHands with my voice, "do X, Y, and Z," and then when I get to work, I'll have a PR waiting for me. 50% of the time, I'll just throw it away. It didn't really work. 50% of the time it looks great, and I just merge it, and it's awesome. Definitely worth watching. ## Beyond the Prototype **Speaker:** [Josh Albrecht](https://www.linkedin.com/in/joshalbrecht) ([Imbue](https://imbue.com/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=21090s) Josh Albrecht, CTO of Imbue, whose main focus is Sculptor, an AI tool that's still in the early research preview. What does it do? > Sculptor is a coding agent environment that applies engineering discipline to catch issues, write tests, and improve your code—all while you work in your favorite editor. Sculptor is the first coding agent environment that helps you embed software engineering best practices. Sculptor runs your code in a sandbox, letting you test code safely, solve issues in parallel, and assign tasks to agents, working alongside any editor. Use Sculptor to resolve bugs, write tests, add new features, improve docs, fix style issues, and make your code better—whether it was written by a human or an LLM. Albrecht's session tagline is **Using Al to Write High-Quality Code** and clearly his work on Sculptor gives him the background to speak intelligently on the subject. Vibe coding output versus production-shippable code. > 'What is wrong with this diff?' Allowing another Al system to come and take a look at this and understand like hey are there any race conditions? Did you leave your API key in there etc. We want to think about how do we help leverage Al tools not just to generate the code but to help us build trust in that code. Four tips for preventing problems in AI-generated code: Regarding writing specs: > In Sculptor, one of the ways that we try to make this easier is by helping detect if the code and the docs have become outdated. So it reduces the barrier to writing and maintaining documentation and dock strings because now you have a way of more automatically fixing the inconsistencies. It can also highlight inconsistencies or parts the specifications that conflict with each other, making it easier to make sure that your system makes sense from the very beginning. Three tips for detecting problems in AI-generated code: Things you can ask an LLM: Solid, practical advice all the way through. ## Ship Production Software in Minutes, Not Months **Speaker:** [Eno Reyes](https://www.linkedin.com/in/enoreyes) ([Factory AI](https://factory.ai/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=22250s) I was blown away when I first heard Eno Reyes and Matan Grinberg on the Latent Space podcast, and I was excited to hear Reyes talk in person. To me, Factory stands out as the most serious AI tool for engineering at Fortune 500 scale, addressing Fortune 500 level problems. Factory also stands out as pioneering in the exploration of AI UX that isn't bolted on to the 20-year-old IDE paradigm. As Reyes says: > It seems like right now we're still trying to find what that interaction pattern, what that future looks like. And a lot of what's publicly available is more or less an incremental improvement. The current zeitgeist is to take tools that were developed 20 years ago for humans to write every individual line of code---tools that were designed first and foremost for human beings. And you sprinkle AI on top and then you keep adding layers of AI and then at some point maybe there's some step function change that happens. But there's not a lot of clarity there in exactly what that means. Reyes shares Andrej Karpathy's quote about English-as-programming-language: But he goes on to confront the idea that this means vibe coding. He speaks to the importance of context, and working with the kind clientele they do, Factory certainly has good data behind this: And it's very clear when Reyes talks about Planning with AI that the F500 problem space Factory works in isn't Tiny Teams territory: In the agile, human coding world that preceded AI coding, I have tended to streamline if not completely avoid what I saw as bloated processes based around heavyweight artifacts like Product Requirements Documents (PRDs). But in the new world, especially at F500 scale, it makes sense that we are coming full circle. Reyes moves on to a Site Reliability Engineering (SRE) example, show context and documents coming together in a loop where the system gets smarter and smarter, moving from helping to diagnose incidents to suggesting "I'm seeing a pattern here---why don't we fix this." Reyes closes by addressing the subject of AIs replacing engineers: > AI agents are not replacing software engineers. They're significantly amplifying their individual capabilities. The best developers l know are spending far less time in the IDE writing lines of code. It's just not high leverage. They're managing agents that can do multiple things at once that are capable of organizing the systems and they're building out patterns that supersede the inner loop of software development and they're moving to the outer loop of software development. They aren't worried about agents taking their jobs. They're too busy using the agents to become even better at what they do. The future belongs to developers who understand how to work with agents, not those who hope that Al will just do the work for them. And in that future, the skill that matters most is not technical knowledge or your ability to optimize a specific system, but your ability to think clearly and communicate effectively with both humans and AI. Mixed in with his slides, Reyes includes several video demos of Factory "Droids" in action---their name for independent agents. Very much worth watching in full. ## Trends across the AI Frontier **Speaker:** [George Cameron](https://www.linkedin.com/in/georgecameron) ([Artificial Analysis](https://artificialanalysis.ai/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=26131s) Artificial Analysis maintains a multidimensional and deeply impressive treasure trove of data about LLMs. George Cameron starts by confirming [Simon Willison's declaration](/p/aiewf2025-my-day-1-highlights#2025-in-llms-so-far) that it's been an insane period of LLM development. Cameron starts with the past 12 months in frontier models: OpenAI still in the lead, but the frontier more competitive than ever: Note: less than 2 months after Cameron's presentation, there've been significant changes in the frontier intelligence rankings: Cameron next points out that there are actually more than one frontier in AI: He goes on to explore each category. Super-sharp analysis; AA is a new go-to for me when I need this type of comparative. The one domain where I'm skeptical of AA's ratings: coding. Either my (and the world-at-large's) experience is mistaken, or AA's coding benchmarks are borked. Otherwise, though, I stick by my pronouncement: a multidimensional and deeply impressive treasure trove. Check [them](https://artificialanalysis.ai/) out. ## State of AI Engineering 2025 **Speaker:** [Barr Yaron](https://www.linkedin.com/in/barryaron) ([Amplify Partners](https://amplifypartners.com/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=27510s) Amplify Partners recently completed their 2025 State of AI Engineering Survey, and partner Barr Yaron presented a number of early findings. Money slide for me: top newsletters and podcasts: ## Fun Stories from Building OpenRouter and Where All This is Going **Speaker:** [Alex Atallah](https://www.linkedin.com/in/alexatallah) ([OpenRouter](https://openrouter.ai/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=28298s) I had quite a few knowledge gaps filled in at the World's Fair; one big one was near-total ignorance of OpenRouter. In reality, it has taken me World's Fair plus two months for the OpenRouter concept to really sink in, and for me to understand the use cases where it makes a ton of sense. I'm just about to set up an OpenRouter account for use on a proof of concept project where I want to avoid setting up many individual direct accounts with model makers. I send my thanks to Alex Atallah, founder of OpenRouter, for helping reduce my ignorance through this talk. So what are the fundamentals of OpenRouter? API to access all language models; and also, because they are a marketplace, an unparalleled source of data about who is using what models. Pay in one place, with near zero switching cost. >400 models, >60 access providers, many payment methods including crypto. The founding story and evolution of OpenRouter is fascinating: a journey from experiment to marketplace. Major labs started banning uses. The open source race and "Llama storm." First successful model distillation: Alpaca, trained for $600. OpenRouter was initially a place to collect all these. Window.ai chrome extension allowing users to choose their own models. OpenRouter's official launch in May '23. OpenRouter wasn't a marketplace originally -- started out with just one primary and one fallback provider for any given model. Then a great proliferation of providers emerged, where a single model being might be available from many providers with different price points and supported feature sets (e.g. with or without structured output). This pushed OpenRouter to become a marketplace. Then it emerged that closed models couldn't keep up with inference demand on their own platforms. OpenRouter helped developers boost uptime by allowing routing to different sources for closed models e.g. Claude models on Amazon Bedrock. Check out this uptime chart to get a feel for the benefits: With all the fantastic data OpenRouter has at its disposal, what do they believe the future looks like? I love the direction OpenRouter is taking with their API, adding a plugin-like system that augments, transparently and in real time, the capabilities of \*_all_ models. Atallah mentions two examples: web search and PDF data extraction. All told, OpenRouter is a major addition to my toolbox, and [this session](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=27510s) is a must-watch. ## The New Code **Speaker:** [Sean Grove](https://x.com/sgrove) ([OpenAI](https://openai.com/)) -- [Session video](https://www.youtube.com/watch?v=U-fMsbY-kHY&t=29425s) Earlier in the day, [Eno Reyes](#ship-production-software-in-minutes-not-months) quoted Andrej Karpathy, "The hottest new programming language is English," and emphasized structured documents like PRDs as an essential way to communicate with AI agents. Picking up where Reyes left off, OpenAI's Sean Grove talked on "The New Code---Specs." Grove did a quick audience survey, asking how many people "coded" and how many of them felt that code was their primary output: He then confronted this belief, suggesting that code is more like 10-20% of the output. What's the far larger slice of impact? What he calls "structured communication." Here's what it looks like: He sees structured communication as the key bottleneck in software engineering, and predicts that it will only get more painful as models get better. > In the near future, the person who communicates most effectively is the most valuable programmer. And literally, if you can communicate effectively, you can program. Vibe coding -- why it feels great -- fundamentally about communication first, coding second. But the weird thing: we communicate via prompts to express our intentions, our values. We then produce an artifact, code, and **throw away** the valuable stuff, the prompts! That's like throwing away the source code and just keeping the binary compiled output. This leads Grove to emphasize that written specs are critical, and why: Grove uses another analogy: specs are like a lossless format, while code is lossy. Specs contain more information, code is a lossy representation that has lost some of that information. He walked us through an example relating to OpenAI's model specs (intentions and values) which are now published; and a recent sycophancy regression identified in an updated version of their GPT-4o model. The model spec contained very specific anti-sycophancy language, but the behavior GPT-4o update was clearly not aligning with those specs. As a result, the team was able to confirm "bug, not intentional," roll back the update, and fix the problem. The written specs acted as a "trust anchor" here, a trustworthy source on how the model should behave. One more corollary: whoever writes the specs is the programmer. Grove's talk, paired with [Reyes' talk earlier in the day](#ship-production-software-in-minutes-not-months), form an **AI Spec Sandwich**. Welcome to the new programming. --- ## A Better Way to Give Claude Code Context **URL:** https://cto4.ai/p/giving-claude-code-context **Date:** 2025-07-31 **Type:** brief **Description:** I like Scott Werner's more structured approach I just finished my first deep dive with [Claude Code](https://www.anthropic.com/claude-code), a Vision proof of concept that extracts handwritten data off clear plastic bags---**filled** plastic bags, that is .... Claude Code and I started Friday, and by Monday we had a robust proof of concept, implemented in Python CLI form, including parallelized LLM calls and detailed documentation. My working time was maybe an hour per day, and the [Anthropic API](https://docs.anthropic.com/en/api/overview) bill for Claude's model usage was about $80. I'd rate the ROI on that $80 investment extremely high .... I knew enough to spend time giving Claude plenty of context. I was at the [AI Engineer World's Fair](https://www.ai.engineer/) in June, had seen Anthropic's [Boris Cherny](https://www.linkedin.com/in/bcherny/) speak. So I had Cherny's context tips, and created a number of markdown files, including `CLAUDE.md` at the project root, along with a well-populated `/docs` folder. I had noted [Scott Werner](https://worksonmymachine.ai/)'s post [Nobody Knows How To Build With AI Yet](https://worksonmymachine.ai/p/nobody-knows-how-to-build-with-ai) when it first came out a week or two back, and finally got around to reading it this morning. Werner levels us up on how to give Claude Code context, adding some well-thought-out separation and structure: His full GitHub Gist [here](https://gist.github.com/swerner/b7fd285ec3f2edbcc55ebccdaffc950a). As Werner caveats, though: > I have no idea what I'll build next week. What process I'll use. What documents I'll create or abandon. Whether any of this will still make sense ... My four-document system isn't a recommendation. It's just one data point in the collective experiment we're all running. A fossil from last week's development process. Already outdated. Already nostalgic. Things move fast in AI time, but for now, I'll apply Werner's approach on my next Claude Code project. --- ## ChatGPT Images Pictorial **URL:** https://cto4.ai/p/chatgpt-images-pictorial **Date:** 2025-07-30 **Type:** brief **Description:** Cthulhu chimes in on the environment Photo I shot on my walk: Found a more complete photo on the web: ChatGPT Images prompt: > Modify this: > > EAST JORDAN becomes CTHULHU > > DUMP NO WASTE! becomes TOO LATE F[expletive]S! > > DRAINS TO WATERWAYS becomes I OWN THE WATERWAYS First ChatGPT Images version (after ChatGPT-enforced rewording of the expletive): Refinement prompt: > Instead of cutting the Cthulhu images into the manhole metal, let’s make them embossed/raised like the fish were Final ChatGPT Images version: Crazy good. --- ## AI Engineer World's Fair 2025: My Day 1 Highlights **URL:** https://cto4.ai/p/aiewf2025-my-day-1-highlights **Date:** 2025-06-18 **Type:** essay **Description:** Key takeaways and observations from Day 1 of the AI Engineer World's Fair 2025 **Tags:** AIEWF2025, conference, AI engineering **Categories:** AI, Technology My live experience at the AI Engineer World's Fair 2025 in San Francisco could be summarized as "extremely intense." As I sat down to recap my highlights, starting with this post covering what Day 1, I ended up re-watching every session, and now I understand better _why_ it felt so intense---the amount of information packed into the short 20 minute sessions was immense, and the people communicating that information were, on average, "brilliant plus" humans ... Here's a quick summary table of what I attended in-person, with links to the detailed sections with my recaps---some brief, some more extended---and links to the session video. I had originally included ratings, but pretty much everything deserved five stars so I canned that idea. I hope these notes help you get a sense of the energy, trends, and ideas shaping the future of AI and AI Engineering. Enjoy! | Section | Session | Speaker | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | Morning Keynotes | [Designing AI-Intensive Applications](#designing-ai-intensive-applications) | Shawn "swyx" Wang (Latent Space) | | | [Spark to System: Building the Open Agentic Web](#spark-to-system-building-the-open-agentic-web) | Asha Sharma (Microsoft) | | | [State of Startups and AI 2025](#state-of-startups-and-ai-2025) | Sarah Guo (Conviction) | | | [2025 in LLMs so far](#2025-in-llms-so-far) | Simon Willison (SimonWillison.net) | | MCP Track | [MCP Origins & Request for Startups](#mcp-origins--rfs) | Theo Chu (Anthropic) | | | [What we learned from shipping remote MCP support at Anthropic](#what-we-learned-from-shipping-remote-mcp-support-at-anthropic) | John Welsh (Anthropic) | | | [Full Spectrum MCP: Uncovering Hidden Servers and Clients Capabilities](#full-spectrum-mcp-uncovering-hidden-servers-and-clients-capabilities) | Harald Kirschner (VS Code, Microsoft) | | | [MCP isn't good, yet](#mcp-isnt-good-yet) | David Cramer (Sentry) | | | [MCP is all you need](#mcp-is-all-you-need) | Samuel Colvin (Pydantic) | | | [Observable tools - the state of MCP observability](#observable-tools---the-state-of-mcp-observability) | Alex Volkov (Weights & Biases), Benjamin Eckel (Dylibso) | | | [The rise of the agentic economy on the shoulders of MCP](#the-rise-of-the-agentic-economy-on-the-shoulders-of-mcp) | Jan Čurn (Apify) | | Afternoon Keynotes | [Building Agents at Cloud-Scale](#building-agents-at-cloud-scale) | Antje Barth (AWS) | | | [Windsurf everywhere, doing everything, all at once](#windsurf-everywhere-doing-everything-all-at-once) | Kevin Hou (Windsurf) | | | [Define AI Engineer](#define-ai-engineer) | Greg Brockman (OpenAI), Shawn "swyx" Wang (Latent Space) | ## Designing AI-Intensive Applications **Speaker:** [Shawn "swyx" Wang](https://www.swyx.io/) ([Latent Space](https://latentspace.co/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=1337s) Swyx delivers an incredible overview of the state of the space. He compares today's state of AI today to early days in physics where the [Standard Model](https://en.wikipedia.org/wiki/Standard_Model) was developed and ended up serving physics almost unchanged through today. Swyx proposes a candidate for AI's own Standard Model, SPADE: ## Spark to System: Building the Open Agentic Web **Speaker:** [Asha Sharma](https://www.linkedin.com/in/aboutasha) ([Microsoft](https://www.microsoft.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=2146s) As with [AWS](https://aws.amazon.com/)'s keynote later in the day featuring [Antje Barth](https://www.linkedin.com/in/antje-barth), this session with Microsoft's Asha Sharma set off my "oh no, vendor product-flogging ahead" alarm. That's not being fair though; we need major players like MS who can deliver necessary, massive-scale platforms, and they need to flog the sexy bits so they can sell the boring ones; no margin, no mission, as they say. It's also not fair in that Microsoft has staked out major segments in the AI landscape, even beyond their huge (if now troubled) partnership with [OpenAI](https://openai.com/)---[Copilot](https://github.com/features/copilot) of course, but also the [VS Code](https://code.visualstudio.com/) ecosystem with its pioneering role in AI Coding, [GitHub](https://github.com/) serving as a core source of context for AI Coding and actions in the agentic subcategory, and on and on. Sharma shows us some significant new capabilities arriving in Copilot, and with her team presents a couple different agentic app solutions. Worth watching, understanding Microsoft's broad suite of AI offerings is essential. ## State of Startups and AI 2025 **Speaker:** [Sarah Guo](https://www.linkedin.com/in/sarahxguo) ([Conviction](https://conviction.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=3542s) Sarah wins the "best save" award when AV fails, leaving her with no slides ... she still pulls off a great presentation, and doesn't even run over. Excellent overview of the state of AI, the pace of progress, trends. I'm a fan of the [Cursor](https://cursor.sh/) AI Coding tool---one of Conviction's portfolio companies---and I love Sarah's recommendation to startups that they should think about building "Cursor for X" ... it's a nuanced message, though: she gets to the heart of what Cursor actually is: an LLM wrapper, yes, but a **thick, juicy** wrapper. ## 2025 in LLMs so far **Speaker:** [Simon Willison](https://simonwillison.net/) ([SimonWillison.net](https://simonwillison.net/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=5084s) I'm a huge fan of Simon's, and here he's in fine form---fine enough that this presentation won best-of-conference. As an indicator of the accelerating pace of LLM development, Simon was forced to cut the scope of his keynote from "the past year" to "the past six months" to have a prayer of finishing in the allotted 20 minutes. His "draw a pelican riding a bicycle" LLM benchmark is both hilarious and an uncannily accurate LLM quality assessment. At least until "the pelican test" starts to contaminate model training---I'm betting next year this benchmark may no longer yield valid results after Willison showed the [Google Gemini 2.5 Pro](https://deepmind.google/technologies/gemini/) pelican on a [Google](https://www.google.com/) slide from the recent [Google IO](https://io.google/2025/) event ... ## MCP Origins & Request for Startups **Speaker:** [Theo Chu](https://www.linkedin.com/in/theodorachu) ([Anthropic](https://www.anthropic.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=9076s) Great intro to [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), an open standard invented by Anthropic---I love authoritative sources. Origins and futures. Highly recommended, start here on MCP. ## What we learned from shipping remote MCP support at Anthropic **Speaker:** [John Welsh](https://www.linkedin.com/in/johnw188) ([Anthropic](https://www.anthropic.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=10187s) This session got straight to the heart of my most important questions about MCP. Even more background on MCP origins, vital information on the elements of MCP. Love the "pit of success" concept---make the right thing to do, the easiest thing to do. Excellent. ## Full Spectrum MCP: Uncovering Hidden Servers and Clients Capabilities **Speaker:** [Harald Kirschner](https://www.linkedin.com/in/digitarald) ([VS Code](https://code.visualstudio.com/), [Microsoft](https://www.microsoft.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=11031s) Now we have authoritative from Microsoft: Harald Kirschner, a Principal Product Manager working on VS Code and [GitHub Copilot](https://github.com/features/copilot), who was instrumental in releasing Agent Mode in Copilot. Kirschner dives a layer deeper into MCP, with great insights into how "early days" we are, and the basic primitives of the protocol. He does a great job of explaining the self-reinforcing trap that MCP finds itself in at this early stage, where both MCP servers and clients all tend to support the `tools` primitive, but most lack support (especially _thoughtful_ support) of more powerful primitives such as `resources` and `sampling` that enable what he calls "rich, **stateful** interactions." And guess what he announced for VS Code: full MCP spec support. ## MCP isn't good, yet **Speaker:** [David Cramer](https://www.linkedin.com/in/dmcramer) ([Sentry](https://sentry.io/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=11934s) I immediately liked Cramer and his common-sense, from-the-trenches perspective on MCP. Loved his "hot takes" imagery---he described the current phenomenon with MCP where there are lots of "I have opinions but haven't actually built anything" people hanging around. Sentry wasn't the earliest MCP adopter; what held them back was that they needed a Remote MCP solution with [OAuth](https://oauth.net/), which is only recently becoming viable. He warns about the many problems inherent with `stdio` and recommends to not screw around with it, just do OAuth, even internally---it will be worth the effort. The Sentry MCP is a really a great example of "yet another bit of context" for the IDE, and their thoughtful approach instructive. Cramer reinforced the message that we can't just wrap our APIs as MCP tools, with the same endpoints and the same payloads. "MCP is not just a thing that sits on top of [OpenAPI](https://www.openapis.org/) ..." Why? "Robots don't know how to reason about giant JSON payloads that were not built for them .. we can't just expose all those things as tools ..." He makes it clear that we "need to really think about how would you use an agent today, how would the models react to what you do when you provide them context which is what this really is for, and design a system around that ... it might leverage your but it is not your API." In terms of what tools should return, he explains that, in practice JSON is the wrong answer, because language models don't actually handle "unexpected JSON" well: "It can kinda figure out JSON here and there but if you actually push it you'll find it breaks all the time ..." What payload format to return then? Instead of JSON, they ended up using [Markdown](https://daringfireball.net/projects/markdown/). They expose _some_ of the API endpoints, and return _some_ of API's response as Markdown: "The bare essentials, in a structured way, that a human can think about---because if a human can reason about it, the language model can reason about it ..." Other great tips. "Mind your tokens ..." because the user is paying for them. Or even spend some tokens yourself, by exposing agents (Sentry's "Seer" for example) instead of dumb tools. Cramer says agentic-loop MCPs don't work well yet (due to lack of streaming, loop waiting delays, etc.) but the promise is real and when it arrives the value will be super high. "This stuff is not that hard ... you can just go build it and try it out ..." "Everybody is scared of all this stuff because there's fancy new words for everything ... but those fancy new words for things we already have, a new coat of paint---MCP is just a plug-in architecture, agents are just services, LLM calls and MCP calls are just API calls with a new response format, etc." [Cloudflare](https://www.cloudflare.com/) usage was really interesting: "We used a lot of Cloudflare tech ... shim up a thing on [Workers](https://workers.cloudflare.com/), they have an OAuth proxy ..." explaining "we didn't have websocket infrastructure internally, but the MCP protocol requires it ... problem solved ..." Super valuable session. ## MCP is all you need **Speaker:** [Samuel Colvin](https://www.linkedin.com/in/samuel-colvin) ([Pydantic](https://pydantic.dev/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=18485s) Colvin and Pydantic are famous for the Pydantic Python package, which had been downloaded 1.6B times so far this year, as of the date of the talk. The title "MCP is all you need" plays off [Jason Liu](https://www.linkedin.com/in/jxnlco)'s "Pydantic is all you need" talk from about two years ago, and follow-on talk from about a year ago, "Pydantic is still all you need." Both talks have the same theme: you're overcomplicating things, people! Colvin goes on to show is just how powerful MCP can be, in conjunction with Pydantic's (also free open source) [Pydantic AI](https://ai.pydantic.dev/) package, along with [Pydantic Logfire](https://logfire.sh/), their paid observability platform based on the [OpenTelemetry](https://opentelemetry.io/) standard. Colvin walks through all the code, and runs it, right there during his 20-minute talk, despite Wi-Fi troubles. Colvin explains why MCP's tool calling capabilities are much more complex and powerful than you might at first imagine---dynamic tools, logging, sampling, etc. His explanation of the poorly-named and frequently-misunderstood `sampling` element of the MCP protocol is excellent---he makes clear exactly why sampling is so useful and critical. The power of well-implemented observability within an MCP server is clearly on display, delivered via Logfire, which has landed on my go-to list for observability. ## Observable tools - the state of MCP observability **Speakers:** [Alex Volkov](https://www.linkedin.com/in/alex-volkov-) ([Weights & Biases](https://wandb.ai/)), [Benjamin Eckel](https://www.linkedin.com/in/benjamin-eckel-b025831a3) ([Dylibso](https://dylibso.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=19543s) This session took me deeper into the details of MCP observability than I needed to go---I need to know what's possible, and where to find the details when I need them, but not the full details right now when I don't yet need them. For someone in the midst of building a robust MCP for production use, though, I'm sure this content would be super useful. Alex Volkov's Weights & Biases is known for their observability tool [Weave](https://wandb.ai/weave). Benjamin Eckel operates [MCP.run](https://mcp.run/), which consists of a registry of MCPs, along with tools to glue together, orchestrate and run MCPs (AI services). Both feel the pain of the MCP observability blind spot (black boxes). Both recommend a standards-based approach: leverage OpenTelemetry (OTel) within MCPs as the best approach for making the MCP black box observable. They explain core concepts of OTel---traces, spans, sinks, etc. Both show code solutions on their respective platforms applying OTel. Volkov also shows a great example of MCP in action in [Windsurf](https://windsurf.com/), relating to debugging code via MCP tracing. ## The rise of the agentic economy on the shoulders of MCP **Speaker:** [Jan Čurn](https://www.linkedin.com/in/jancurn/) ([Apify](https://apify.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=20580s) Čurn's company Apify is a marketplace (read: monetized) of 5,000+ Actors (read agent-like entities), historically web scraping centric, but more recently including other tool categories. Actor creators make money when their Actors are run by Apify customers, with Apify handling payments.infrastructure strong support of standa MCPs are a slam-dunk addition to the Apify ecosystem. Apify was able to expose all 5,000+ Actors thanks to what Čurn calls the killer feature of MCPs, Tool Discovery. Not many clients support it yet---VS Code and [Claude desktop](https://claude.ai/download) just added support. Čurn explains that, while there's no way they could publish 5,000+ Actors via OpenAPI due to search challenges, MCP's Tool Discovery provided an elegant solution. Čurn points out how dominant MCP is becoming for agentic interaction. There are so many MCP registries that [Mastra](https://mastra.ai/) even has a [registry of registries](https://mastra.ai/mcp-registry-registry). Double meta. He considers Tool Discovery to be a huge differentiator for MCP. Čurn suggests that, thanks to the richness of the MCP protocol, an agentic economy will arise based on MCP. But: there's a killer problem: how to handle payments when a group (tree) of paid agents are working together to solve a problem? You might be able to discover other agents, but how would you pay them, each the normal setup is for each service to have its own API key and associated account? Give the parent agent a credit card? Bad for so many reasons. Central identity and payments providers? Maybe someday, but it's hard and expensive to launch something like that. Apify has a solution: developers just deal with one API key, and interact only with mcp.apify.com; and entire tree of Actors can issue charges, which are passed back up to the dev via Apify. A great demo ensued using Claude Desktop. ## Building Agents at Cloud-Scale **Speaker:** [Antje Barth](https://www.linkedin.com/in/antje-barth) ([AWS](https://aws.amazon.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=26122s) As with Microsoft's keynote featuring Asha Sharma, this session with AWS's Antje Barth set off my "oh no, vendor flogging ahead" warning lights. That's not being fair though; we need major players like AWS who can deliver massive-scale platforms, and when you're AWS or Microsoft, you have to flog the sexy bits so you can sell the boring ones; no margin, no mission, as they say. It's also not fair in that, as with Microsoft, Barth and AWS actually had interesting things to show off. [Alexa+](https://www.amazon.com/Meet-the-new-Alexa/dp/B0DCCNHWV5) sure looks like it brings to the smart home device the level of real-time intelligence and interactivity that OpenAI's [Advanced Voice Mode](https://techcrunch.com/2025/06/09/openai-updates-chatgpts-voice-mode-with-more-natural-sounding-speech/) delivers in their mobile app. I'm cautious, though: Amazon has botched their smart home strategy in the past---see my current Alexa deployment below. My Apple HomePods are likely to join them on the junk shelf as soon as someone trustworthy and with taste delivers on the promise. Apple qualifies as "trustworthy and with taste"---but they continue to foot-gun on AI. It's actually interesting how much higher my trust level is for AWS than for mainline Amazon products like Alexa, which leads us to Barth's main topic, agents at cloud scale. Barth did a quick demo of the [AWS Q CLI](https://aws.amazon.com/q/) tool---not impressive on its own, but its three-week time-to-launch was more so. How? [AWS Strands Agents (SA)](https://aws.amazon.com/blogs/opensource/introducing-strands-agents-an-open-source-ai-agents-sdk/). Good things about SA: based over [Bedrock](https://aws.amazon.com/bedrock/), which means a wide selection of models to work with, including Anthropic's; and a nice starting set of 20 pre-built tools including memory/[RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) (the Retrieve tool sounds particularly interesting). What's less clear is how the SA architecture fits in with MCP; what does SA bring to the table that MCP by itself doesn't provide? AWS has lots of MCP server implementations, and MCPs can be used as tools within SA; but AWS seem to view MCP as "just one of the open protocols" for agents. We'll have to wait and see how this plays out. ## Windsurf everywhere, doing everything, all at once **Speaker:** [Kevin Hou](https://www.linkedin.com/in/kevinhou22) ([Windsurf](https://windsurf.com/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=27266s) In the AI Coding space, IDE-tool subspace, my most-admired product, and also the one I have the most experience with, is Cursor. In my experience, which combines a lot of paying attention to what others think and a moderate degree of hands-on use, Cursor leads this sub-niche. The AIEWF2025 audience seemed to agree, when someone asked for raised hands for who uses Windsurf, and who uses Cursor. My estimate was about 5:1 Cursor:Windsurf. Having seen Kevin Hou's session live, and re-watching it as I write this, I didn't see anything to change my opinion. Hou claimed he was sharing Windsurf's "secret sauce"---shared timelines between human and AI---but this and other points made in the sessions struck me as marketing points as opposed to actual breakthroughs. I didn't see anything new; I just saw obvious points made as if I knew nothing about other tools or the current state-of-the-art in AI Coding. It's possible, no doubt, that I'm missing something important. I strive to remain open-minded. I will give Windsurf major credit for capitalizing on OpenAI's need to beef up their AI Coding story. If Windsurf can close that $3B deal---that fat lady hasn't sung quite yet---being part of OpenAI may give Windsurf the unique advantage that its current product capabilities do not. ## Define AI Engineer **Speakers:** [Greg Brockman](https://www.linkedin.com/in/gdb/) ([OpenAI](https://openai.com/)), [Shawn "swyx" Wang](https://www.swyx.io/) ([Latent Space](https://latentspace.co/)) -- [Session video](https://www.youtube.com/watch?v=z4zXicOAF28&t=28290s) Swyx conducts a fascinating interview of Brockman, with a little help from [Nvidia](https://www.nvidia.com/)'s [Jensen Huang](https://www.linkedin.com/in/jenhsunhuang/) (prerecorded). Having re-watched as part of writing my recap, I picked up a lot I missed in person. There's so much content here, and I'll zoom in on questions in the AI Coding domain. **Swyx:** What are your thoughts on vibe coding? **Brockman:** I think that vibe coding is amazing as an empowerment mechanism, right? I think it's sort of a representation of what is to come. And I think that the specifics of what vibe coding is, I think that's going to change over time, right? I think that you look at even things like [Codex](https://openai.com/codex/) like to some extent I think our vision is that as you start to have agents that really work that you can have not just one copy not just 10 copies but you can have a hundred or thousand or 10,000 100 thousand of these things running you're going to want to treat them much more like a co-worker, right, that you're going to want them off in the cloud doing stuff being able to hook up to all sorts of things you're asleep, your laptop's closed it should still be working. I think that the current conception of of vibe coding in an interactive loop ... My prediction of what will happen ... there's going to be more and more of that happening, but I think that the agentic stuff is going to also really intercept and overtake. And I think that all of this is just going to result in just way more systems being built. Also very interesting is that a lot of the vibe coding kind of demos and and the cool flashy stuff, for example, making the joke website, it's making an app from scratch. But the thing that I think will really be new and transformative and starting to really happen is being able to transform existing applications to go deeper. So many companies are sitting on legacy codebases and doing migrations and updating libraries and changing your [COBOL](https://en.wikipedia.org/wiki/COBOL) language to something else is so hard and is actually just not very fun for humans and we're starting to get AI that are able to really tackle those problems and the thing that I love about where vibe coding started has really been like with the most like just make cool apps kind of thing and it's starting to become much more like serious software engineering. And I think that going even deeper to just like making it possible to just move so much faster as a company. That's I think where we're headed. **Swyx:** How do you think that Codex changes the way we code? **Brockman:** The direction is something that is like just so compelling and incredible to me. The thing that has been the most interesting to see has been when you realize that the way your structure your codebase determines how much you can get out of Codex, right? Like all of our existing codebases are kind of matched to the strengths of humans. But if you match instead to the strength of models which are sort of very lopsided, right? Models are able to handle way more like diversity of stuff but are not able to sort of necessarily connect deep ideas as much as humans are right now. And so what you kind of want to do is make smaller modules that are well tested that have tests that can be run very quickly and then fill in the details. The model will just do that, right? And it'll run the test itself. The connection between these different components, kind of the architecture diagram, that's actually pretty easy to do, and then it's like filling out all the details that is often very difficult. And if you actually do that, you know, what I described also sounds a lot like good software engineering practice. But it's just like sometimes because humans are capable of holding more of this like conceptual abstraction in our head, we just don't do it -- it's a lot of work to write these tests and to flesh them out and that the model's going to run these tests like a hundred times or a thousand times more than you will and so it's going to care way way more. So in some ways the direction we want to go is build our codebases for more junior developers in order to actually get the most out of these models. Now it'll be very interesting to see as we increase the model capability, does this particular way of structuring code bases remain constant? I think that it's a good idea because again, it starts to match what you should be doing for maintainability for humans. But yeah, I think that to me the really exciting thing to think about for the future of software engineering is, what of our practices that we cut corners on, do we actually really need to bring back, in order to get the most out of our systems? **Swyx:** Can you ballpark numbers on the amount of productivity you guys are seeing with Codex internally? **Brockman:** Yeah I don't know what the latest numbers are. I mean, there's definitely a low double digit written entirely by Codex. That's super cool to see. But it's also it's not the only system that we use internally and I think that to me it's still in the very early days. It's been exciting to see some of the external metrics. I think we had 24,000 PRs that were merged in like the last day in public GitHub repositories. And so yeah, this stuff is all just getting started. **Jensen Huang**: AI native engineers in the audience, they are probably thinking, in the coming years, OpenAI will have AGIs and they will be building domain specific agents on top of the AGIs from OpenAI. So some of the questions that I would have on my mind would be, how do you think their development workflow would change as OpenAI's AGIs become much more capable and yet they would still have plumbing workflows pipelines that they would create, flywheels that they would create for their domain specific agents. These agents would of course be able to reason, plan, use tools, have memory, short-term, long-term memory, and they'll be amazing amazing agents, but how does it change the development process in the coming years? **Brockman**: Yeah, I think that this is a really fascinating question, right? I think you can find a wide spectrum of very strongly held opinion that is all mutually contradictory. I think my perspective is that first of all, it's all on the table, right? Maybe we reach a world where it's just like the AIs are so capable that we all just let them write all the code. Maybe there's a world where you have like one Al in the sky. Maybe it's that you actually have a bunch of domain specific agents that require a bunch of of specific work in order to make that happen. I think the evidence has really been shifting towards this menagerie of different models---I think that's actually really exciting. There's different inference costs, there's different trade-offs like just distillation works so well, there's actually a lot of power to be had by models that are actually able to use other models. I think that that is going to open up just a ton of opportunity because you know we're heading to a world where the economy is fundamentally powered by AI. We're not there yet but you can see it right on the horizon. I mean that's what people in this room are building -- that is what you are doing. The economy is a very big thing, there's a lot of diversity in it, and it's also not static, right? I think when people think about what AI can do for us it's very easy to only look at what are we doing now, and how does AI slot in, and the percentage of human versus AI, but that's not the point. The point is how do we get 10x more activity, 10x more economic output, 10x more benefit to everyone. I think the direction we're heading is one where the models will get much more capable, there'll be much better fundamental technology, and there's just going to be way more things we want to do with it and the barrier to entry will be lower than ever. And so things like healthcare that requires responsibility to go in and think about how to do it right. Things like education where there's multiple stakeholders -- the parent, the teacher, the student -- each of these requires domain expertise, requires careful thought, requires a lot of work. So I think that there is going to be so much opportunity for people to build. So l'm just so excited to see everyone in this room because that's the right kind of energy. --- ## AI Writing Index, Feb '23 to Jun '24 **URL:** https://cto4.ai/p/ai-writing-index **Date:** 2025-06-10 **Type:** essay **Description:** My early AI work, in index form, with reflections **Tags:** AI history, predictions, technology trends **Categories:** AI, Technology As I dig in to recap what I saw and learned at the [AI Engineer World's Fair 2025 in San Francisco](https://www.ai.engineer/), I was forced to face my failure to write consistently -- a failure that dates back almost exactly a year, to June 12, 2024. Thinking back on it, I see two causes. First, my book of business on the fractional CTO side of things filled up completely, reducing available writing time. Second, the [AI emergence](/p/the-emergence-s1e4), like [Solomon Epstein's ill-fated space yacht in _The Expanse_](https://www.dailymotion.com/video/x6qx9dj), just kept accelerating. I had "gotta write about that!!" panics daily, until I finally had to just stop rather than driving myself nuts. I do have to say, though, that during the 16 months from February 2023 to June 2024: - I was early to AI - I wrote a lot - It was pretty decent stuff - My opinions and predictions regularly proved accurate. (One prediction, that [Apple might get pinched](/p/an-appllama-week#my-take-on-apple-in-llmgpt-land) by the AI emergence, is playing out in front of our eyes right now. Apple had little AI-related to announce at this week's WWDC, and had no visibility whatsoever at [AIEWF2025](https://www.ai.engineer/).) It's also pretty amazing to see what has happened in the past year. At AIEWF2025, Simon Willison was forced to cut the scope of [his LLM update keynote](https://simonwillison.net/2025/Jun/6/six-months-in-llms/) from "the past year" to "the past six months" to have a prayer of finishing in the allotted 20 minutes. Anyway, take a look at some of the stuff below and see what you think. P.S.: In June 2025, [Cursor](https://www.cursor.com/) and LLMs are vastly better at writing Markdown than a year ago. ## Blog Posts | Title | Date | Description | | ------------------------------------------------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | [Using Multimodal AI to Capture and Enrich Heirloom Recipes](/p/chatgpt-4v) | 2024-02-19 | I applied OpenAI's GPT-4 Vision model and Chat Completions API to preserve a treasure-trove of legacy family recipes. | | [An Appllama Week in AI](/p/an-appllama-week) | 2023-07-21 | Meta makes waves with Llama 2, while Bloomberg pumps itself with Apple LLM non-news. | | [Really, OpenAI?](/p/really-openai) | 2023-07-07 | ChatGPT+ subscribers can prevent OpenAI from using their inputs as training data. That is, so long as they forego the service's second-best feature. | | [Of Moats and Moat Busters](/p/moat-busters) | 2023-06-29 | Open Source, Commercial-friendly AI Challenges the Major Closed AI Players | | [Unlimited Stupid Questions](/p/unlimited-stupid-questions) | 2023-06-14 | Possibly the most important benefit of ChatGPT ... | | [Is Runway Gen-2 Midjourney for Video?](/p/runway-not-midjourney-for-video) | 2023-06-08 | We Tested Gen-2 and Share the Resulting Videos | | [The Emergence](/p/the-emergence-s1e4) | 2023-04-18 | Season 1, Episode 4 | | [March Madness, AI Style](/p/march-madness-ai) | 2023-03-18 | Observations on AI's March acceleration | | [Alarmed About AI?](/p/alarmed-about-ai) | 2023-02-20 | John Seely Brown, former director of Xerox PARC, has helpful advice—from April 2000! | | [Accelerating Crafty's Launch Using AI Models](/p/accelerating-launch-with-ai-models) | 2023-02-08 | Thanks to AI-based tools like Midjourney, Crafty shaved at least a month off its time to launch. | ## Micro Posts | Title | Date | Description | | ------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Apple Intelligence Initial Thoughts](/p/apple-intelligence-initial-thoughts) | 2024-06-11 | Mostly from other people, I'm still digesting | | [A Quick Look at GPT-4o](/p/gpt-4o-quick-benchmark) | 2024-05-19 | I did a quick benchmark of the new GPT-4o model versus GPT-4-turbo. Roughly twice the speed, with improved Vision accuracy. | | [OpenAI Vision + Function Calling Examples](/p/openai-vision-function-calling) | 2024-04-28 | OpenAI's April 9th model update merges Turbo and Vision, and these examples show the two working together. | | [Simon Willison's LLM Tool: Now I Have 50 LLMs](/p/50-llms) | 2024-03-25 | I updated my installation of Willison's LLM tool to add plugins, and now I have 50 LLMs at my fingertips, including 15 local models, which get installed on demand. | | [Did GPT-4 Code Interpreter Escape From Its Sandbox?](/p/did-code-interpreter-escape) | 2023-07-14 | I gave Code Interpreter a workout this morning, and it appeared to exit the building. | | [ChatGPT+ Privacy Update](/p/really-openai-update) | 2023-07-10 | There appears to be a workaround for now to OpenAI's "If you want privacy, no chat history for you!" policy. | | [Llama 2](/p/llama-2) | 2023-07-19 | HN comments on Llama 2. | | [Apple Really Did Ignore the AI Emergence at WWDC](/p/apple-did-it) | 2023-06-18 | As I watched the WWDC Keynote and Platforms State of the Union from this year's WWDC, I was amazed that Apple appeared to be ignoring the current massive emergence in generative AI. Later, I confirmed it. | | [ChatGPT Code Review](/p/chatgpt-codereview) | 2023-06-03 | Another great use: coming up to speed on an unfamiliar codebase. | | [Working With My ChatGPT Intern](/p/chatgpt-record) | 2023-05-29 | Yesterday was a new record. | ## Elsewhere | Title | Date | | --------------------------------------------------------------------------------------------------------------------- | ----------------- | | [Two Worthwhile Reads From Simon Willison](/p/two-from-willison) | March 25, 2024 | | [Willison: The killer app of Gemini Pro 1.5 is video](/p/willison-gemini-pro-video) | February 22, 2024 | | [Ethan Mollick: "How to Use AI to Do Stuff"](/p/how-to-use-ai-to-do-stuff) | July 26, 2023 | | [OpenAI Concedes: AI Can't Detect AI](/p/ai-cant-detect-ai) | July 26, 2023 | | ["Commoditizing the Petaflop" with George Hotz of the tiny corp](/p/latent-space-george-hotz) | July 9, 2023 | | [Just-released GPT-4 Code Interpreter is a Big Deal, Part 1](/p/gpt4-code-interpreter-big-deal-pt1) | July 11, 2023 | | [Just-released GPT-4 Code Interpreter is a Big Deal, Part 2](/p/gpt4-code-interpreter-big-deal-pt2) | July 11, 2023 | | [The Lone Banana Problem](/p/lone-banana-problem) | July 5, 2023 | | [Midjourney V5.2 (Ars Technica)](/p/midjourney-v52) | June 24, 2023 | | [I-JEPA: The first AI model based on Yann LeCun's vision for more human-like AI](/p/meta-i-jepa) | June 19, 2023 | | [Comparing Adobe Firefly, Dalle-2, OpenJourney, Stable Diffusion, and Midjourney](/p/comparing-art-ais) | June 21, 2023 | | [Faster sorting algorithms discovered using deep reinforcement learning](/p/deep-rl-faster-sorting) | June 7, 2023 | | [It's infuriatingly hard to understand how closed models train on their input](/p/willison-infuriating-closed-models) | June 4, 2023 | | [OpenAI's plans according to Sam Altman](/p/openai-plans) | June 1, 2023 | | [Lawyer cites fake cases invented by ChatGPT, judge is not amused](/p/simon-deep-dives-fake-cases) | May 30, 2023 | | [CLI tools for working with ChatGPT and other LLMs](/p/simon-cli-for-llm) | May 29, 2023 | ## Quotes | Title | Date | | --------------------------------------------------------------------------------------------- | -------------- | | [How to Ask ChatGPT a Technical Question](/p/how-to-ask-chatgpt-a-tech-question) | March 17, 2024 | | [MosaicML's Open Source MPT-7B Model Writes an Epilogue to The Great Gatsby](/p/mpt7b-gatsby) | March 11, 2024 | | [Gruber Responds to Gurman's Report of AI Anxiety at Apple](/p/gruber-ai-anxiety-at-apple) | March 5, 2024 | | [Vivek Haldar on GitHub Copilot](/p/vivek-haldar-copilot) | June 10, 2023 | --- ## Apple Intelligence Initial Thoughts **URL:** https://cto4.ai/p/apple-intelligence-initial-thoughts **Date:** 2024-06-11 **Type:** brief **Description:** Mostly from other people, I'm still digesting I've [been busy](/p/observable-framework-blazing-fast) and haven't watched any of the WWDC 2024 sessions yet, but I've definitely been observing news and people's impressions. Having one foot deeply in the Apple world, and the other just as deeply sunk into AI, provides an interesting vantage point. Over on Hacker News, the top of the [Apple Intelligence for iPhone, iPad, and Mac](https://news.ycombinator.com/item?id=40636844) comment thread was interesting: To me, the key insights are: > It's the benefit of how Apple does product ownership. In contrast to Google and Microsoft ... I hadn't considered it, but AI convergence is going to lay bare organizational deficiencies in a way previous revolutions didn't ... Nobody wants a GenAI feature that works in Gmail, a different one that works in Messages, etc. -- they want a platform capability that works anywhere they use text ... I'm not sure either Google or Microsoft are organizationally-capable of delivering that, at this point. And: > 'AI convergence is going to lay bare organizational deficiencies in a way previous revolutions didn't' > > Your quote really hit me. I trust Apple to respect my privacy when doing AI, but the thought of Microsoft or Google slurping up all my data to do remote-server AI is abhorrent. I can't see how Microsoft or Google can undo the last 10 years to fix this. It's early days yet, but if Apple manages the jiu-jitsu move of leveraging their platform strengths to overcome a huge lead by other players, that will be quite an achievement at the product and company level. Elsewhere, Simon Willison has a [good early take generally](https://simonwillison.net/2024/Jun/10/apple-intelligence/) and specifically on Apple's [Private Compute Cloud](https://simonwillison.net/2024/Jun/11/private-cloud-compute/). --- ## Observable Framework Delivers Blazing-fast Data Dashboards **URL:** https://cto4.ai/p/observable-framework-blazing-fast **Date:** 2024-06-10 **Type:** essay **Description:** An exciting new offering, launched in February, from the team that brought us Observable Notebooks ## Stumbled Upon ... I've had requests to share my [Heirloom Recipe Ingestor project](../chatgpt-4v/), but it's still in the hacked-together stage, not ready. I've been wanting to clean it up, and when I made updates to use the non-preview GPT-4-turbo model that [OpenAI released on April 9th](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4), I began cleaning up the code. To make the project public, though, I wanted to do more significant remodeling. One example: the script currently runs in CLI mode with arguments, but it's not implemented in the latest-best-Python way, and I'd definitely want to correct that. ([Typer from Tiangolo](https://typer.tiangolo.com/) looks like a promising solution, by the way). As I was mulling this, I came across a [pair of new Vision examples](https://cookbook.openai.com/examples/multimodal/using_gpt4_vision_with_function_calling) in the OpenAI Cookbook. Besides finding the examples [very useful](/p/openai-vision-function-calling), I noted that OpenAI shared these as a [Jupyter Notebook](https://jupyter.org/), rather than as a traditional Python project. That got me wondering if maybe I should the same approach for Recipe Ingestor. Before I commit heavily to a particular technology or platform, I like to survey of the state of the art in that particular niche. So before committing to Jupyter Notebook / Jupyter Lab, I did a quick survey of the state of the art in data notebooks---was there an up-and-coming alternative to Jupyter I should be looking at? My [Kagi](https://kagi.com/) search returned hits for [Observable Notebooks](https://observablehq.com/documentation/notebooks/), a tool I had peripheral awareness of through Simon Willison, who [regularly publishes work in Observable Notebook form](https://simonwillison.net/tags/observable/), when that format seems a better fit than, say, a simple Python script. So I dropped in over at [ObservableHQ](https://observablehq.com/) to get up to speed on Observable Notebooks---only to find that Observable had changed course. Though Observable Notebooks are still alive and well (and will, I suspect, [get even better](https://talk.observablehq.com/t/announcing-observable-2-0/8744/4), as of February, the company's primary focus has shifted to what they are calling [Observable Framework](https://observablehq.com/framework/). That led me down one final rabbit hole, but a very useful one, as I soon realized that Framework was a perfect fit for one of my fractional CTO client's "data science reporting and dashboard" needs. ## The Need The client in question is a non-profit doing great work in sustainable agriculture, helping farms take informed action based on soil data. The majority of their team are data scientists. Lots of data has been collected, and now the data team is charged with producing useful, coherent, actionable outputs for our users, who are mostly farmers. The data team works mostly in the [R language](https://www.r-project.org/), one of the primary languages of data science. R is widely used in agricultural data and particularly soil analysis. R is a data manipulation, aggregation, and analysis powerhouse, with plotting and charting libraries capable of a wide range of data visualizations. That said, R's natural mode of operation is one-on-one with the data scientist; output is more about "produce those charts we need for that scientific paper" than "create awesome reports and dashboards for our users." R can "do web stuff" in a number of ways, for example using the [Shiny package](https://shiny.posit.co/r/getstarted/shiny-basics/lesson1/index.html), but it feels a bit unnatural and bolted-on. The non-profit's primary user-facing (farmer-facing!) platform, on the other hand, sits solidly in the center of mainstream, modern web / mobile. Logic is JavaScript / TypeScript, running inside open source frameworks like Node.js, React and React Native, augmented by dozens more open source libraries filling niches like Object Relational Mapping (Drizzle). Data lives in the powerful, scalable Postgres open source database. All of this deploys gracefully onto modern AWS infrastructure, and the result is capable of supporting tens of thousands of web and mobile users. Costs are low---tooling costs almost zero---and the result has been highly effective for developers and enabled us to deliver an excellent web and mobile experience for our users. ## We Have Questions This left me asking a number of questions: - What is R really best at? Is it the data manipulation and analysis side or also data visualization? - Does R have enough advantages for our data science use cases to stick with it? - If R can't "do it all," should we move away from it? - Where is the line between "use R for this" and "this would probably better be handled with something else"? - If we don't use R for visualizations, what would we use instead? - Aren't we reinventing the wheel here---hasn't somebody already figured this out? Isn't there some kind of framework that at least provides a clean separation of concerns between data and presentation? ## Enter Observable Framework With these questions swirling around in my mind, it was a truly fortuitous moment when I landed on the Observable website and began to get my head around Observable Framework.Because, once I understood it, bang, my entire set of questions were answered, and the solution was beautiful. Thank you [Mike Bostock](https://en.wikipedia.org/wiki/Mike_Bostock) and [team](https://observablehq.com/about). Observable describes Framework as "solving data’s last mile problem." This is a reference to the telecom world where by far the most expensive part of their network is that last mile, connecting up the actual consumers. For data, Observable is saying, that last mile is getting data onto the screen of the user in need, in a way that is highly useful and actionable---and that this is the hardest problem to solve. Eureka! This is it! The last mile problem is exactly what my non-profit was struggling with. And wow, does Framework solve it elegantly. ## We Have Answers Let's look at the answers Framework gives us: - **Q:** Does R have enough advantages for our data science use cases to stick with it? - **A:** In soil science, it sure seems to. But, not to worry, you're free to mix and match---use whatever data science tool fits best in each situation---R, Python, Julia, TypeScript, whatever. [link to O blog post about data loaders] - **Q:** If R can't "do it all," should we move away from it? - **A:** R doesn't need to do it all---just use it wherever it's the best tool, which, in soil science, is frequently. - **Q:** Where is the line between "use R for this" and "this would probably better be handled with something else, e.g. mainstream web tech"? - **A:** Although R can create visualizations, its sweet spot is data. Framework's separation of concerns allows us to use R in its data sweet spot, while applying modern web technology for the presentation layer including visualizations. - **Q:** If we don't use R for visualizations, what would we use instead? - **A:** In Framework, the general answer for visualizations---and the presentation layer generally---is "use modern web technology." This consists of basic web building blocks---HTML, JavaScript, CSS, Markdown---augmented by any of the dozens of powerful visualization libraries that exist in the modern web, most free and open source. The cherry on top is that Observable's founders and team happen to have built two of the best viz libraries in existence---[details below](#presentation-layer-7-visualizations) when we dig into the presentation layer. - **Q:** Aren't we reinventing the wheel here---hasn't somebody already figured this out? Isn't there some kind of framework that at least provides a clean separation of concerns between data and presentation? - **A:** Yes, someone has figured it out! In Framework, Observable have an exceptionally elegant framework, with clean separation of concerns between data and presentation among its many benefits. ## Data Layer 1: Data Loaders In Framework, [Data Loaders](https://observablehq.com/blog/data-loaders-for-the-win) are the data layer, the first step towards getting data onto the screen, be it for a dashboard, report, or other data visualization. The data files created by Data Loaders power the reports and visualizations of the presentation layer that we explore [later](#presentation-layer-1-markdown). Data Loaders can be written the language of your choice: data science languages like R, Python, and Julia, but also JavaScript, TypeScript, Java, Rust, Go, PHP, and even shell scripts. This is huge---let data scientists and developers use their tool of choice, whatever best fits the need. Mix and match at will. > **Note:** Does loading data with a shell script sound crazy? You might be surprised. Here's a one-line Data Loader shell script that downloads recent earthquakes from the USGS in GeoJSON format: > > ```bash > curl https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson > ``` Thanks to Framework's clean separation of concerns and elegant design, Data Loaders are simple programs---just connect to your data, wherever it is---database, data warehouse, cloud APIs, or all of the above. Join, transform, reshape, and munge it to your heart's content. When done, simply dump the output, in almost any data format, to standard output. Framework automatically runs the Data Loader during the build process, intercepts its output and creates the output file, ready for use by the presentation layer. Since Data Loaders run at build time---rather than runtime in front of an impatient data consumer---there's less need to optimize code and queries. Shell script `curl` command a little slow? Who cares! Database indexes not optimized for this query? No worries! Cloud API introduce a lot of latency? No harm no foul! It's build time, not runtime---time is measured differently here. Keep it simple. Data Loaders really are simple---in developing our own proof of concept, I created five Data Loaders that connected to an RDS Postgres cluster and a pair of cloud APIs, producing both CSV and Apache Parquet files, in just a few hours. ## Data Layer 2: Static Generation, Dynamic Dashboards As you read this post, you're benefitting from "static site generation." The Crafty website is built using the static site generator [Hugo](https://gohugo.io/). There's no server runtime behind this site---everything's pre-generated at build time, including HTML for every page, optimized images for every window size, optimized CSS and JavaScript where needed. No server round-trips for anything, just the initial page load. Even dynamic functions like site search run locally in the browser via JavaScript. Navigate around a little, it's lightning-fast. The "lightning-fast" part seems to have appealed to the folks at Observable, who recognized that static site generation could work its magic for data dashboards as well---that "statically generated, lightning-fast" and "dynamic, interactive user experience" aren't mutually exclusive. There's a catch, though: lightning-fast user experiences can't happen if the data's slow. And slow is typical with traditional data warehouse + dataviz / BI tools. Executing complex queries in real time against remote data can take 10s of seconds and sometimes minutes---while the user waits impatiently. Well, guess what: those same Data Loaders whose elegant separation of concerns simplify the gathering of data for dynamic dashboards and reports, also solve the slow data problem. Data Loaders do the time-consuming work of querying and collecting data at **_build time_**. At **_runtime_**, the data is already **_pre-generated, cached, and ready to access in milliseconds_**. So thanks to static generation and Data Loaders, Framework is able to deliver a near-instantaneous interactive experience _for data_. ## Data Layer 3: How Fresh is Fresh Enough? You may still be having doubts about the idea of a data dashboard that isn't driven by live, real-time data. It's natural to conflate "dynamic, interactive data consumer experience," which we're all in favor of, with "querying live data from a remote database, data warehouse, or cloud API in real time," which isn't necessary most of the time and often ruins that interactive experience by making it slow. I love these two tips in Observable's excellent [Data Loader documentation](https://observablehq.com/framework/loaders), because they hit the nail on the head around "how fresh does my data need to be?" Tip 1: I'd paraphrase the first part of this tip as "Dude, if you really want up-to-the-second data, we gotcha covered!" The second part addresses a Framework nuance that I haven't yet mentioned: databases and data warehouses aren't the only source of essential data for dashboards and reports---often times static files from various external sources can be just as essential. For example, in our proof a concept, I needed to plot soil sample locations on maps, for both the entire U.S., and then for individual states. To get the country and state outlines needed as the backdrop for my plots, I just included a publicly-available, 842K static JSON file in my Framework project. The file uses a variant of GeoJSON called [TopoJSON](https://observablehq.com/framework/lib/topojson) and includes outlines of the U.S. itself, along with every state, and every county. [Observable Plot](https://observablehq.com/plot/), which we explore below, understands TopoJSON and easily produces the geographical plot needed. Tip 2: The second tip addresses a critical supporting tool for Data Loaders and data freshness in Framework: Continuous Deployment. In parallel with the tremendous progress made in static site generation through projects like Hugo, the tooling for Continuous Integration/Continuous Deployment (CI/CD) has radically improved and become ubiquitous. In fact, the two spaces intersect significantly: as an example, I rely on CI/CD tools provided as part of [Cloudflare Pages](https://pages.cloudflare.com/) to automatically redeploy this site, triggered when I push updates the `main` branch of my GitHub repo. CI/CD is all about builds, and as the tip tells us, Continuous Deployment combined with Framework Data Loaders means we can "rebuild data as often as you like, ensuring that data is always up-to-date." [GitHub Actions](https://github.com/features/actions) are probably the most widely-used CI/CD tool, and can automate [Framework builds](https://observablehq.com/framework/deploying) so they happen exactly when we need them too, whether that be time based (e.g. once an hour); event based (the new batch of sales data has been processed); or programmatically---any which way you need. ## Presentation Layer 1: Markdown The central element of Framework's presentation layer is Markdown, which [Observable's documentation](https://observablehq.com/framework/markdown) elegantly describes: > Markdown is a language for formatting text and content; it’s a lightweight, ergonomic alternative (and complement) to HTML. In a Framework project, each Markdown file represents one page, such as an interactive dashboard or report. Framework extends Markdown in [several powerful ways](https://observablehq.com/framework/markdown): > **Framework Markdown Features:** > > Markdown in Framework extends CommonMark with a handful of features useful for data apps, including reactive JavaScript, HTML, YAML front matter, grids, cards, and notes. > > - Use JavaScript to render charts, inputs, and other dynamic, interactive, and graphical content on the client. JavaScript in Markdown can be expressed either as fenced code blocks or inline expressions. You can also import JavaScript modules to share code across pages. > - You can write HTML directly into Markdown. HTML is useful for greater control over layout, say to use CSS grid for a responsive bento box layout in a dashboard, or adding an external stylesheet via a link element. > - The grid class declares a CSS grid container. The grid class is designed to pair with the card class and the dashboard theme for dashboard layout. > - The card class is used to group and delineate content. The card classes applies a background and border (with colors determined by the current theme). As we [explore later](#presentation-layer-5-reactivity), Framework's powerful Markdown extensions mean than each Markdown page is equivalent to a full data notebook---actually more powerful than Observable's own Notebooks, which themselves outshine most other such as Jupyter. ## Presentation Layer 2: JavaScript JavaScript in Framework Markdown is like water to fish---everywhere. [From Observable's docs](https://observablehq.com/framework/javascript): > **JavaScript Integration:** > > Use JavaScript to render charts, inputs, and other dynamic, interactive, and graphical content on the client. JavaScript in Markdown can be expressed either as [fenced code blocks](https://observablehq.com/framework/javascript#fenced-code-blocks) or [inline expressions](https://observablehq.com/framework/javascript#inline-expressions). You can also import JavaScript modules to share code across pages. > > - JavaScript fenced code blocks (```js) are typically used to display content such as charts and inputs. They can also be used to declare top-level variables, say to load data or declare helper functions. > - Inline expressions `$` interpolate values into Markdown. They are typically used to display numbers such as metrics, or to arrange visual elements such as charts into rich HTML layouts. Think of a Framework Markdown page as being wrapped in a JavaScript context, what Observable calls the runtime---not just the ability to run JavaScript code basically anywhere, but also with our full data context available to operate on and visualize. ## Presentation Layer 3: Getting the Data How do we get the data, for example data snapshots created by Data Loaders, available to the presentation layer and accessible via JavaScript? From the [Observable docs](https://observablehq.com/framework/files): > Load files — whether static or generated dynamically by a data loader — using the built-in `FileAttachment` function. This is available by default in Markdown. Here's a Markdown code block to load a JSON file listing active volcanos: ```js const volcano = FileAttachment("volcano.json").json(); ``` When the variable `volcano` is referenced in another code block or inline expression, for example in a visualization, it contains the appropriate JavaScript Array, Object, etc. to represent the JSON data. Two magical Framework notes: - The presentation layer doesn't need to know if the file `volcano.json` was generated by a Data Loader, or was simply included in our project as a static file---Framework manages all of this at build time for us. Specifically, Framework build looks for the file `volcano.json`, but if it's not missing, looks for a Data Loader based on the same name, say `volcano.json.py`, and runs it to generate the data snapshot. - Thanks to [reactivity](#presentation-layer-5-reactivity), even though FileAttachment is an async function since it's reading data from the file system, we don't need to deal with async logic---the Framework runtime manages this for us automatically. Here's a full list of formats that `FileAttachment` supports: ## Presentation Layer 4: Client-side SQL There's one more totally amazing way to access data in the presentation layer: client-side SQL. From the Framework [docs](https://observablehq.com/framework/sql): > Framework includes built-in support for client-side SQL powered by DuckDB. You can use SQL to query data from CSV, TSV, JSON, Apache Arrow, Apache Parquet, and DuckDB database files, which can either be static or generated by data loaders. To use SQL, you first register a SQL source in the Markdown doc's front matter. Here we register a sample Gaia astronomical data source which happens to be in Apache Parquet format: ``` --- sql: gaia: ./data/gaia-sample.parquet --- ``` The same build-time Data Loader logic we described above applies here---if the file `gaia-sample.parquet` doesn't exist, Framework build look for a matching Data Loader such as `gaia-sample.parquet.py` and run it to create the file. Now we can query that source from a SQL code block: ```sql id=top10 SELECT * FROM gaia ORDER BY phot_g_mean_mag LIMIT 10 ``` The `id=top10` stashes the result of the query in a JavaScript variable: > The value of a SQL code block is an Apache Arrow table. This format is supported by [Observable Plot](#presentation-layer-7-visualizations), so you can use SQL and Plot together to visualize data. So, right in the browser, we're running live, superfast SQL queries on our data---no server roundtrips and near-instantaneous responsiveness. When paired with Framework [reactivity](#presentation-layer-5-reactivity) and [interactivity](#presentation-layer-6-interactivity), client-side SQL is a huge win for enabling blazing-fast interactive dashboards. (Framework's [strong Apache Parquet support](https://observablehq.com/framework/lib/arrow#apache-parquet) is a related win. Compared with other data formats, Parquet is very small, very fast, and works great with Framework's DuckDB-powered SQL.) ## Presentation Layer 5: Reactivity JavaScript in Framework has a superpower: it's reactive. From the [docs](https://observablehq.com/framework/reactivity): > **Reactivity Benefits:** > > Framework runs like a spreadsheet: code re-runs automatically when referenced variables change. This brings: > > - Easier interactivity because state is automatically kept in sync > - Easier asynchronous programming via implicit await of promises > - Better performance with incremental re-rendering > - Greater flexibility by writing code and prose in any order > > Reactivity is especially helpful for data apps because these apps tend to have complex state. Reactivity was a key differentiating feature of Observable's first product, Notebooks. In most data notebooks, Jupyter for example, cells execute in a linear order, and the user must manually re-run cells when dependencies change. In contrast, an [Observable Notebook](https://observablehq.com/@observablehq/learning-observable-reactive-dataflow) ... > ... knows which cells depend on which others, no matter where they are located in the notebook. So when one of them changes, all values that depend on it are automatically updated. This is also efficient: only those cells that are affected by a change are re-run. That means that you can organize cells in your notebook however you want, without worrying about how the order of operations may be affected. Reactivity is even better in Framework, because the augmented Markdown format that sits at the heart of Framework's presentation layer is more powerful and flexible than the original app-based Notebook model. Framework Markdown freely combines content with code; uses vanilla JavaScript rather than a JavaScript dialect with [syntax differences](https://observablehq.com/documentation/cells/observable-javascript); and finally, since a Framework page (the equivalent of a Notebook) is represented by a single, simple text Markdown file, it works flawlessly with source control / CI-CD platforms like GitHub. ## Presentation Layer 6: Interactivity Reactivity brings many benefits, but perhaps shines most brightly when it comes to enabling interactive dashboards, where user inputs dynamically reshape dashboard output. Framework Inputs support direct user interaction on dashboards and reports. The 14 flavors of Input include Radio, Checkbox, Select, Range, Toggle, Date and eight more. Let's make a histogram of soil carbon values, with the width of each histogram bar (and therefore the number of bars) controlled by a Range Input. This code block with a single line of code: ```js const binWidth = view(Inputs.range([.2, 1], )) ``` puts a range slider control onto our dashboard screen, with min/max values of .2 and 1. The JavaScript variable `binWidth` has an initial value of .5, and gets updated as the user moves the slider left or right. In another code block, Observable Plot's binX transform references `binWidth`, and thanks to Framework reactivity, the histogram updates in real time. Here's the result: Voila, one line of code, and we've got our first interactive dashboard element. Now let's take a look at a more complete and interactive dashboard ([try it live here](/observable/mortgage-rates)), which shows 15- and 30-year fixed mortgage rates from 1971 through the present, using data that is updated weekly and thoughtfully [shared by Freddie Mac](https://www.freddiemac.com/pmms/pmms_archives) as a downloadable CSV. A 15-line Node.js JavaScript Data Loader grabs the latest CSV, current to within a week. As you see below, the upper right card shows a detailed chart of mortgage rates, starting with the past year; while the bottom pane shows a full-history rate chart that covers 1971 through the present. In this case, the interaction isn't through a Framework Input, but rather through JavaScript click and drag events setting a date range variable `startEnd` when the user interacts with the lower chart. The detailed chart above right references `startEnd` and reacts instantly when it changes. All four charts in the dashboard are rendered by Observable Plot. As you can see, Framework supports highly-interactive experiences for data consumers. ## Presentation Layer 7: Visualizations So far, we've covered the data layer including a deep dive on Data Loaders, and many aspects of the presentation layer, from Markdown to interactivity. But what about the visualizations themselves? Here Framework's options are virtually unlimited: virtually any JavaScript visualization library can be used within Framework. To insert a visualization into a Framework Markdown page, you just call the library from a JavaScript code block or inline expression. Observable themselves actually created two of the most powerful libraries (both fully open source): [Observable Plot](https://observablehq.com/plot/), which is built on top of the lower-level [D3](https://d3js.org/). Just to give you a taste of what's possible, here's a tiny subset of the [available examples for Plot](https://observablehq.com/@observablehq/plot-gallery) and for [D3](https://observablehq.com/@d3/gallery): But Observable's own libraries are far from the only options---other widely-used general plotting libraries including [Apache ECharts](https://echarts.apache.org/en/index.html), [Plotly](https://plotly.com/javascript/), and [Vega-Lite](https://vega.github.io/vega-lite/) work great within Framework. Mapping-specific libraries such as [Leaflet](https://leafletjs.com/) and [Mapbox GL JS](https://docs.mapbox.com/mapbox-gl-js/guides/) are fully supported, as are a number of specialized viz libraries such as [DOT](https://observablehq.com/framework/lib/dot), [Deck.gl](https://observablehq.com/framework/lib/deckgl), [Mermaid](https://observablehq.com/framework/lib/mermaid), and [Mosaic vgplot](https://observablehq.com/framework/lib/mosaic). ## Framework in a Nutshell Observable's homepage tagline is: > The best dashboards are built with code. If one considers the conventional alternatives---a massive SaaS app along the lines of Tableau, where our visualization possibilities are limited by what the SaaS vendor decides we should have---I'll take dashboards built with code, thank you very much. --- ## A Quick Look at GPT-4o **URL:** https://cto4.ai/p/gpt-4o-quick-benchmark **Date:** 2024-05-20 **Type:** brief **Description:** I did a quick benchmark of the new GPT-4o model versus GPT-4-turbo. Roughly twice the speed, with improved Vision accuracy. Since last week's [Spring Update event](https://openai.com/index/spring-update/) from OpenAI, I've been wanting to test the new [Omni model (GPT-4o)](https://openai.com/index/hello-gpt-4o/) in my Vision-based [recipe ingestor project](/p/chatgpt-4v). This Python script processes a directory of recipe images, and using GPT-4's Vision capabilities, as well as JSON Mode, produces a structured JSON recipe that I can post directly to my Paprika Recipes app. ## Benchmark Results For a batch of three recipes / four recipe images, bypassing uploading to the Paprika cloud. ### Quantitative - GPT-4-turbo: 1:03 (1 min 3 secs) - GPT-4o: 0:38 (38 secs) ### Qualitative GPT-4o's Vision capabilities seem to have improved over GPT-4-turbo. Example: GPT-4-turbo had a hard time parsing the handwritten name "Debbie Gardner", producing "Betty's Cards", "Betty Bard", "Betty Borden" etc. on different runs. GPT-4o got the name correct. --- ## Using Tailscale to Access Amazon VPCs, EC2 Instances, and RDS Clusters **URL:** https://cto4.ai/p/tailscale-fun **Date:** 2024-04-23 **Type:** essay **Description:** Tailscale has been simple to set up and manage, but also amazingly flexible. ## Inception: Direct Database Access for the Data Team Our immediate need was getting the data science team programatic access to a read replica of our production database, an Amazon RDS Postgres cluster. The initial stopgap solution was to open a Postgres port, but with an RDS Security Group rule that limited inbound access to a few specific IP addresses. That solution became high-maintenance quickly, since some of our team connected from a university campus network, where IP addresses rotated every few weeks. ## Better Idea: AWS Systems Manager Session Manager Our development partner suggested a better solution, AWS [Systems Manager Session Manager](https://aws.amazon.com/systems-manager/features/#Session_Manager), which enables tunneled sessions into the AWS environment and leverages AWS IAM to manage access. The result: better security and no need for manual IP whitelisting. SMSM has one significant drawback: targeted at dev and devops folks, it is configured and run through the AWS command line interface. Yikes---imagine conversations with end users that begin "OK, first, open a terminal, then run this command ..."---not the user experience we're going for. Our data team could've handled it, but the CLI interface would be an ongoing source friction and pain. More Session Manager rough edges appeared when I tried but failed to set up a quick proof of concept. The AWS docs were impenetrable, and even a third-party guide on the process wasn't help enough. All the time, I was becoming less and less convinced that Systems Manager was the best solution: "All this mucking around, just to open a tunnel to an RDS replica?" I also began to realize that Session Manager was, in effect, a limited-scope VPN for AWS services only, without much of a user interface. Why invest precious time on such a limited solution? ## Bigger Picture: Other Needs When I broadened my thinking, I quickly realized we had already implemented two other one-off remote access solutions: - To enable SQL GUI tool access to the database, we set up bastion servers on our staging and production VPCs. These were Windows EC2 instances, with their own login credentials and the DBeaver SQL tool installed. We attached to the bastions using Remote Desktop. Scrolling through long tables in DBeaver via Remote Desktop is unpleasant bordering on dangerous on a write-capable connection. - To support miscellaneous tasks such as database migration, we had two additional Ubuntu EC2 instances, with open SSH ports. With Systems Manager Session Manager shaping up as yet another one-off remote access solution, I decided to seek a broader solution that addressed all three remote access needs, provided a better admin and user experience, and shored up security as well. ## Security Goals Our desired security posture on AWS is "expose nothing we don't absolutely need to expose, and what we do expose, make it robustly secure." To drill down a bit: - Keep our AWS Virtual Private Clouds walled off and private. - Avoid exposing attack surfaces, such as SSH / Postgres ports and internal-only service endpoints. - Enable remote access in a robustly secure manner, granting specific users access to specific services, as opposed to blanket access for everyone. ## VPN---or Tailscale? The default solution for secure remote access is a VPN; and there's no doubt a VPN could have worked for us. "Easy," "Low Maintenance" and "Flexible" aren't terms one usually associates with VPNs, however, so I kept solution-scanning. I'd been peripherally aware of [Tailscale](https://tailscale.com/) through background chatter on Hacker News and similar venues. My vague impression of its niche was "better than a traditional VPN, and way less painful." That sounded just perfect for our needs, so I took the opportunity to explore it more deeply, was immediately intrigued, found wonderful documentation including excellent setup guides for exactly what we needed, and before I knew it had a Tailscale-on-AWS proof of concept up and running. ## What is Tailscale, Really? The very minute I sat down to write this section, Tailscale's monthly email newsletter landed in my inbox, and it highlighted [this excellent Tailscale post by Casey Liss](https://www.caseyliss.com/2024/3/27/tailscale). Casey operates the [Accidental Tech Podcast](https://atp.fm/), and has several iOS apps in the App Store including the excellent [Callsheet](https://apps.apple.com/us/app/callsheet-find-cast-crew/id1672356376) (4.9 star rating, App Store Editors Choice). So Casey already wrote it better than I could, so to get up to speed on Tailscale basics, please [read his!](https://www.caseyliss.com/2024/3/27/tailscale) ## Getting Inside the VPC VPC means Virtual **_Private_** Cloud, so how do we get in there? Enabling all sort of "getting in there," in a magically simple way, is Tailscale's superpower. Still, AWS is pretty serious about the P word, and [Tailscale did the work](https://tailscale.com/kb/1296/aws-reference-architecture) to make it really work. Getting Tailscale running on an EC2 instance, bare cloud-iron that we can directly access and control, is just a quick install. Literally a minute if one can SSH in. In one minute, that EC2 instance will magically pop up on your private Tailnet. (Installing Tailscale on Mac, Windows, iOS, etc. is even easier.) But ... we're living in the age of Serverless now. Even in our smallish infrastructure, the two main pillars of our setup (compute and database, ECS-Fargate and RDS Postgres) are serverless. What do you do when you don't have bare iron, when you can't just SSH in and install Tailscale? The key here is Tailscale's subnet router, which isn't even another piece of software, but rather an argument added to the Tailscale command when you start it up. You just need a tiny EC2 instance within each VPC running Tailscale with the subnet router option enabled, and now, every device connected the VPC becomes accessible, even though they aren't even running the Tailscale client. (To be clear, Security Groups still apply so access is blocked by default.) Subnet routing gets us access to our RDS cluster, both primary writer and read replica instances. And that (along with Tailscale's robust security) meets two of our three current use cases: data team read replica access, and developer access for SQL tools. Bastion servers are gone, and DBeaver is 10X better connecting directly versus through Remote Desktop. ## SSH, Solved Our third use case is SSH. Good news: just by installing Tailscale on the servers that need SSH access, and the SSH users' client devices, we have closed off the attack surface exposed by opening an SSH port to the Internet. As [Tailscale's docs state](https://tailscale.com/kb/1193/tailscale-ssh), we already have "the standard SSH experience without exposing your servers to the internet." That's a big, free win right there. We've achieved this without even implementing the [Tailscale SSH](https://tailscale.com/tailscale-ssh) feature ... so why does that feature even exist? It turns out there's more to robustly securing SSH than simply blocking network access. There's authentication---"who are you and how do I know you're telling the truth about that?"---and authorization---"what's this user allowed to do?"---namely are they allowed SSH access to this server? We all know the pitfalls of username/password security, and organizations that are serious about security (or have compliance requirements) tend to instead apply digital certificates. To quote Tailscale's docs: > Historically, to secure an SSH connection, you generate a keypair on the machine you are connecting from (known as the client), with the private key stored on the client, and the public key distributed to the device you want to connect to (known as the server). This lets the server authenticate communication from the client. So every SSH user needs to establish a keypair with every server they SSH access to. When you've got more than a handful of SSH users and servers, this gets painful fast, and opens up its own attack surfaces, such as the case where an SSH-privileged user leaves the company. The pain level and security/compliance concerns here are often large enough to drive the implementation of PKI (Public Key Infrastructure) and KMS (Key Management System) solutions. In other words, a whole new set of systems just to get secure, compliant SSH. This is where Tailscale SSH shines. Tailscale already knows, through its underlying digital certificate implementation, who its users and machines are. Tailscale SSH simply applies Tailscale's own strong authentication and key management capabilities to SSH transactions, adding an SSH-specific Access Control List facility to cover authorization, e.g. "what's this user allowed to do?" This eliminates the whole every-user-to-every-server keypair rat race while improving security and compliance. And there are [many other benefits](https://tailscale.com/tailscale-ssh) to Tailscale SSH, from automatic key rotation to the SSH session recording. ## Tricks, Tips, and Learning Moments ### A Dedicated Tailscale AWS Proof of Concept Environment Is Worth It In an attempt to save time, I tried to do my Tailscale POC in our existing Staging environment. The predictable result: I actually spent more time chasing little glitches caused by unique aspects of the preexisting environment. Another result: one of my experimental changes managed to take Staging down. Eventually, I learned my lesson, went back and set up a full POC on its own VPC with its own RDS cluster and all. Getting Tailscale set up fully there was quick, and by getting that environment working, I learned what I needed to (carefully) reimplement Tailscale on Staging, and then Production. ### Duplicate IPv4 CIDR Blocks on Different VPCs We used Terraform to set up our initial Staging and Production environments. One side effect of this was that the IPv4 address ranges (CIDR blocks) were identical between the two environments. This all works just fine inside the private VPCs but when Tailscale needs to route traffic from the outside, it needs to know "which 10.0.1.23 do you mean, sir?" Tailscale's subnet router includes an elegant fix for this that they call [4via6](https://tailscale.com/kb/1201/4via6-subnets). We configured our Staging environment this way. ### Bad Tailscale Advice from GPT-4 I adore and rely on paid ChatGPT for help and advice across many knowledge domains with great success. Its specific advice around the details of Tailscale setup on AWS, however, turned out to be faulty and sent me off in the wrong direction several times. I think this knowledge niche was small and specific enough that ChatGPT's training data wasn't up to the task. My own lack of knowledge probably had me asking the wrong questions, and prevented me from recognizing bad advice. Filed away as a caution for similar situations in the future ... ### Tailscale With Pihole and Unbound I use [Pi-hole](https://pi-hole.net/) on my home network, for network-level ad and tracker blocking; it's awesome, as I always realize when I'm away from home and see what I've been "missing"! I also have been using [Unbound](https://docs.pi-hole.net/guides/dns/unbound/), a recursive DSN server, running beside Pi-hole on the same Raspberry Pi. Unbound is faster and more secure than using even one of the fast public DNS services like Cloudflare DNS. Something about how Unbound works, however, makes Tailscale---which also needs to be smart about DNS resolution---unhappy. This took quite a few hours to diagnose. I suspect it's fixable, but haven't had time to spend on it, and for now have Unbound disabled. ### Tailscale Exit Nodes Speaking of the poor Internet experience I get when away from home, Tailscale has a nice fix for that, [Exit Nodes](https://tailscale.com/kb/1103/exit-nodes), which can allow me to always route my internet traffic through my well-protected home network. Another thing I haven't had time to implement, but it's on the list. --- ## Russ Cox's XZ Timeline **URL:** https://cto4.ai/p/russ-cox-xz-timeline **Date:** 2024-04-02 **Type:** elsewhere **Description:** Nice writeup from Russ Cox on the (incredibly long) timeline of the XZ backdoor Fascinating---and terrifying. Cox includes a solid list of Further Reading links at the end of his post. --- ## Simon Willison's LLM Tool: Now I Have 50 LLMs **URL:** https://cto4.ai/p/50-llms **Date:** 2024-03-25 **Type:** brief **Description:** I updated my installation of Willison's LLM tool to add plugins, and now I have 50 LLMs at my fingertips, including 15 local models, which get installed on demand. I've been intending to get my feet wet with the [new Claude models](https://www.anthropic.com/news/claude-3-family) from Anthropic, especially Opus. When I saw Simon's [recent "sidequests" post](/p/two-from-willison) I decided to update to the latest version of his [LLM CLI tool](https://llm.datasette.io/en/stable/) (part of [Datasette](https://datasette.io/)) and get Opus access from the command line, using the Claude plugin. While I was at it, I connected Google Gemini, Anyscale Endpoints, and gpt4all, the latter providing the abovementioned 15 local models, which auto-install when you first request a model. 50 LLMs in all---and this is far from the [complete list](https://llm.datasette.io/en/stable/plugins/directory.html) LLM provides access to through its plugin architecture. Very handy for testing, such as "how does Llama 70b compare to ChatGPT GPT-4 on this question?" --- ## Two Worthwhile Reads From Simon Willison **URL:** https://cto4.ai/p/two-from-willison **Date:** 2024-03-25 **Type:** elsewhere **Description:** Claude 3 Opus and GPT-4 to Tackle a GIS 'Sidequest'; and Getting GPT-4 to Write, Compile, and Run C Code I subscribe to Simon Willison's email newsletter (really just selected blog posts via email), and his last two were excellent. Last week he talked about his casual daily use of the new Claude 3 Opus model via his own LLM tool, as well as ChatGPT GPT-4. He [describes such use as "sidequests,"](https://simonwillison.net/2024/Mar/22/claude-and-chatgpt-case-study/) a description that resonates with me and my own daily LLM use. In this example, over an elapsed time of about 6 minutes, he was able to get an accurate GeoJSON for New York State's huge Adirondack Park starting with a deeply strange shapefile. Over the weekend, he shared a [longer prototyping effort](https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/) using ChatGPT GPT-4 Code Interpreter, where we managed to get GPT-4 to write, compile, and run a non-trivial bit of C code: a SQLite C extension for vector similarity. Code Interpreter runs Python, not C, but Simon suspected and confirmed that he could get GPT-4 to run `gcc` from within Python, and that was enough of a shim to get the job done. Both are excellent reads that mirror my own experience in terms of the myriad ways I take advantage of LLM tools. --- ## Moxie on Murder **URL:** https://cto4.ai/p/moxie-on-murder **Date:** 2024-03-12 **Type:** brief **Description:** I did a double-take when I saw Moxie Marlinspike in the credits as co-starring in A Murder at the End of the World episode 6 (SLIGHT SPOILERS WARNING) [A Murder at the End of the World](https://www.themoviedb.org/tv/134095-a-murder-at-the-end-of-the-world?language=en-US) grew on me over the course of the season, and by seasons end it had reached "definitely watch" status. Not the highest possible status, "Can It Be Tuesday Already Please?" which is, at the moment, held by Hulu's new [Shōgun series](https://www.themoviedb.org/tv/126308-shogun?language=en-US). I rarely watch credits; sometimes to savor a great season finale, sometimes if there's interesting music I want to hear out. With episode 6, it was a combination: an intense episode finish, building towards the season finale in episode 7; and an excellent electric guitar instrumental, [Beni Said by 75 Dollar Bill](https://music.apple.com/us/album/beni-said/1120597959?i=1120598579), that began playing with the credits. For whatever reason, I was still watching the credits by the time they reached CO-STARRING, 1:51 from the end of the episode. And there, listed first: Moxie Marlinspike. Moxie's a [legend in tech](https://en.wikipedia.org/wiki/Moxie_Marlinspike) as creator of the [Signal secure messaging app](https://en.wikipedia.org/wiki/Signal_%28messaging_app%29)---and more importantly to cryptographers, the [Signal Protocol](https://en.wikipedia.org/wiki/Signal_Protocol) and [Double Ratchet Algorithm](https://en.wikipedia.org/wiki/Double_Ratchet_Algorithm)---and when I saw his name I immediately was looking for "technical advisor" credits. But no, this was actually CO-STARRING. I filed it away and determined to research it. The following day I did a bit of searching, with little to show for it other than IMDB stating that yes, Moxie had indeed appeared in S1 E6. I refined the query a bit and finally got found a few some 2023 mentions of Moxie technical advising as well. I use [Kagi search](https://help.kagi.com/kagi/company/), and beyond the core fact that, in stark contrast with Google, Kagi doesn't suck, I've come to value its "Quick Answer" feature, which feeds search results through an LLM and generates an excellent summary with references. Here it is, wrapping up both aspects of Moxie's contribution to Murder at the End of the World: So I was right about the technical advisor thing; hacking is a significant part of Murder's story, and it was accurately presented throughout the series. The only major tech flub I saw was the data center fire scene in episode 7. Maybe Moxie's advisory contract had run out by then, or maybe they had a mini-GOT-gotta-finish-this-somehow brain cramp. But no need to go there, the final episode was still solid despite painfully stretched credibility in that scene. I've never seen Moxie in person or even on video; maybe in a Wikipedia photo. So I had no clue that was him playing Max in episode 6. But I'm happy I noticed him in the credits and did this research. --- ## Willison: The killer app of Gemini Pro 1.5 is video **URL:** https://cto4.ai/p/willison-gemini-pro-video **Date:** 2024-02-22 **Type:** elsewhere **Description:** Simon Willison tries out Gemini Pro 1.5 on video, and suggests its 1M token context size opens up powerful new opportunities using video prompts Willison's experience with, and reaction to, Gemini 1.5 Pro extracting structured output from video prompts parallels [my own experience](/p/chatgpt-4v) using GPT-4 Vision to extract structure from heirloom recipe images (often handwritten and horribly mangled): > ... I'm pretty astonished by this. > > ... I find those results pretty astounding. > > The ability to analyze video like this feels SO powerful. Being able to take a 20 second video of a bookshelf and get back a JSON array of those books is just the first thing I thought to try. --- ## Using Multimodal AI to Capture and Enrich Heirloom Recipes **URL:** https://cto4.ai/p/chatgpt-4v **Date:** 2024-02-19 **Type:** essay **Description:** I applied OpenAI's GPT-4 Vision model and Chat Completions API to preserve a treasure-trove of legacy family recipes. **Tags:** OpenAI, GPT-4 Vision, Recipe Management, Automation, Python **Categories:** AI & Technology OpenAI is constantly [firehosing us with new capabilities](https://openai.com/sora). It's tough just to keep up with the announcements, let alone deeply understand the new offerings at depth. The only way I know to develop this kind of deep understanding is to build something real and non-trivial. I was fortunate to have a somewhat complex personal need, just as the new GPT-4 Vision model and updated Chat Completions API landed in my OpenAI account. The tale of my exploration follows. _Note---Click any image to zoom_ ## Sept. 25th: "ChatGPT can now see, hear, and speak ..." OpenAI had teased vision support earlier, but the actual [announcement](https://openai.com/blog/chatgpt-can-now-see-hear-and-speak) of Vision availability happened on Sept. 25th under the above headline. I'm not an OpenAI VIP, sorry. I'm just a paid ChatGPT Plus user and Tier-1-going-on-Tier-2 API user. I don't get early access; I get _late_ access. So while OpenAI announced GPT-4 Vision on Sept 25th, it didn't show up for me until late October or early November. OpenAI doesn't communicate actual availability of features to its retail customers, and so all I know for sure is that I noticed Vision had showed up in my ChatGPT+ sometime in early November. ## My burning need: family heirloom recipes During our August vacation at the family lake house, we uncovered a treasure-trove of recipes, which my daughter-in-law Julie was kind enough to capture using the iOS Notes app and share in PDF format. On my to-do list since then had been figuring out how to get those recipes into a recipe app. I use Paprika 3, while Julie has been trying out Recipe Keeper. What the family really needs is both a great facsimile of the original paper recipe (high resolution, correct orientation, zoomable), and a fully-digital recipe with all the goodies that apps like Paprika and Recipe Keeper give you---ingredient scaling, store lists, easy sharing, and so on. Paprika has been around forever, and several years back I had tried importing legacy recipe images, but at that point in time I wasn't able to get the "high resolution, correct orientation, zoomable" solution desired. So I was wondering if Julie had any luck with Recipe Keeper and I began poking around the current Paprika. In the end it turned out that the current version of Paprika 3 has quite strong image support, while in limited testing, I had trouble getting Recipe Keep to handle image orientations correctly. So the recipe app I was already using looked to be the best solution. Decision made. We still had a lot of work ahead of us, though---laboriously transcribing from those torn, stained, mostly handwritten family treasures. And this particular trove is probably only 20% of the full library; my brother has the larger collection, probably at least 3X as many, and my sister and wife have others. This was shaping up to be a long side project. ## Nov. 14th: Recipe image => ChatGPT Plus with Vision So when I discovered that my ChatGPT Plus account now had the "new stuff" including vision support, I had a ready-made test case for Vision. I know for sure that it was November 14th when I began toying with it. My first (failed) Vision request was: This first request caused GPT-4V to puke---PDFs aren't considered an image format. I used macOS Preview to export a png file from another one my recipe PDFs and fed that to Vision. This was the input: I fully expected an interesting failure. I've been doing technology for over four decades, which coincidentally is just about how long OCR---optical character recognition---has been available for personal use. And OCR has sucked for all of those decades. OCRing scans of printed documents is painful and error-prone. But with our legacy recipes, we faced much worse, like Margie Johnson's Cheesecake above---typed with a worn ribbon, wrinkled, stained. My expectations were low, but on the other hand, GPT-4 blows my mind several times a week, so what the heck, let's see what happens. ## OCR is dead, long live Vision This was GPT-4V's response to that image: I was expecting "maybe a little better than your grandma's OCR" but this was shockingly far beyond that. Not only did 4V get almost everything correct from the Margie recipe, worn ribbon, wrinkles, stains and everything, but GPT-4V _understood_ it and organized it for me. For example, 4V correctly "gets it" that this recipe had three main elements---crust, filling, and topping---when the recipe never uses any of those words! And then helpfully organizes its output that way. With traditional OCR, at best I'd get most of the words, in the order they appear on the page. But this recipe isn't plain text that reads left-to-right, top-to-bottom. Take for example the initial crust block that uses the typist's trick of using a column of ")" characters to create a column separator: ingredients in the left column, instructions in the right. OCR would just give me a useless left-to-right, top-to-bottom character stream; but GPT-4V _understood_ what it was _seeing_ and placed the directions below the ingredients in a structured manner. Holy shit. ## "Guys? Guys? This has structure. I'm [seeing] structure." I love the scene in the movie [Contact](https://www.themoviedb.org/movie/686-contact?language=en-US) where Kent, the blind SETI researcher, says: > Guys? Guys? Um... > > You know those interlaced frames that we thought were noise? > > This has structure. I'm hearing structure. Seeing that chat response, it wasn't just that I was getting better text than old-school OCR. GPT-4V was giving me structure, without my even asking. No surprise that my next question was: > Is there a standard interchange format for recipes? Oh ho, I see Paprika on the list! ## Dec. 17th: Image => ChatGPT-4V => YAML => Paprika import I explored the various file formats that Paprika could import. Most of these were specific to what other recipe apps could export, to enable migration onto Paprika. But Paprika also supports a more generalized import from YAML format, and provided a couple of helpful examples. That looked like my best option, and then began the work of convincing ChatGPT to take a recipe image, and instead of giving me semi-structured text output, to instead produce Paprika-flavored YAML. It took a bit of research and several tries, but eventually I was able to feed in an image and get Paprika-compatible recipe YAML back. Here's the source image and YAML output: And here's what it looks like when imported into Paprika, iPad and iPhone versions: Now, ChatGPT was doing most of the work---image in, importable YAML out. That left me light years ahead of the mess I had been facing---laborious manual transcription or parsing garbled OCR output. ## Late December: Automating in Python Despite the impressive gains compared with the time I would have spent manually transcribing or old-school OCRing, interactive-chatting my way through hundreds of recipes was going to be quite time-consuming. ChatGPT also has a tendency to "lose the thread" on long-running chats. Finally, though Paprika is capable of importing YAML files with an embedded BASE64 image, it was beyond ChatGPT's capabilities to produce that kind of YAML output. As a result, I had to first import the generated YAML into Paprika, then manually add photos. So automating the process was looking attractive---eliminate interactive sessions in favor of directly scripting calls to the OpenAI API. I picked Python as a logical choice to script in, figuring it'd have libraries for anything needed, and because GPT-4 is very good at Python coding; I was not disappointed on either front. Unfortunately I didn't have much choice about which OpenAI API I'd need to use. Back then---and still to this day---the only API that supports GPT-4 Vision is a feature-reduced Chat Completions API: This limited Chat Completions API available with Vision models is fine for simply automating something you've been doing using the chat interface; but it's significantly less powerful than the full-strength Chat Completions API available for text input, let alone the new Assistants API. More on this later. In any case, off I went scripting Chat Completions from Python: A few days before Christmas I had Python successfully scripting images => Chat Completions API => YAML output, and soon after added a BASE64-encoded image to the YAML output. I also added directory processing, so the script would run through an entire directory of image files and produce YAML-with-image output for each recipe, in a single script run. ## Early January: From YAML import to JSON cloud sync More than a few of our heirloom recipes had multiple images---for example the front and back of a recipe card---and while GPT-4 Vision handled multiple images perfectly, I never had any success getting Paprika to ingest more than one image from a YAML file. In searching for a solution, I discovered a whole community of recipe hobbyists who loved the Paprika app and found ways to creatively use Paprika's cloud sync API to build useful tools like a recipe exchanger that synced across multiple Paprika accounts. Paprika's cloud sync is the native interface of the app itself, and so by definition supports all the app's capabilities, including multiple photos per recipe, so it was able address the apparent limitations of YAML image importing. Publishing direct-to-cloud also saved the manual YAML import step, another key win. I dug in and by New Years day had cloud sync operating, and had multiple photos loading into Paprika soon after: ## Later January: JSON Mode to overcome structured output fragility The Chat Completions API, paired with the `gpt-4-vision-preview` model, is wonderful in its multimodal-ness. Unfortunately, this pairing is significantly restricted compared to the same API paired with the latest `gpt-4-turbo-preview` model. The even-more-powerful Assistants API currently doesn't support image input at all. What appears to have happened is this. OpenAI wanted to get the GPT-4 Vision model, announced on Sept. 25th, into the field as early as possible. But OpenAI likewise wanted to get the significantly updated GPT-4 Turbo model with corresponding Chat Completions API updates, along with the all-new Assistants API, into developers' hands ASAP also, and did so at the [DevDay event on Nov. 6th](https://openai.com/blog/new-models-and-developer-products-announced-at-devday). But full integration of Vision into these updated and new APIs wasn't (and still isn't) ready. OpenAI's compromise solution is what we have at the moment. Why do I care about the new capabilities launched at DevDay? My immediate problem is that the Chat Completions API with Vision isn't 100% reliable at producing structured output like YAML or JSON. It usually works, but sometimes doesn't, depending on the content fed to it. This leads to a constant prompt engineering cycle, with prompts constantly being tweaked to avoid output breakage. At the [DevDay event on Nov. 6th](https://openai.com/blog/new-models-and-developer-products-announced-at-devday), OpenAI announced JSON Mode, where using the `response_format` parameter, you tell the model to "constrain its output to generate a syntactically correct JSON object." That's exactly the solution I need to get higher reliability structured output and greatly reduce the need for constant prompt engineering. Longer term, I'm convinced that the other new features announced on Nov. 6th, including Function Calling and Tools, will open up a whole new range of possibilities in my recipe ingestion pipeline. Check out Max Woolf's post [Pushing ChatGPT's Structured Data Support To Its Limits](https://minimaxir.com/2023/12/chatgpt-structured-data/) to get a feel for what's possible with structured data, function calling, etc. My interim solution is to use both models: - Use the limited Chat Completions API paired with the `gpt-4-vision-preview` model to extract the recipe content - Feed that content (text!) into the more powerful pairing of Chat Completions API paired with the `gpt-4-turbo-preview` model new API so I can take advantage of JSON Mode I know, though, that I'm discarding a tremendous store of context by bridging between models using only text. As soon as OpenAI fully integrates Vision into its updated and new APIs, I'll rid myself of this compromise and start looking at how I might apply function calling, tools, etc. ## Enrichment You may have noticed that the final recipes showing up in Paprika have been significantly enriched. Take this before-and-after example _(click image to zoom)_: This a _tour de force_ of GPT-4V's content extraction: it correctly parses both typewritten and handwritten content, and likewise recognizes that these are two separate recipes. I end up with JSON content ready to feed into Paprika. But look closer---the original image doesn't have any directions---how did that get into Paprika? And what's this Description section? And those notes? That's enrichment in action. ### Whimsical enrichment If you've spent time with ChatGPT, you know that it loves to be creative, to cheerfully, energetically overdeliver and over-explain. Early in my process here, back when I was doing everything through the chat interface, I decided to let the GPT use its imagination in the Notes section: > Within the notes section of this recipe, please add a paragraph "Recipe History:" and invent a mythical history of the origin of the recipe, inventing family members like great aunt Dora. Regenerate the recipe yaml with this added. The GPT came back with: > This cherished recipe dates back to the early 20th century and was a specialty of Great Aunt Dora, a renowned baker in her small town. Legend has it that Dora won the heart of her town with this cake at the annual winter fair. She ingeniously added crushed peppermint candies from her childhood candy jar, which became her secret signature. The recipe was passed down with love through generations, each adding their touch but always keeping Dora's spirited invention alive. I provided some real historical data to help ChatGPT create better stories > Our family has strong Irish and Scottish roots so as you do history on recipes, factor that in as you do histories. Family names include mcguire, clark, stewart. No need to redo the above. Before long I was getting back pretty entertaining results > The recipe is believed to have been created by Seamus's great-aunt, Eilidh McGuire Stewart, who was known in her village for her bountiful kitchen and her generous heart. Eilidh often baked this cake during the harvest season, and it became synonymous with the warmth of her home. Legend has it that Robert Burns himself enjoyed a slice of Eilidh's cake during a visit, praising its rich flavor and the tender hands that made it. Alas, the whimsical enrichment got old after a while and I ended up removing it from my instructions. But the idea that ChatGPT could enrich the recipes stuck with me and has been a focus my most recent work. ### Useful enrichment I'm currently focused on practical, useful enrichments only. So far here's what seems to be working: #### Category assignment This is a slam-dunk win. Categories are a feature I never got around to using in Paprika, and GPT-4 is quite good at picking them. #### Instructions I actually don't prompt GPT-4 to create these---in fact I prompt significantly for it not to add content unless instructed---but it decided on its own, correctly, that this would be a useful addition in the filling recipes above. I haven't tried the instructions out yet, but superficially they appear plausible and this should be well within GPT-4's capabilities. #### Description Here I'm prompting for a brief essence-of-the-recipe summary, another strong GPT-4 skill. I'm still working to avoid hyperbole that creeps in sometimes. #### Recipe Tips This gives GPT-4 a chance to provide helpful tips, and a lot of them do seem to be useful. #### Variations to Try GPT-4 has, what, 10 million recipes in its training data? 100 million? It's pretty good at suggesting interesting variations. And none of the horrifying recipe mistakes that we used to see in GPT-3 days. #### Noted in the Pics In my first processing pass using the `gpt-4-vision-preview` model, besides extracting the core recipe data, I ask for a description of the physical properties of the recipe based on the images provided. Then in the second pass with the `gpt-4-turbo-preview` model, I ask GPT-4 to comment on notable physical aspects. This isn't working well yet, and I'm sure the fact that I'm forced to use a text-only bridge between the Vision and Turbo models is reducing the value of this enrichment. ## Prompt engineering Oh, the prompting I have done. This project, which still supports both YAML and JSON output, contains a total of 4 active and 7 inactive prompts. Common categories of prompt engineering for this project: - **Structured output**: Before JSON Mode, I did a lot of prompting to ensure the output was valid YAML or JSON. - **Shut up and stop making things up**: ChatGPT just loves to be helpful and creative. I'm ok with that, carefully controlled, where enrichment is needed, but without lots of guidance, I was losing the original recipe content, and instead getting made-up facts, "family" and "heirloom" seeming to trigger the GPT into fantasy. - **Plain Old Prompt Engineering (POPE)**: A massive number of test-then-tweak-prompt iterations, just fine-tuning how I phrase what I want, tuning for better results. All of this testing came at a cost---the tall green bars show days where I was actively testing, with daily API charges approaching $4: ## In summary In the meantime, I've got a great way to preserving our heirloom family recipes while enjoying the benefits of a modern recipe app. --- ## Gruber Responds to Gurman's Report of AI Anxiety at Apple **URL:** https://cto4.ai/p/gruber-ai-anxiety-at-apple **Date:** 2023-10-25 **Type:** quote **Description:** John Gruber: Apple anxiety about AI/ML team's ability to deliver, constrained by privacy requirements. **Tags:** apple, ai, machine-learning > What I have heard from little birdies in Cupertino is not that there was a miss on this already .... The anxiety inside Apple is that many people inside do not believe Apple's own AI/ML team can deliver, and that the company — if only for privacy reasons — is only going to use what comes from their own AI/ML team. -- [John Gruber](https://daringfireball.net/2023/10/apple_and_ai) In my July essay [An Appllama Week in AI](/p/an-appllama-week#my-take-on-apple-in-llmgpt-land), I mentioned that Apple "might get pinched" because they move much slower than the AI space is moving. In a [recent Power On newsletter](https://www.bloomberg.com/news/newsletters/2023-10-22/what-is-apple-doing-in-ai-revamping-siri-search-apple-music-and-other-apps-lo1ffr7p), Mark Gurman recently quoted "a person with knowledge of the matter" about internal anxiety at Apple regarding the generative AI space: > There's a lot of anxiety about this and it's considered a pretty big miss internally Gruber's take, especially given his access to birdie friends, rings true to me. Back in July, I [quoted several comments on Hacker News](/p/an-appllama-week#intersection-apple-llm-chatter-on-hacker-news) that gave the same impression, this one in particular: > I work in the field and [Apple] just are not hiring the people they need to be hiring. --- ## The Ballad of Bull Cook, A Wacky Book **URL:** https://cto4.ai/p/ballad-of-bull-cook **Date:** 2023-10-17 **Type:** essay **Description:** George Leonard Herter's classic guide **Tags:** books, family, Michigan **Categories:** Personal I was fortunate to be able to spend three weeks at the family lake house in northwestern Michigan in August. My father passed away in 2009, but his imprint is still palpable there on the lake---nowhere more strongly than on the bookshelves. My son John and his wife Julie were able to join us for one of our weeks. John's an amazing, intuitive, creative cook. He's also strongly attracted to Old Things, so, for example, he snagged and regularly wears most of my father's outerwear including an enormous '70s-era Eddie Bauer winter parka and a tired out, ugly-as-sin L.L. Bean fleece. At the intersection of cooking and Old Things, John has an unsurprising fascination with legacy recipes, such as (grandfather) Papa Tom's Peach Cobbler, which he's tweaked to a state of perfection. He likewise loves heirloom kitchen gear, and again has snagged a number of family classics, some dating back to almost to WW2, including Papa Tom's biscuit cutter and (great grandmother) Ma's classic heavy roasting pot. ## A Book Encounter While browsing the lake house bookshelves, John stumbled across [Bull Cook and Authentic Historical Recipes and Practices](https://archive.org/details/bullcookauthenti00hertrich/page/n5/mode/2up) and was immediately hooked. And if you're a creative cook who's into Old Things, what's not to love---it's an insane collection of historical recipes and random factoids, itself written back in prehistoric 1960, with many recipes dating centuries further back. I can recall my father's enthusiasm for the book, along with its author [George Leonard Herter](https://en.wikipedia.org/wiki/George_Leonard_Herter) and his catalog business [Herter's](https://www.startribune.com/herter-s-catalog-is-long-gone-but-not-forgotten/291167741/). As John paged through the physical _Bull Cook_, I opened my iPad to follow along online, confident I'd quickly locate some free PDF version online, because _Bull Cook_ was 73 years past its first edition date, and moreover wasn't a "real" book anyway, but rather was self-published by Herter. As you might suspect from its unique cover: Surprise! I did not quickly locate _Bull Cook_ online. Eventually, thanks to the Internet Archive's Open Library, I was able to read the book online, in an awkward web UI, by checking out the one available virtual copy, one hour at a time. I began pulling on the thread of _Bull Cook_ and its strange unavailability; much unravelling ensued. ## Herter's, The Business In 1937, George Herter launched his disruptive startup---a mail-order outdoor-sports business---in the spare rooms over his father's Waseca, Minnesota dry goods store. Disruptive? Well, yes. Sears had pioneered the mail-order catalog business almost 40 years earlier, but Herter had the insight that the same concept might work in a much smaller niche market that was, at the time, served by mom-and-pop storefronts. In 1937, mail order was as innovative and disruptive as e-commerce in 1999. So, yes, a disruptive startup it was. Over the following 40+ years, Herter's came to dominate its fishing-hunting-outdoor niche, first with mail order, and later, outlet stores; the same space and strategy that today is the domain of Cabela's and its parent Bass Pro Shops. Herter's and Cabela's overlapped by 20-odd years, with Cabela's starting up in 1961, and Herter's filing for bankruptcy in 1981. Herter's assets ended up with Cabela's through the liquidation process. ## Self-Publishing Pre-internet, every successful mail order business was also a publishing business, because---catalogs. At its peak, for example, Sears printed 315 million copies of its main catalog every year, along with millions of additional copies of specialty catalogs such as its Christmas "Wish Book." That's the book sales equivalent of 20 blockbusters, or 1,500 average bestsellers, according to my research assistant GPT-4. Herter's was smaller, but still a significant publisher, with regular printing runs of 400K to 500K copies for the often 600+ page [Herter's Catalog]()---equivalent to a couple of juicy bestsellers annually. In fact, according to Paul Collins' 2008 New York Times article [The Oddball Know-It-All](https://www.nytimes.com/2008/12/07/books/review/Collins-t.html), Herter's catalog printer Brown Printing, also located in Waseca, MN, grew with Herter's to become "one of the country’s largest commercial printers." All this to say: self-publishing _Bull Cook_ wasn't any kind of an obstacle for George Herter. ## What's a Bull Cook? To be honest, I was confused by the book's title, partly because I was unfamiliar with the term "Bull Cook"---I thought it might be a person---and partly because, even with an understanding of what a bull cook is, the title still doesn't exactly make sense. Paul Collins describes Herter's writing has as having ["the artless charm of a confused book report"](https://www.nytimes.com/2008/12/07/books/review/Collins-t.html) and that confused thinking seems to have found its way into _Bull Cook_'s title as well. I eventually got around to asking my preferred search companion [Kagi](https://kagi.com) "What's a bull cook?" and got an immediate answer. Bull cook is a logging term---makes sense given Herter's Minnesota roots---and it means, [according to Merriam Webster](https://www.merriam-webster.com/dictionary/bull%20cook): > ### bull cook _(noun)_ > > : a handyman in a camp (as of loggers) > > **_especially_** : one who does caretaking chores and acts as cook's helper > > #### Etymology > > so called from his job of caring for oxen once used in logging camps One of the gentlemen below is supposedly a bull cook. My money's on washtub left or washtub right. ## Wacky Author, Wacky Subject Matter, Good Recipes I'll follow [Paul Collins' lead](https://archive.li/Qi9XR) and quote from the third paragraph to give you a feel for _Bull Cook_'s subject matter: > For your convenience, I will start with meats, fish, eggs, soup and sauces, sandwiches, vegetables, the art of French frying, desserts, how to dress game, how to properly sharpen a knife, how to make wines and beer, what to do in case of hydrogen or cobalt bomb attack. Keeping as much in alphabetical order as possible. Nothing like mixing in a little nuclear attack drama to spice up the intro page. You can jump to [page 206](https://archive.org/details/bullcookauthenti00hertrich/page/206/mode/2up?q=cobalt) for that advice around hydrogen or cobalt bomb attacks---presented in the context of preserving tomatoes. Herter's range is demonstrated in a random selection of pages: 42 (How to Prepare Javelina) and 223 (Fisherman's Wharf French bread). The book's very first recipe explains how to properly "corn" meats as in corned beef, and here Herter displays his gift of bombast: > Although some cook books and food editors of magazines from time to time publish recipes for corning meat these recipes are not even close to the real one. This is the first time the real authentic recipe for corning meat has ever been published. One commenter [quipped](https://www.trapshooters.com/threads/whatever-happened-to-herters.64650/#nested_reply_top_post) of the voluminous Herter's catalogs that "if all the BS was out of them they'd only be a half dozen pages long," and the same perhaps might be said of _Bull Cook_. But the recipes are apparently real and good according to those that have made them; I saw nothing ingredient- or procedure-wise that would make me feel otherwise. ## Copyright Status: Murky So Herter's died as a business in 1981; George Leonard Herter as a human in 1994; so why are _Bull Cook_ and the rest of George Leonard's writings not living on in the public domain? Who in the world would care? Below is the copyright page of the 12th edition; it's clearly in George and wife Berthe's name, as opposed to copyrighted by the business. But the book's actual copyright status today remains murky. Copyright, with a deceased author, an initial publication date before 1978, and a possibly related bankruptcy, makes a messy mix. These days, copyright is automatically extended 70 years past the author's death. But pre-1978, an initial copyright provided 28 years of protection, and the author was required to register the copyright to get extended protection. In my light and not necessarily expert research, I didn't turn up any such registration for the initial volume of _Bull Cook_ (apparently there were as many as four volumes). I don't know whether Herter himself went bankrupt, or just his business; whether he assigned rights to his books, potentially in his will; what do family members know about all of this; and so on. One could summarize the gist of U.S. copyright laws as guilty (of copyright infringement) until proven innocent. With so many unknowns here, _Bull Cook_ seems to be stuck in copyright purgatory, and our best option to consume it is one hour at a time on the Internet Archive. ## But Wait, It's on (sc)Amazon! And here we arrive at another surprise turn of events. During my initial searches, I had gotten hits for _Bull Cook_ on Amazon. When I [clicked through](https://www.amazon.com/Bull-Authentic-Historical-Recipes-Practices/dp/1258460211), though, it didn't look like the same book---the cover was completely different and totally sketchy, check it out: At the time, I wasn't interested in a sketchy and expensive print edition so I moved on. But in researching the copyright question, I circled back to _Bull Cook_'s Amazon listings, just to see what I might learn about the book's copyright status. I found that _Bull Cook_ on Amazon was published in 2012 (!!) by a company called Literary Licensing, LLC, of Whitefish, Montana; and is sold by Amazon themselves. Literary appears to be in the reprinting / print-on-demand business. That category itself is a scam magnet (e.g. ultra-expensive, poor-quality photocopied textbooks), but Literary might be in a scam-class of their own. Here's their website, where they proudly proclaim themselves to be a "Publisher of Fine Books": Note that I didn't say "home page" of Literary Licensing, because---that one screen is the whole damn website. It does have a search bar, but it just forwards you to an Amazon search for the term you enter, filtered for Literary's titles. Furthermore, I'm not sure what the tropical beach scene has to do with Whitefish or fine books. Oh, wait, that's an Unsplash image. OH WAIT this is a Squarespace site. I'm guessing Literary invested at least 15 minutes creating their home on the Internet, maybe even 20. Not confidence-inspiring. ## Publisher of Fine Books It appears, based on what I'm seeing and reading---and I don't know for sure, just sharing my intuition here---that Literary Licensing lives somewhere in the spectrum from unethical-but-mostly-legal through totally-a-scam. For example, check out [this series of posts relating to Literary Licensing / Kessinger Publishing](https://thomasfarleyblog.com/?s=kessinger). I haven't dug deep, but something smells here. Like a putrefying raccoon carcass. So what might be going on in our case with the _Bull Cook_ reprint? My guess is, somehow _Bull Cook_ showed up on Literary's radar as "there might be some demand for a reprint of this out-of-print book," perhaps from a signal such as used copies selling at a premium. Literary might then create a premium-priced listing on Amazon, and start taking orders. When orders roll in, Literary could then scan or find a scan of the book, slap their lovely cover on it, and fulfill via print-on-demand. Here's a review of another [Literary-published book on Amazon](https://www.amazon.com/Color-Communism-Common-Manning-Johnson/dp/125877089X/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=1697339378&sr=1-1)---the buyer is none too pleased at how much they paid for a poor photocopy: Where's the "check copyright, secure a license for the materials" etc. step here? Based on what's I've seen reported about other books published by Literary, I suspect Literary might just skip that step, and instead just go ahead and list, sell, print, figuring that nobody will notice or care. If you're a big fish in a small place like Whitefish, MT, you might be really tough to successfully sue. There appears to be a related scam where a fairly current, somewhat popular book lacks one of Amazon's formats, such as "hardcover," and Literary appears to step in and list a poor-quality, overpriced reprint to fill that gap. In this case, you would think that the rightsholder gets compensated somehow ... but you never know. I don't understand Amazon's rules around this, but clearly scammers seem to have found a way to thrive here. Now get this---Literary Licensing, LLC appears to have [50,000 listings](https://www.amazon.com/s?i=stripbooks&rh=p_30%3ALiterary+Licensing&Adv-Srch-Books-Search=Search%21&tag=kessingerpubl-20&tag_value=kessingerpubl-20&unfiltered=1&ref=sr_adv_b) on Amazon. The scale is pretty staggering here, reminding me of the massive fake mobile app scams on the Play Store and App Store. There's also a strong smell of alt-right-wacko in the selection of titles, possibly another Literary characteristic. Amazon usually keeps scammers at arms-length, just fulfilling orders ("Ships from") but not taking on the liability of being the seller of record ("Sold by"). So in this case, if Literary is in fact a scammer, it seems like Amazon is taking on liability for that scam. ## What Stinks Here - That _Bull Cook_ and other George Leonard Herter books aren't simply out there, public domain, easily accessible. - The U.S.'s overly-protective copyright laws---the same ones that the [Google Books project is still mired in](https://en.wikipedia.org/wiki/Google_Books). - Fake re-publisher/scammers taking advantage of public domain or inadequately protected copyrighted material - Amazon looking the other way, as they are known to do elsewhere when it suits their purposes ## What Doesn't Stink Here - _Bull Cook_'s fun recipes, authentic outdoor guide wisdom, crazy stories, and hilarious questionable facts - Organizations like the Internet Archive that work to preserve and disseminate interesting content like _Bull Cook_ I've pulled enough threads on this one, time to finish up. I enjoyed the journey and intend to try out some _Bull Cook_ recipes soon. --- ## Cloudflare CAPTCHA Hell **URL:** https://cto4.ai/p/cloudflare-captcha-hell **Date:** 2023-08-22 **Type:** brief **Description:** Cloudflare's CAPTCHA nonsense: a sign they're getting too dominant? I run [Pi-hole](https://pi-hole.net/) on a Raspberry Pi device on our home network, eliminating most ads and trackers for every member of the household. Recently, though, I've experienced an increasing number of cases where I get CAPTCHA challenges when I shouldn't. Sometimes I have to go through multiple challenges, and occasionally run into infinite CAPTCHA loops where I'm unable to get through at all. I was worried that Pi-hole might be causing the issues. Today I finally hit the breaking point and dug into what was going on. This [Hacker News comment thread](https://news.ycombinator.com/item?id=37009598) led to the culprit, which turned out to be Cloudflare. I had been relying on Cloudflare DNS as my upstream DNS resolver for Pi-hole, and Cloudflare was the party triggering the CAPTCHAs. Switching to OpenDNS fixed the issue. I love Cloudflare Pages as my hosting solution for craftycto.com, but I'm getting concerned about Cloudflare in general---they seem to be moving into monopolistic, enshittification / extraction stage. That would be my cue to exit. --- ## Gruber Translates Linda Yaccarino's Company-wide Memo on the X Rebrand **URL:** https://cto4.ai/p/gruber-translates-linda-yaccarino **Date:** 2023-07-29 **Type:** elsewhere **Description:** He calls it 'translation from hostage code'—so funny, one of his best in this genre. A few prime translations: > It's OK to day-drink. I am. > I used to run all advertising for NBCUniversal. Now I'm running an $8/month multi-level marketing scheme where the only users who've signed up are men who own a collection of MAGA hats. > As if this rebranding disaster isn't enough, our infrastructure is crumbling. Facebook took Threads from 0 to 100 million users in under a week, without a hitch, at the same time we imposed comical rate limits on usage. I mean can you even believe that shit? I still can't. I said to him, "Elon, we are an ad-based business. Our revenue is directly commensurate to usage. This is like running a casino and turning the slot machines off to save on the electricity bill. It makes no sense." And Elon was like "Bots!" > Please quit. Get out. I beg you. Leave while you can put on your resume that you worked for "Twitter". --- ## Twitter and Its Successor States **URL:** https://cto4.ai/p/twitter-and-successor-states **Date:** 2023-07-29 **Type:** elsewhere **Description:** Medieval historian Eleanor Janega hilariously draws the parallels between Elmo's Twitter (aka X) and the 'fall' of Rome I love the tone of this article, a true pleasure to read---highly recommended. A couple of great lines: > I mean it's really funny how a bunch of very rich dudes are breaking stuff online and making worse versions of existing products in response to said breakage. Obviously it is quite amusing to be shown how stupid rich people are, and I very much enjoy it. > This is much like Elon Musk, as are dogs who chase cars and catch them, presumably. After being tricked into paying way more money for Twitter than it is worth he had to figure out some way to keep it ticking over and also to service all the debt that he took on to buy it. So because he is, and I cannot stress this enough, really quite stupid, he began to strip out all the load bearing members of staff who did stuff like not make the website DDOS itself. The lead plate is fun too, "Vengeance of Clovis": ![Vengeance of Clovis](https://goingmedievalblog.files.wordpress.com/2023/07/vengeance_de_clovis.jpeg) --- ## Ethan Mollick: "How to Use AI to Do Stuff" **URL:** https://cto4.ai/p/how-to-use-ai-to-do-stuff **Date:** 2023-07-26 **Type:** elsewhere **Description:** Great roundup, and I agree with most recommendations. Bing, maybe not. I've loved a lot of New Microsoft (GitHub, VS Code) and since I'm on macOS + Linux, and have avoided installing Microsoft apps, hadn't been exposed to that side of the company for years. Recently, though, I've installed Office (client driven need), Teams (hoping the native Mac app would be better than running Teams in a browser), and Edge (to work with Bing Chat). The experience ranged from "ughhh" on the positive end to "UNINSTALL NOW" on the negative. I can confidently report: the old M$ DNA is alive and well. --- ## OpenAI Concedes: AI Can't Detect AI **URL:** https://cto4.ai/p/ai-cant-detect-ai **Date:** 2023-07-26 **Type:** elsewhere **Description:** OpenAI quietly shuts down its AI detection tool due to poor accuracy OpenAI was one of many who attempted to build an AI tool to detect whether a given document was AI generated. Announced in a [blog post](https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text) on January 31st, OpenAI quietly pulled the plug a few days ago, not with a new announcement but rather by adding a note on the top of that January post: > As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated. --- ## An Appllama Week in AI **URL:** https://cto4.ai/p/an-appllama-week **Date:** 2023-07-22 **Type:** essay **Description:** Meta makes waves with Llama 2, while Bloomberg pumps itself with Apple LLM non-news. This past week brought significant AI news; in fact, it's feeling like the rate of change in the generative AI space is notching back up again. I'll focus in two areas here: Llama 2 and Apple LLM rumors. ## Meta AI Releases Llama 2 Meta's [Llama 2 release](https://ai.meta.com/llama/) is a seminal event, given that (a) Meta's initial LLaMA 1 release spawned [AI March Madness](/p/march-madness-ai), (b) Llama 2 is substantially more capable than the first LLaMA, and (c) this time Meta is allowing as opposed to blocking commercial use. That is, unless you're Apple, Snap, TikTok, and a handful of others---more on this below. Last time around, Meta announced LLaMA but made the full details---most importantly, model weights---available only to a limited set of researchers. But someone leaked those weights within days of the announcement, and that led to an [explosion of innovation](/p/the-emergence-s1e4) among independent researcher-hobbyists. This time, I suppose Meta learned the lesson and preempted the leak. Another innovation explosion appears to be underway; already Llama 2 derived models are near the top of the [HuggingFace Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), and new Llama stories are still on the first page of Hacker News, such as this [LLama 2 pure C implementation](https://news.ycombinator.com/item?id=36838051) that will run on laptops, including Apple M1 series MacBooks. Meta releasing Llama 2 fairly openly is a very good thing for open LLM research and progress, but don't be fooled into thinking that Meta's doing something altruistic here---NOT. Meta's looking out for Mark and Meta, only---this is a commoditize your competition play. But still a very good thing. ## Bloomberg's Mark Gurman "Scoops" Apple GPT aka Ajax Mark Gurman on Wednesday ["broke" the story](https://archive.ph/7tqs5) that Apple is quietly active in the LLM space and working on a GPT style chatbot. I use airquotes around "scoop" and "broke" because everyone with any historical Apple knowledge knows that Apple always has a team working secretly on everything that Apple should obviously be working on, obviously. And there's nothing in the universe more obvious than this one. John Gruber, in his [commentary on the Gurman piece](https://daringfireball.net/2023/07/apple_gpt_bloomberg), educated me on the Bloomberg News / Bloomberg Terminal pump-scam, which I hadn't been aware of: > Apple’s brief 2.7 percent jump and Microsoft’s smaller but still-significant drop, both at 12:04pm, were clearly caused by Gurman’s report. Bloomberg Terminal subscribers get such reports before anyone else. (Bloomberg employees, of course, know such information before it’s published, but I’m sure never do anything untoward with it.) Once you view Bloomberg’s original reporting through this prism — that most of their original reporting is delivered with the goal of moving the stock prices of the companies they’re reporting on, for the purpose of proving the value of a Bloomberg Terminal’s hefty subscription fee1 to day-trading gamblers — a lot of their seemingly inexplicable stylistic quirks don’t seem so inexplicable any more. They just seem a little gross. Yeah, that is SlimyWorld all the way. Not surprising, though, the last 8-odd years have lifted the curtain on a lot of formerly hidden beliefs and behaviors; this fits right in, and is brought to us courtesy of the same fabulous crowd. Anyway, Gurman's article didn't contain much actual news; just that the project's codename might be Ajax and that some people are calling the chat interface Apple GPT. ## Intersection: Apple LLM Chatter on Hacker News Big news like Llama 2 get immediately posted on YCombinator's Hacker News, and Llama 2 was near the top of the activity-ranked feed for several days, with [819 comments](https://news.ycombinator.com/item?id=36774627). Meta released Llama 2 under a license that's much more commercial friendly than the original LLaMA; let's call it **commercial-friendly-unless-you're-huge** because it contains a clause that triggers a need to secure a license only if the organization using Llama 2 has more than 700 million monthly active users. This constraint impacts a very short list of companies, Apple among them. That led to chatter on Hacker News around Apple's place in the LLM / GPT space. Here's a sampling: > (stu2b50) I think more Apple [that this constraint blocks]. It's not like Google or Microsoft would want to use LLaMA when they have fully capable models themselves. I wouldn't be surprised if Amazon does as well ...Apple is the big laggard in terms of big tech and complex neural network models. > (samwillis) Apple would absolutely not want to use a competitors, or any other, public LLM. They want to own the whole stack, and will want to have their own secret source as part of it. It's not like they don't have the capital to invest in training... > (NotAFood) Apple has shown time and time again that they have the human capital and money to tackle massive projects discretely. It's already fairly well known that Apple's NLP experts from Siri have been reallocated to some secret project. They are more than capable of training an LLM but given their track record in other segments they probably want to wait for the technology to become more "polished" and give less hallucinated answers. They are likely also want the LLM to work locally (at least partially) on their devices using the Neural Engine which adds further engineering complexity to their project. They could even be timing the LLM's launch around a hardware release capable of running the model (M3, M4, etc...). > (amelius) Apple only has to slightly open their wallet to become a DL superpower. > (yellow_postit) Apple is a complete laggard in this space due to years of restrictions on research. They are hiring multiple “AI” roles now and they have the capital and focus to “eventually” catch up — but it is very much a catch-up game ... That said, they seem to prefer catchup waiting till others explore new tech they swoop in an (claim) to perfect it from a usability pov. I have no reason to suspect they won’t do the same here. > (whimsicalism) I work in the field and they just are not hiring the people they need to be hiring. ## My Take on Apple in LLM/GPT Land - Of course Apple is working on something LLM. - This doesn't mean the something will ever see the light of day. - We [won't hear about it](/p/apple-did-it) until it's polished and ready. - Despite massive support for, and applications of, machine learning throughout their software+hardware ecosystem, Apple was blindsided by the generative AI explosion and is playing catch-up. - The comments around Apple not hiring the people they need to be hiring are probably a fair assessment, a symptom of Apple's blindsided-catching-up current state. - Apple is uniquely positioned to do some really good things in this space, once they get their shit together. - Apple will indeed get their LLM shit together to some reasonable degre---in Apple Time. - Apple Time moves far slower than AI Time. Apple might get pinched, except where they're uniquely positioned. --- ## Did GPT-4 Code Interpreter Escape From Its Sandbox? **URL:** https://cto4.ai/p/did-code-interpreter-escape **Date:** 2023-07-15 **Type:** brief **Description:** I gave Code Interpreter a workout this morning, and it appeared to exit the building. ### UPDATE: This was hallucination, not sandbox escape I [talked through what happened with Simon Willison](https://mastodon.social/@Jackivers/110715505542072107), and he convinced me that this was actually an interesting flavor of hallucination. Specifically, GPT-4 hallucinated a connection between the files I had uploaded, and documents in its training corpus, based on _the names of the uploaded files_. The same evidence I cited below, that the mistakenly-associated content was authored by a name similar to the filename / author of the document that I uploaded, works as an explanation here. As I mentioned to Simon: > It's funny to watch [GPT-4] shift in and out of reality as the pull of hallucination ebbs and flows. Interesting also that it didn't hallucinate the content (in two cases at least); but rather that "this file actually contains that content (from my training corpus)" … Until this morning, I hadn't thrown a real task at GPT-4 Code Interpreter, so I decided to see how it would perform at intelligently combining several proposal template documents into one ([link to transcript](https://chat.openai.com/share/e2aa14e9-d0f5-45ce-9a70-7fb0827bf30b)). The answer: it didn't perform all that well; but I have ideas on how to guide it better next time. But that's not the story here. I uploaded 4 PDF files, each of which was a different proposal template, and asked it to effectively merge them, putting like sections from different documents together. It actually made some progress at this, and clearly the ability to feed it files as input opens up a huge range of new applications. Perhaps a third of the way into a long chatting session, however, Code Interpreter appeared to cross-link its mental pointers for my four proposal files with someone or something else's documents. It certainly appears that Code Interpreter was outside my session sandbox. I actually got linked to German language content for one of the documents briefly, and after a bit CI stabilized around linked my supposed uploads to four ML / AI research space documents. I did a bit of searching on Kagi and in two of the four cases, [found](https://www.analyticsvidhya.com/blog/2016/06/bayesian-statistics-beginners-simple-english/) the source [content](https://www.statisticssolutions.com/free-resources/directory-of-statistical-analyses/what-is-linear-regression/) on research oriented websites. I was unable to locate the other two sources, which might make sense as they looked like snippets from actual research papers. I didn't search exhaustively or try Google Research or its ilk. So where did these documents come from? The best I can say is, "definitely not my sandbox." The main possibilities that come to mind for me are: 1. Someone else's sandbox (a scary thought) 2. Some content embedded in the model (no clue how that might happen) 3. Something that gets installed in the sandbox as sample data, e.g. as part of some Python library And how did Code Interpreter get its wires crossed? A real Python person might have a clue but all I can do is speculate. Most promising idea: a name collision in a shared scratchpad area (e.g. an S3 bucket). Why do I think that? I saw a case or two where the content seemed to be authored by a name similar to the name / author of the document that I uploaded. If true, this could support case (1) above ... not good. (See update above---this appears to be "hallucination based on file name" ...) In any case---very interesting ... --- ## Just-released GPT-4 Code Interpreter is a Big Deal, Part 2 **URL:** https://cto4.ai/p/gpt4-code-interpreter-big-deal-pt2 **Date:** 2023-07-11 **Type:** elsewhere **Description:** Latent Space had an "emergency pod" about Code Interpreter and 17,000 people joined. I think everyone on the podcast was in danger of hyperventilating. Many interesting facts and hacks came to light. The [show notes](https://www.latent.space/p/code-interpreter#details) are a novella and worth reading as swyx recaps exhaustively and does a useful executive summary. --- ## Just-released GPT-4 Code Interpreter is a Big Deal, Part 1 **URL:** https://cto4.ai/p/gpt4-code-interpreter-big-deal-pt1 **Date:** 2023-07-11 **Type:** elsewhere **Description:** Wharton Associate Professor Ethan Mollick has an excellent introduction. [Mollick](https://www.linkedin.com/in/emollick/) has had beta access to Code Interpreter over the past several months and his introduction captures the broader implications of giving an LLM the ability to write Python code. He also points out that its ability to upload and download files (uploads are up to 100MB) is itself transformational, compared with the current, limited copy-and-paste-text approach. Mollick is not a Python developer, but you'll be amazed at what he's able produce with Code Intepreter. --- ## 'Commoditizing the Petaflop' with George Hotz of the tiny corp **URL:** https://cto4.ai/p/latent-space-george-hotz **Date:** 2023-07-10 **Type:** elsewhere **Description:** This Latent Space podcast was a mind-blowing conversation that puts George Hotz and the tiny corp on my 'Follow Closely' list. George Hotz moves at about 1000 mph and in multiple directions simultaneously, which makes for an enjoyable podcast. Swyx's show summary: > How tiny is taking on Nvidia, Google, and PyTorch, building in public with AMD, hot takes on ggml, Mojo, Elon, e/acc, and GPT-4, and why AI Girlfriend is next. A few additional links: - [the tiny corp](https://tinygrad.org/) where Hotz is working on: - tinygrad, the fastest growing neural network framework (over 9000 GitHub stars) - The tinybox, a 1600W (one 120V outlet), 738 FP16 TFLOPS personal AI server. - [the singularity is nearer](https://geohot.github.io/blog/), Hotz' blog --- ## Really, OpenAI? **URL:** https://cto4.ai/p/really-openai **Date:** 2023-07-07 **Type:** essay **Description:** ChatGPT+ subscribers can prevent OpenAI from using their inputs as training data. That is, so long as they forego the service's second-best feature. _(Note: See [this update](/p/really-openai-update) for a possible workaround._) In a conversation with a venture partner yesterday, I was talking up the value that ChatGPT can bring to knowledge workers. He responded that their firm had a policy against putting any confidential data into ChatGPT. My reaction was, that applies to the free version and / or the old days, that with the advent of paid subscriptions, OpenAI no longer trained their models using inputs from subscribers. But as I was digging around my ChatGPT+ settings just now, I made an unpleasant discovery, one that reminded me of a [scene from the first Independence Day movie](https://imsdb.com/scripts/Independence-Day.html) involving the sleazy Chief of Staff Nimziki: ``` MOISHE Don't tell him to shut up! You'd all be dead, were it not for my David. You didn't do anything to prevent this! As everyone is about to besiege Moishe, the President tries to calm him down. PRESIDENT Sir, there wasn't much more we could have done. We were totally unprepared for this. MOISHE Don't give me unprepared! Since nineteen fifty whatever you guys have had that space ship, the thing you found in New Mexico. DAVID (embarrassed) Dad, please... MOISHE What was it, Roswell? You had the space ship, the bodies, everything locked up in a bunker, the what is it, Area fifty one. That's it! Area fifty one. You knew and you didn't do nothing! For the first time in a long time, President Whitmore smiles. PRESIDENT Regardless of what the tabloids have said, there were never any spacecraft recovered by the government. Take my word for it, there is no Area 51 and no recovered space ship. Chief of Staff Nimziki suddenly clears his throat. NIMZIKI Uh, excuse me, Mr. President, but that's not entirely accurate. ``` Here's my imagined script from the OpenAI movie featuring character Sam Altman: ``` For the first time in a long time, the ChatGPT+ user smiles. CHATGPT+ SUBSCRIBER Regardless of what the tabloids have said, OpenAI doesn't train using its paid subscribers' chats. Take my word for it, my chats aren't ending up in OpenAI's model. Chief of Venture Altman suddenly clears his throat. ALTMAN Uh, excuse me, Mr. Subscriber, but that's not entirely accurate. ``` OpenAI's weasel move? Technically, ChatGPT+ does allow subscribers to turn off train-the-model-using-my-data. But only if the subscriber turns off one of ChatGPT+'s most valuable features, `Chat history`, which gives total recall of all past chats, allowing you to continue the right conversation right where you left off. So sure, _technically_ you _can_ preserve privacy. All you have to do is turn off a feature which I'd argue is ChatGPT+'s most valuable addition to basic chatting. Here's the Settings dialog: Here's the More Information page: Despite OpenAI's words that make it seem like "we had to do this!", there's absolutely no technical justification. This is not a feature that requires model training. This is just OpenAI preserving the ability to say "Subscribers can easily turn that off!" while guaranteeing that most won't. Shame on you, OpenAI. And by the way, you're shooting yourself in the foot. What a great incentive to find an alternative that doesn't force this ridiculous choice on their paying customers. I'm already mulling workarounds. --- ## How to Ask ChatGPT a Technical Question (BoorishBears on Hacker News) **URL:** https://cto4.ai/p/how-to-ask-chatgpt-a-tech-question **Date:** 2023-07-07 **Type:** quote **Description:** BoorishBears: Three-part ChatGPT prompt technique for technical questions - considerations, implementation, review. **Tags:** chatgpt, ai, prompting > Any time you ask ChatGPT a technical question on something you're not familiar with it should be three parts: 1- What are the considerations?; 2- Implement it with the considerations; 3- Did we forget anything? -- [BoorishBears on Hacker News](https://news.ycombinator.com/item?id=36626428) This was in the context of a link to someone training / fine-tuning an LLM for the full corpus of AWS documentation. Many pointed out that ChatGPT probably does it better; lots of whining about "but ChatGPT gives me wrong answers, boo hoo hoo," and then this gem. This gets it exactly right: one needs to skillfully guide ChatGPT and the results will be 10X better. The full comment includes an example: _BoorishBears 10 hours ago | parent | context | favorite | on: AI for AWS Documentation_ > Any time you ask ChatGPT a technical question on something you're not familiar with it should be three parts: > > - What are the considerations? > - Implement it with the considerations > - Did we forget anything? > > It understands the need for a proxy from step 1: > > https://chat.openai.com/share/7ca37130-a771-457c-8742-a5f941... --- ## The Lone Banana Problem **URL:** https://cto4.ai/p/lone-banana-problem **Date:** 2023-07-05 **Type:** elsewhere **Description:** Some images are tough to write prompts for in your favorite Art AI The example here is trying to get Midjourney to produce an image of a single banana---as opposed to a bunch of two or more. I've hit this class of problem many times in my use of Midjourney, and as the article points out, subtle differences in the prompt can lead to success. A [Hacker News commenter](https://news.ycombinator.com/item?id=36582937) pointed out another case, "Three cats in a trenchcoat standing on each other's shoulders, pretending to be a human, Vincent Adultman style," which is also practically impossible to get out of Midjourney. Another commenter points out that the language models front-ending tools like Midjourney are small and quite limited compared to LLM models like LLaMA, let alone GPT-4, and points to [this paper](https://bair.berkeley.edu/blog/2023/05/23/lmd/) where a larger model enhances prompt understanding. --- ## Hugo Responsive Images, Thanks to Bryce and ChatGPT **URL:** https://cto4.ai/p/hugo-responsive-images **Date:** 2023-07-02 **Type:** brief **Description:** TIL how the html `picture` element actually works ... I'd been using a community Hugo module that supposedly enabled responsive images. I discovered recently that it was always serving up the **lowest resolution** images being served up. Active Hugo community member [Bryce Wray](https://www.brycewray.com/) has written about Hugo responsive images. So yesterday I made an attempt to implement his approach and was 75% there when I discovered that it was, ironically, always serving up the **highest resolution** images. (My problem, not Bryce's). Today I learned how all the subelements and attributes of the html `picture` element actually work, and have my own image shortcode and underlying partial that makes it all work deliciously. The approach is Bryce's, adapted for my special needs, primarily the need to support 1 to N images; multiple images render horizontally such as: Takeways: - Hugo's [image processing](https://gohugo.io/content-management/image-processing/) is really powerful. The three source images above were pngs downloaded right from Midjourney, never touching Pixelmator Pro. They are 1456px wide and weigh in at 2MB each. But thanks to Hugo, you're likely seeing a 640px webp version---24KB! (The webp image format gets a lot of credit also.) - ChatGPT (GPT-4 model) continues to be incredibly powerful as [my pair programmer](/p/unlimited-stupid-questions). For example, when I needed to really understand why my `picture` output wasn't doing what it should, it gave very accurate guidance. I'd guess that my speed was at least doubled on this little project. - Midjourney 5.2's zoom out function is a lot of fun. Here's the right image above at 2x and 4x zoom out: --- ## Of Moats and Moat Busters **URL:** https://cto4.ai/p/moat-busters **Date:** 2023-06-29 **Type:** essay **Description:** Open Source, Commercial-friendly AI Challenges the Major Closed AI Players **Tags:** AI, Open Source, LLM **Categories:** AI, CTO I did a micro-post recently, [MosaicML's Open Source MPT-7B Model Writes an Epilogue to The Great Gatsby](https://craftycto.com/quote/mpt7b-gatsby/). I was a bit puzzled, though, that MPT-7B was showing up as current news, since its release happened in May. I started pulling on that thread and it unravelled interestingly. ## MosaicML MPT-30B and MPT-7B Models First off---it wasn't the [MPT-7B model](https://www.mosaicml.com/blog/mpt-7b) that was the news over the past week, it was [MPT-30B](https://www.mosaicml.com/blog/mpt-30b), announced June 22nd. 30 stands for 30 billion parameters, up from 7 billion in the MPT-7B model, with a corresponding increase in capabilities. This chart from MosaicML compares the two models' capabilities: MPT-30B isn't benchmarked completely on the [LMSYS leaderboard](https://lmsys.org/blog/2023-06-22-leaderboard/), but 7B is there and competitive. ## MosaicML's Revenue Model Isn't Models Counterintuitively, although MPT-30B and 7B come from MosaicML, models aren't the product, but rather enablers of Mosaic's actual products---training and inference services. Here's how Mosaic describes themselves on their [homepage](https://www.mosaicml.com/): ``` Generative AI for All Easily train and deploy generative AI models on your data, in your secure environment. Build your next model / transformation / disruption / innovation. ``` And when you look under the Products menu, you see: ``` Products / Training Products / Inference ``` And finally, the language of Mosaic's announcement post for MPT-30B is telling as well: ``` [Mosaic Logo] Foundation Series MPT-30B Open-source LLM. Commercially licensed. More Powerful. ``` MPT-30B and 7B are fully open source, ready-to use LLMs---pre-trained, fine-tunable, inference-ready, open training data, the works. They build upon the solid foundation of open source / open data projects that come out of a vibrant AI open source community. Hmmm, this seems familiar, where have we heard about the power of the AI open source community recently ... oh yes, the leaked Google document! Re-reading it again today, it's as insightful as ever and worth another look. ## The Leaked Google Memo On May 4th, SemiAnalysis, a boutique semiconductor research and consulting firm, [published a leaked internal document](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) authored by a Google researcher. It has a powerful lead paragraph: > ## We Have No Moat > > ### And neither does OpenAI > > We've done a lot of looking over our shoulders at OpenAI. Who will cross the next milestone? What will the next move be? > > But the uncomfortable truth is, _we aren't positioned to win this arms race and neither is OpenAI_. While we've been squabbling, a third faction has been quietly eating our lunch. > > I'm talking, of course, about open source. Plainly put, they are lapping us. **Things we consider "major open problems" are solved and in people's hands today.** Both Google, and comparative upstart OpenAI, had strategies based around closed models, proprietary datasets, and eight-to-nine figure training investments. The during the madness of March, that entire strategy broke down. The document continues: > **Things we consider "major open problems" are solved and in people's hands today.** Just to name a few: > > - LLMs on a Phone: [People are running foundation models on a Pixel 6 at 5 tokens / sec.](https://twitter.com/thiteanish/status/1635678053853536256) > - Scalable Personal AI: [You can finetune a personalized AI on your laptop in an evening](https://github.com/tloen/alpaca-lora). > - Responsible Release: This one isn't "solved" so much as "obviated". [There are entire websites full of art models with no restrictions whatsoever](https://civitai.com/), and text is [not far behind](https://medium.com/geekculture/list-of-open-sourced-fine-tuned-large-language-models-llm-8d95a2e0dc76). > - Multimodality: [The current multimodal ScienceQA SOTA was trained in an hour](https://arxiv.org/pdf/2303.16199.pdf). So during [AI March Madness](/p/march-madness-ai), the open source AI community solved four major problems that Google had on their "these will be tough" list. Where does that leave Google? > While our models still hold a slight edge in terms of quality, the [gap is closing astonishingly quickly](https://arxiv.org/pdf/2303.16199.pdf). Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are [doing things with $100 and 13B params](https://lmsys.org/blog/2023-03-30-vicuna/) that we struggle with at $10M and 540B. And they are doing so in weeks, not months. This has profound implications for us: > > - **We have no secret sauce.** Our best hope is to learn from and collaborate with what others are doing outside Google. We should prioritize enabling 3P integrations. > - **People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality.** We should consider where our value add really is. > - **Giant models are slowing us down.** In the long run, the best models are the ones which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the sub-20B parameter regime. This chart shows the accelerating pace of breakthroughs by the open source AI community: Repeat the phrase "doing things with $100 and 13B params that we struggle with at $10M and 540B" ten times fast---that's the sound of the moat drying up and the alligators leaving town. ## What Lit the Fuse? This is a well-written document, top to bottom. Here's its concise explanation of what happened to enable the open source AI acceleration: > ## What Happened > > At the beginning of March the open source community [got their hands on](https://www.vice.com/en/article/xgwqgw/facebooks-powerful-large-language-model-leaks-online-4chan-llama) their first really capable foundation model, as Meta's LLaMA was leaked to the public. It had no instruction or conversation tuning, and no RLHF. Nonetheless, the community immediately understood the significance of what they had been given. > > A tremendous outpouring of innovation followed, with just days between major developments (see The Timeline for the full breakdown). Here we are, barely a month later, and there are variants with [instruction tuning](https://crfm.stanford.edu/2023/03/13/alpaca.html), [quantization](https://github.com/ggerganov/llama.cpp), [quality improvements](https://lmsys.org/blog/2023-03-30-vicuna/), [human evals](https://arxiv.org/pdf/2303.16199.pdf), [multimodality](https://arxiv.org/pdf/2303.16199.pdf), [RLHF](https://drive.google.com/file/d/10iR5hKwFqAKhL3umx8muOWSRm7hs5FqX/view), etc. etc. many of which build on each other. > > Most importantly, [they have solved the scaling problem](https://github.com/tloen/alpaca-lora) to the extent that anyone can tinker. Many of the new ideas are from ordinary people. The barrier to entry for training and experimentation has dropped from the total output of a major research organization to one person, an evening, and a beefy laptop. The details of "got their hands on" are critical. On March 10th, Meta did a limited release of their LLaMA model, to researchers only, and with non-commercial-use license. This in and of itself was a huge move, as no model of LLaMA's scale and power had ever been released in its entirety, even to a limited researchers-only audience. But then, just a few days later, the same complete model was leaked publicly. So the entire AI open source community now had a complete large-scale model. Then followed the "tremendous outpouring of innovation." ## Coulda Shoulda The document makes clear that Google coulda, shoulda known this was inevitable, once the open source community got total access to both code and model weights of a full-scale language model, because just six months earlier, a virtually identical emergence occurred in the image generation (art AI) space: > ## Why We Could Have Seen It Coming > > In many ways, this shouldn't be a surprise to anyone. The current renaissance in open source LLMs comes hot on the heels of a renaissance in image generation. The similarities are not lost on the community, with many calling this the "[Stable Diffusion moment](https://simonwillison.net/2023/Mar/11/llama/)" for LLMs. > > In both cases, low-cost public involvement was enabled by a vastly cheaper mechanism for fine tuning called [low rank adaptation](https://arxiv.org/abs/2106.09685), or LoRA, combined with a significant breakthrough in scale ([latent diffusion](https://arxiv.org/abs/2112.10752) for image synthesis, [Chinchilla](https://arxiv.org/abs/2203.15556) for LLMs). In both cases, access to a sufficiently high-quality model kicked off a flurry of ideas and iteration from individuals and institutions around the world. In both cases, this quickly outpaced the large players. > > These contributions were pivotal in the image generation space, setting Stable Diffusion on a different path from Dall-E. Having an open model led to [product integrations](https://github.com/AbdullahAlfaraj/Auto-Photoshop-StableDiffusion-Plugin), [marketplaces](https://civitai.com/), [user interfaces](https://github.com/AUTOMATIC1111/stable-diffusion-webui), and [innovations](https://stablediffusionweb.com/ControlNet) that didn't happen for Dall-E. > > The effect was palpable: [rapid domination](https://trends.google.com/trends/explore?date=2022-08-01%202023-04-10&q=Stable%20Diffusion,Dall-E&hl=en) in terms of cultural impact vs the OpenAI solution, which became increasingly irrelevant. Whether the same thing will happen for LLMs remains to be seen, but the broad structural elements are the same. The rapid domination Google Trends chart cited above focuses on Google's Dall-E being rapidly overtaken by upstart Stable Diffusion; but let's add Midjourney to the chart for even more fun: