AI Research Residency
Ten modules, run in public, from writing a transformer by hand to a capstone somebody else can pick apart. Self-directed, not a program I was admitted to, and run alongside a full-time MBA. The curriculum and roadmap are the source of truth. This page follows them.
- Progress
- 1 of 10 modules active
- Built so far
- A transformer and a policy iteration walkthrough
- Reading now
- GRPO and RL post-training
Words I had to actually learn
Everything below uses these. Nobody defines them and they are all simpler than they sound.
- Word by word
- A model does not compose a sentence and then write it. It guesses the next word, adds it, and guesses again. Every impressive thing you have seen is that, on a loop, very fast.
- Non-deterministic
- Ask the same question twice and you can get two different answers. There is deliberate randomness in how the next word gets picked. This is why you cannot test AI the way you test normal software, where the same input always gives the same output.
- Benchmark
- A fixed set of questions with known answers that everybody runs, so two models can be compared on the same thing. Good for comparing models. Bad at telling you whether your product works for your users.
- Eval
- Your own test set for your own product. Realistic requests, run through your system, graded. The thing a benchmark cannot do for you.
- Rubric
- The written rules that decide what counts as a good answer. Usually not written down, which is the problem.
- Hill climbing
- Tuning until the number goes up. Sometimes that is real progress and sometimes the system has just learned the test. From the outside the two look identical, which is why the test has to be built carefully.
The ten modules
Every module has to end in something someone else can look at. Code that runs, a result you can argue with, or a writeup that says what did not work. Each module breaks into days, and a day is one serious working session, not a calendar day. The days I have done are checked. The rest is a plan, and says so.
Transformers From Scratch
Attention, tokenization, decoding, and a small transformer you can actually run.
- ✓ Day 1Write the whole thing from memory and notes: config, embeddings, causal attention, feed-forward blocks, generation. The reference implementation exists so I have a destination, not so I can peek at it. Ships A scratch transformer that passes the smoke tests.
- Day 2Trace one batch through every tensor shape by hand, then change heads, width, and context length until something breaks. The shapes I cannot predict are the parts I do not understand yet. Ships Shape notes saying which combinations are valid and why.
- Day 3Train it on a short corpus and add top-k and nucleus sampling. Same code before and after, and only one of them can write. Ships Generation samples from before and after training.
- Day 4Visualize the attention weights and write the walkthrough of every class. If I cannot explain a line I wrote, I copied it. Ships Annotated walkthrough plus the architecture diagram, explained.
Artifact Minimal transformer implementation and diagram
Logan's take I built one. Half a million moving parts. The thing that stuck was not the architecture, it was that the worst bug did not crash anything: I forgot to trim one array and the model trained happily on slightly wrong information for an hour. Most AI failures look like that.
Training and Evaluation
Optimization, loss curves, keeping experiments honest, and comparing two models fairly.
- Day 1Build the smallest training loop that is still honest: batching, optimizer steps, validation, checkpoints that resume. Every corner I cut here becomes an experiment I cannot trust later. Ships A training script the module 1 transformer can run on.
- Day 2Log train and validation loss on a schedule and learn to read the curves. Underfitting, overfitting, and instability all look like a wobbly line until you have stared at a few. Ships Loss curves for a baseline run, with a diagnosis.
- Day 3Run three experiments changing one thing at a time. Slower than changing five things at once, except that changing five tells you nothing. Ships A comparison table saying what helped and what hurt.
- Day 4Write the failure analysis for the weakest run and the model card for the best one. A result nobody can rerun is not a result, it is an anecdote. Ships A report with the command, config, seed, and commit hash to rerun it.
Artifact Training loop and experiment report
Why it is on the list This is where hill climbing becomes a real risk. If I cannot rerun my own experiment and get the same answer, nothing downstream of it means anything.
Data, Embeddings, and RAG
Dataset design, vector search, retrieval quality, and answers grounded in real sources.
- Day 1Build the ingestion pipeline over the residency notes themselves: collect, clean, chunk, embed. Run it twice, and if the index comes out different, day one is not over. Ships An ingestion script that produces the same index twice.
- Day 2Write the golden questions and their source passages before testing anything, then compare two chunking strategies against them. Expectations written after the fact have a way of matching the output. Ships Retrieval precision at top-k for both strategies.
- Day 3Add reranking, context assembly, and citations that link back to the pages the answers came from. An answer I cannot trace to a source is the model writing fiction, confidently. Ships A Q&A assistant that cites where its answers came from.
- Day 4Add an abstention rule and sort the retrieval failures into a taxonomy. The questions the corpus cannot answer are the important test cases, because those are the ones it will answer anyway. Ships An error taxonomy plus the final retrieval metrics.
Artifact Grounded retrieval application
Why it is on the list Most products calling themselves AI are mostly search with a model on the end. The quality is in the retrieval, not the writing.
Agents and Tool Use
Tool contracts, planning loops, memory, and agents that survive things going wrong.
- Day 1Design three typed tools for a research assistant agent, with validation tests before any model gets to call them. A tool contract is a promise about the worst input, not the best one. Ships Tool schema docs and passing validation tests.
- Day 2Build the loop: plan, act, observe, recover. Log every step, because the trace is the only honest witness to what the agent actually did. Ships One task completed end to end, with the full trace.
- Day 3Run the agent on five tasks and classify every failure. Rerunning until it works is how you get a demo. Writing down why it did not is how you get a product. Ships A failure taxonomy with a mitigation for each class.
- Day 4Add retry policies and a human approval boundary. The interesting question is not what the agent can do, it is what it should stop and ask about. Ships A working demo plus the written policy for risky actions.
Artifact Tool-using agent prototype
Why it is on the list The eight transcripts I write about are agent traces. I want to be building them, not just grading them.
Multimodal Systems
Vision and language together, audio pipelines, and interface design.
- Day 1Pick one modality that is not text and build the smallest pipeline that reasons over it. Narrow and working beats broad and imaginary. Ships A working demo on real inputs, however small.
- Day 2Write the 25 examples with expected behavior before running any of them, while I still do not know what the system will say. Ships The eval set, inputs and expectations included.
- Day 3Compare zero-shot and few-shot prompting on the same task and collect the failures. Five documented failure modes teach more than fifty screenshots of it working. Ships At least five failure modes, with examples.
- Day 4Add a confidence convention and write down where the system should not be trusted. Every demo shows where it works. The useful document is the other one. Ships The error taxonomy and the do-not-trust note.
Artifact Vision-language or audio-language demo
Why it is on the list The proof of mastery here is an error taxonomy across 25 examples, which is the part I care about more than the demo.
Evaluation and Observability
Benchmarks, judge models, tracing, error taxonomies, and gates that block a bad release.
- Day 1Build a baseline eval for one earlier project, plus ten adversarial cases. Cases where it might do too much, and cases where it might refuse the thing it should have done. Ships A versioned eval dataset.
- Day 2Stand up an LLM judge and compare its scores against manual labels before believing a word it says. A judge is a model too, which means it needs an eval too. Ships A calibration table: judge versus human, case by case.
- ✓ Day 3Grade a set of traces, have someone else grade the same set, and measure where we split. The disagreement is not noise. It is the missing sentence in the rubric. Ships A rubric plus the list of cases where careful graders still split.
- Day 4Wire a regression gate that blocks a known bad change, and write the release criteria it enforces. A gate that has never caught anything is a decoration. Ships A gate that provably catches the planted regression.
Artifact Evaluation harness
Logan's take This one I am doing at work rather than waiting for. Eval generation and rubric design is the job at Thoughtful Labs, and it is what most of my writing is about.
Alignment and Safety
Preference learning, red teaming, policy design, and writing down the risks you are accepting.
- Day 1Write a safety policy for a system I built earlier, concrete enough that a test can fail it. A policy no test can fail is a press release. Ships The policy, with examples of allowed and disallowed.
- Day 2Build the red-team set: benign, borderline, and disallowed, in honest proportions. Mostly benign is the part everyone skips, and it is where over-refusal hides. Ships The red-team dataset.
- Day 3Run it and count both ways of being wrong: the harmful thing allowed and the harmless thing refused. Only one of those shows up in an incident report, which is exactly why I have to count the other. Ships False positive and false negative rates, side by side.
- Day 4Write the mitigations and the risk register, including the risks I am accepting rather than fixing. Writing those down is the difference between a decision and a hope. Ships A risk register with severity, likelihood, and residuals.
Artifact Safety policy and red-team suite
Why it is on the list The failure I keep seeing is the boring one: a model refuses something harmless and everyone calls that safe. Unhelpfulness is a failure too, it is just an easy one to defend.
Research Methods
Reading papers properly, replication, ablations, and research memos.
- Day 1Read ten papers for claims, methods, assumptions, and evidence. Not the abstracts. The abstract is the paper's marketing. Ships An annotated bibliography.
- Day 2Pick one claim to reproduce and define success before running anything. Criteria written after the results have a suspicious habit of being met. Ships A reproduction plan with compute constraints.
- Day 3Run the reproduction and log everything, especially the misses. A replication that only records the hits is hill climbing with extra steps. Ships An experiment log and result table.
- Day 4Add one ablation the paper did not run, then write the memo: expected, observed, and the gap. The gap is the finding. Ships A research memo with limitations and next steps.
Artifact Paper reproduction
Why it is on the list Reproducing somebody else's result is the cheapest way to find out whether I actually understood it.
Systems and Deployment
Serving, latency, monitoring, cost, and shipping releases.
- Day 1Put one earlier project behind a stable API and deploy it. Works on my machine and deployed are two different kinds of true. Ships A running service with a documented interface.
- Day 2Add structured logs for every request and model call, then run a small load test. Whatever numbers come back are the ceiling on the experience, no matter what the model can do. Ships A latency, throughput, and cost report.
- Day 3Add caching, retries, and fallbacks, and measure what each one actually buys. Every reliability feature is a claim until there is a before-and-after number. Ships Before-and-after numbers for cost and latency.
- Day 4Write the runbook: the common failures, what the alerts mean, how to roll back. The test of a runbook is whether it works at two in the morning for someone who is not me. Ships An operations note and a release checklist.
Artifact Hosted inference service
Why it is on the list Cost and latency are product decisions wearing engineering clothes. They decide what the experience can be.
Capstone
One complete piece of work: a demo, an evaluation, and a write-up.
- Day 1Turn the idea into a question an experiment can lose. A proposal that cannot fail is not a proposal, it is an announcement. Ships The one-page proposal and milestone plan.
- Day 2Run the baseline before building the interesting part, so there is something honest to beat and no way to move the goalposts after. Ships Baseline results, written down first.
- Day 3Build the system the question needs and nothing more. Scope is the capstone's real adversary. Ships The working system.
- Day 4Evaluate it with ablations, removing one piece at a time. If the result survives losing its favorite component, it might be real. Ships Evaluation results and reproducible scripts.
- Day 5Write it up paper-style and record the demo: claims, evidence, limitations, future work. The limitations section is the part that proves the rest is honest. Ships The write-up, the demo, and the final presentation.
Artifact Portfolio research project
Why it is on the list The whole point of the sequence. If it does not end in one artifact somebody else can pick apart, it did not happen.
The longer writeup is Your Eval Score Is Measuring You. Everything else is in the repository.