AI Research Residency

1. ✓ Day 1Write the whole thing from memory and notes: config, embeddings, causal attention, feed-forward blocks, generation. The reference implementation exists so I have a destination, not so I can peek at it. Ships A scratch transformer that passes the smoke tests.
2. Day 2Trace one batch through every tensor shape by hand, then change heads, width, and context length until something breaks. The shapes I cannot predict are the parts I do not understand yet. Ships Shape notes saying which combinations are valid and why.
3. Day 3Train it on a short corpus and add top-k and nucleus sampling. Same code before and after, and only one of them can write. Ships Generation samples from before and after training.
4. Day 4Visualize the attention weights and write the walkthrough of every class. If I cannot explain a line I wrote, I copied it. Ships Annotated walkthrough plus the architecture diagram, explained.

Artifact Minimal transformer implementation and diagram

Why it is on the list The public implementation brings embeddings, causal attention, feed-forward blocks, and generation together in one place to inspect.

Module notes

1. Day 1Build the smallest training loop that is still honest: batching, optimizer steps, validation, checkpoints that resume. Every corner I cut here becomes an experiment I cannot trust later. Ships A training script the module 1 transformer can run on.
2. Day 2Log train and validation loss on a schedule and learn to read the curves. Underfitting, overfitting, and instability all look like a wobbly line until you have stared at a few. Ships Loss curves for a baseline run, with a diagnosis.
3. Day 3Run three experiments changing one thing at a time. Slower than changing five things at once, except that changing five tells you nothing. Ships A comparison table saying what helped and what hurt.
4. Day 4Write the failure analysis for the weakest run and the model card for the best one. A result nobody can rerun is not a result, it is an anecdote. Ships A report with the command, config, seed, and commit hash to rerun it.

Artifact Training loop and experiment report

Why it is on the list This is where hill climbing becomes a real risk. If I cannot rerun my own experiment and get the same answer, nothing downstream of it means anything.

Module notes

1. Day 1Build the ingestion pipeline over the residency notes themselves: collect, clean, chunk, embed. Run it twice, and if the index comes out different, day one is not over. Ships An ingestion script that produces the same index twice.
2. Day 2Write the golden questions and their source passages before testing anything, then compare two chunking strategies against them. Expectations written after the fact have a way of matching the output. Ships Retrieval precision at top-k for both strategies.
3. Day 3Add reranking, context assembly, and citations that link back to the pages the answers came from. An answer I cannot trace to a source is the model writing fiction, confidently. Ships A Q&A assistant that cites where its answers came from.
4. Day 4Add an abstention rule and sort the retrieval failures into a taxonomy. The questions the corpus cannot answer are the important test cases, because those are the ones it will answer anyway. Ships An error taxonomy plus the final retrieval metrics.

Artifact Grounded retrieval application

Why it is on the list Most products calling themselves AI are mostly search with a model on the end. The quality is in the retrieval, not the writing.

Module notes

1. Day 1Design three typed tools for a research assistant agent, with validation tests before any model gets to call them. A tool contract is a promise about the worst input, not the best one. Ships Tool schema docs and passing validation tests.
2. Day 2Build the loop: plan, act, observe, recover. Log every step, because the trace is the only honest witness to what the agent actually did. Ships One task completed end to end, with the full trace.
3. Day 3Run the agent on five tasks and classify every failure. Rerunning until it works is how you get a demo. Writing down why it did not is how you get a product. Ships A failure taxonomy with a mitigation for each class.
4. Day 4Add retry policies and a human approval boundary. The interesting question is not what the agent can do, it is what it should stop and ask about. Ships A working demo plus the written policy for risky actions.

Artifact Tool-using agent prototype

Why it is on the list The eight transcripts I write about are agent traces. I want to be building them, not just grading them.

Module notes

1. Day 1Pick one modality that is not text and build the smallest pipeline that reasons over it. Narrow and working beats broad and imaginary. Ships A working demo on real inputs, however small.
2. Day 2Write the 25 examples with expected behavior before running any of them, while I still do not know what the system will say. Ships The eval set, inputs and expectations included.
3. Day 3Compare zero-shot and few-shot prompting on the same task and collect the failures. Five documented failure modes teach more than fifty screenshots of it working. Ships At least five failure modes, with examples.
4. Day 4Add a confidence convention and write down where the system should not be trusted. Every demo shows where it works. The useful document is the other one. Ships The error taxonomy and the do-not-trust note.

Artifact Vision-language or audio-language demo

Why it is on the list The proof of mastery here is an error taxonomy across 25 examples, which is the part I care about more than the demo.

Module notes

1. Day 1Build a baseline eval for one earlier project, plus ten adversarial cases. Cases where it might do too much, and cases where it might refuse the thing it should have done. Ships A versioned eval dataset.
2. Day 2Stand up an LLM judge and compare its scores against manual labels before believing a word it says. A judge is a model too, which means it needs an eval too. Ships A calibration table: judge versus human, case by case.
3. ✓ Day 3Grade a set of traces, have someone else grade the same set, and measure where we split. The disagreement is not noise. It is the missing sentence in the rubric. Ships A rubric plus the list of cases where careful graders still split.
4. Day 4Wire a regression gate that blocks a known bad change, and write the release criteria it enforces. A gate that has never caught anything is a decoration. Ships A gate that provably catches the planted regression.

Artifact Evaluation harness

Logan's take This one I am doing at work rather than waiting for. Eval generation and rubric design is the job at Thoughtful Labs, and it is what most of my writing is about.

Module notes

1. Day 1Write a safety policy for a system I built earlier, concrete enough that a test can fail it. A policy no test can fail is a press release. Ships The policy, with examples of allowed and disallowed.
2. Day 2Build the red-team set: benign, borderline, and disallowed, in honest proportions. Mostly benign is the part everyone skips, and it is where over-refusal hides. Ships The red-team dataset.
3. Day 3Run it and count both ways of being wrong: the harmful thing allowed and the harmless thing refused. Only one of those shows up in an incident report, which is exactly why I have to count the other. Ships False positive and false negative rates, side by side.
4. Day 4Write the mitigations and the risk register, including the risks I am accepting rather than fixing. Writing those down is the difference between a decision and a hope. Ships A risk register with severity, likelihood, and residuals.

Artifact Safety policy and red-team suite

Why it is on the list The failure I keep seeing is the boring one: a model refuses something harmless and everyone calls that safe. Unhelpfulness is a failure too, it is just an easy one to defend.

Module notes

1. Day 1Read ten papers for claims, methods, assumptions, and evidence. Not the abstracts. The abstract is the paper's marketing. Ships An annotated bibliography.
2. Day 2Pick one claim to reproduce and define success before running anything. Criteria written after the results have a suspicious habit of being met. Ships A reproduction plan with compute constraints.
3. Day 3Run the reproduction and log everything, especially the misses. A replication that only records the hits is hill climbing with extra steps. Ships An experiment log and result table.
4. Day 4Add one ablation the paper did not run, then write the memo: expected, observed, and the gap. The gap is the finding. Ships A research memo with limitations and next steps.

Artifact Paper reproduction

Why it is on the list Reproducing somebody else's result is the cheapest way to find out whether I actually understood it.

Module notes

1. Day 1Put one earlier project behind a stable API and deploy it. Works on my machine and deployed are two different kinds of true. Ships A running service with a documented interface.
2. Day 2Add structured logs for every request and model call, then run a small load test. Whatever numbers come back are the ceiling on the experience, no matter what the model can do. Ships A latency, throughput, and cost report.
3. Day 3Add caching, retries, and fallbacks, and measure what each one actually buys. Every reliability feature is a claim until there is a before-and-after number. Ships Before-and-after numbers for cost and latency.
4. Day 4Write the runbook: the common failures, what the alerts mean, how to roll back. The test of a runbook is whether it works at two in the morning for someone who is not me. Ships An operations note and a release checklist.

Artifact Hosted inference service

Why it is on the list Cost and latency are product decisions wearing engineering clothes. They decide what the experience can be.

Module notes

1. Day 1Turn the idea into a question an experiment can lose. A proposal that cannot fail is not a proposal, it is an announcement. Ships The one-page proposal and milestone plan.
2. Day 2Run the baseline before building the interesting part, so there is something honest to beat and no way to move the goalposts after. Ships Baseline results, written down first.
3. Day 3Build the system the question needs and nothing more. Scope is the capstone's real adversary. Ships The working system.
4. Day 4Evaluate it with ablations, removing one piece at a time. If the result survives losing its favorite component, it might be real. Ships Evaluation results and reproducible scripts.
5. Day 5Write it up paper-style and record the demo: claims, evidence, limitations, future work. The limitations section is the part that proves the rest is honest. Ships The write-up, the demo, and the final presentation.

Artifact Portfolio research project

Why it is on the list The whole point of the sequence. If it does not end in one artifact somebody else can pick apart, it did not happen.

Module notes