Guides · ← All guides

How to Audit Your Codebase for Retired Model IDs in 20 Minutes

Last updated: October 13, 2026 · 9 min read

By the Ingenious AI team

A retired model ID in your code needs no validation to keep working. Every call succeeds until the shutdown date — then the next call errors, and a customer finds out first. This audit finds every ID before the date does. Twenty minutes, four steps.

Set a timer. The point of the timebox is to force completeness over perfection: you are building the full list first and judging it second. Judgment is slow; listing is fast. Keep them separate and the twenty minutes is real.

Minutes 0–5: Grep the obvious places

Start with provider prefixes across your whole repo, not just the file you remember:

grep -rn "gpt-\|o1\|o3-\|o4-\|ft-" --include="*.py" --include="*.js" --include="*.ts" .
grep -rn "claude-\|gemini-\|grok-\|deepseek" --include="*.py" --include="*.js" --include="*.ts" .

Then the places people forget — config, not code:

grep -rn "MODEL" .env .env.example config/ 2>/dev/null
grep -rn "model" docker-compose.yml Dockerfile k8s/ terraform/ 2>/dev/null

And the CI layer, where model IDs hide in plain sight as "just config":

grep -rn "gpt-\|claude-\|MODEL" .github/workflows/ 2>/dev/null

Write down every match with its file and line. At this stage you're building an inventory, not fixing anything — judgment comes later, speed comes now.

One more pattern while you're here: search for prompt IDs and names, not just model IDs. If your code references pmpt_… IDs or named prompts, the model choice lives in the provider's dashboard, not your repo — flag those separately, because your grep just found a dependency it can't see inside. (And note the calendar: OpenAI's v1/prompts API shuts down November 30, 2026.)

Minutes 5–10: Catch what the code doesn't say

Some model references never appear in your repository:

• Framework defaults. LangChain and LlamaIndex default to gpt-3.5-turbo when no model is specified — code that never mentions a model at all breaks on October 23, 2026. Check your dependency versions and their default-model configs. This is the find that surprises senior engineers: the search comes back clean, and the product still breaks, because the model ID was chosen by a library.
• Dashboard-stored prompts. If your code references prompt IDs or names, the actual prompt text — and its model setting — lives in the provider's dashboard. Open the dashboard, read the model field on every stored prompt, and write them down.
• Automation panels. n8n, Make, Zapier nodes, serverless function environment variables, cron jobs, and any third-party tool consuming your API key each hold their own model setting. Grep can't reach them; your account dashboards can. Open each one and read the model field. Yes, all of them — the one you skip is the one that retires.
• Vendor SDK defaults in wrappers. If you call models through a gateway, proxy, or internal wrapper, the default model may be set three layers away from your code. Read the wrapper's config, not just your call site.

Minutes 10–15: Cross-reference your real usage

The repo tells you what's written down. The usage dashboards tell you what actually runs — and they disagree more often than anyone expects.

OpenAI's usage page can group by model over the last 30 days: those names are what you really run. Anthropic documents the same procedure for its Console: open the Usage page, click Export, and review the CSV broken down by API key and model. Do the same for every other provider you touch.

Now compare the two lists. Anything in the usage logs that wasn't in your repo grep is a shadow integration: note it, find its owner, add it to the inventory. In our experience this is the highest-yield five minutes of the whole audit — the usage dashboard routinely surfaces the integration nobody remembered: the contractor's script still running on a cron, the marketing automation, the prototype that quietly became load-bearing.

While you're in the dashboard, check one more thing: who receives the deprecation emails. The notices go to the account that holds the API key — which is sometimes an engineer who left two jobs ago, or a shared inbox nobody reads. If the recipient is wrong, fix it now. The best audit in the world doesn't help if the next announcement goes to a dead inbox.

Minutes 15–20: Classify every ID

For each ID on your list, assign one status from the Model Retirement Calendar:

• Retired — shut down already. Migrate now; every call to it is failing (or about to be).
• Announced, with a date — schedule the migration backward from the date with a buffer, and benchmark the replacement on your own prompts before switching. Don't let "we have six months" become "we have six days."
• Active — no announced retirement. Put it on a recheck schedule anyway; announcements arrive by email to whoever created the API key, which is sometimes nobody who still works there.
• Alias vs pinned — decide which failure mode you're exposed to: the pinned snapshot dies loudly on shutdown day; the bare alias drifts silently whenever the provider repoints it. (Read the two failure modes →)

Then sort the "announced" pile by blast radius: revenue-critical features first, internal tools last. That ordering is your migration plan. If October 23 taught the industry anything, it's that the teams with a ranked list moved calmly and the teams with a pile of grep output panicked.

If you find a retired ID mid-incident

Sometimes this audit happens because something is already broken — the 404s started this morning. Triage order:

1. Confirm the cause. Check the error body: HTTP 404 with "code": "model_not_found" means a retired model, not a routing bug. Check the provider's status page and deprecations page before you touch code.
2. Route to the substitute, behind a flag. Swap the ID to the provider's recommended replacement as a hotfix, but keep the eval honest — you are buying uptime, not declaring victory.
3. Run the smoke eval. Your top 10–20 production prompts against the replacement, diffed. If the outputs hold, promote the flag. If they don't, you now know exactly what's degraded instead of finding out from users.
4. Do the real migration after. A hotfix is not a migration. Schedule the full playbook — eval harness, canary, cost validation — for the replacement you just shipped under pressure.

After the audit: make it continuous

A one-off audit rots in weeks. Someone merges a new integration, a contractor pins a snapshot, a dashboard prompt gets created — and your list is stale. The durable version is a watched inventory: register every model your product depends on in one place, with an owner and a last-verified date, and get alerted when any of them gets a deprecation notice. Re-run this audit quarterly; update the inventory in the same PR as any model-reference change.

That's exactly what ModelMinder's free tier does — 3 tracked models, the public shutdown calendar in your inbox via /shutdowns.ics, and a one-click migration eval that diffs your prompts against the replacement before you move traffic. The audit finds the IDs; the radar makes sure you never have to hunt them again.

FAQ

How do I find all the AI model IDs my code uses?

Grep your source tree for provider prefixes (gpt-, o1, o3-, o4-, claude-, gemini-, grok-), check .env and config files for MODEL variables, search infrastructure-as-code and CI configs, inspect framework defaults (LangChain and LlamaIndex default to gpt-3.5-turbo), and cross-reference your provider usage dashboards — OpenAI's usage page grouped by model, or Anthropic's Console usage CSV export.

Where else can retired model IDs hide besides my code?

Environment variables, CI secrets, GitHub Actions configs, Docker and Kubernetes manifests, Terraform, n8n/Make/Zapier automation panels, serverless function configs, cron jobs, dashboard-stored prompts (OpenAI's v1/prompts shuts down Nov 30, 2026), and third-party tools consuming your API key. A repo grep alone misses all of these.

What do I do with the model IDs I find?

Classify each one: retired (migrate now), announced retirement with a date (schedule the migration and benchmark the replacement), active (recheck on a schedule), and alias vs pinned (decide which failure mode you're exposed to). Then register the list somewhere it gets watched.

How often should I audit my codebase for model IDs?

Run the full audit quarterly, and re-run the grep step in the same PR as any change that adds or touches a model reference. The audit rots fast — the durable version is a watched inventory where every entry has an owner and a last-verified date.

Start the watched inventory

After the 20 minutes: making the audit stick

A one-off audit decays. Codebases gain new model references every week — a new integration, a copied snippet, a notebook that became production. The teams that stay ahead turn the 20-minute audit into a habit with three lightweight controls.

First, decide your pinning policy and write it down. Pinned snapshots give you reproducibility at the cost of hard retirement dates; aliases give you continuity at the cost of silent behavior drift. Most production systems want pins with a calendar reminder, and aliases only in non-critical paths. The policy matters less than the fact that it's explicit — "we always pin" beats a mix nobody chose.

Second, keep the inventory alive. The spreadsheet from your audit rots within a quarter unless something updates it. A model dependency inventory that refreshes from your codebase — or at minimum a quarterly calendar reminder to re-run the 20-minute grep — keeps the blast-radius map honest. Pair it with a deprecation radar and the inventory updates itself when statuses change.

Third, gate the front door. Add the retired-model-ID grep to CI so no new reference to a dead model can merge. It's a five-line check that turns every future audit from archaeology into a glance at a dashboard.