Back to the blog
Codex, Copilot and Claude Code: what to ask your software supplier
AI & automation

Codex, Copilot and Claude Code: what to ask your software supplier

Between 22 September and 1 October 2026 Anthropic, OpenAI and GitHub updated their coding agents, from models for long jobs to cloud environments and agent merge. For anyone commissioning software, six questions on review, merging and access.

8 October 2026· 7 min·by NaCode Studios
Share
In short

Between 22 September and 1 October 2026 Anthropic, OpenAI and GitHub updated their coding agents, from models for long jobs to cloud environments and agent merge. For anyone commissioning software, six questions on review, merging and access.

Between 22 September and 1 October 2026 Anthropic introduced Claude Opus 5.5, OpenAI gave Codex reusable cloud environments and Codex Security Cloud, and GitHub published its September summary for Copilot in VS Code, with scheduled automations and agent merge. If you commission software, ask your supplier who reviews the code the agents write, who can merge it into the main branch, where the agents run and what they can reach.

Updated 8 October 2026. The text rests on the sources listed at the end. Agent merge, automations and Codex Security Cloud are in preview and may change; the cost and speed figures for Opus 5.5 are Anthropic's. This text is general information and does not replace legal advice.

What is new from 22 September to 1 October

Anthropic says that at default settings Opus 5.5 costs 40% less to run than Opus 5 on typical workloads and generates output more than 30% faster. It describes the model as particularly good at long, sprawling jobs such as migrations and audits of whole codebases. Fast mode is available in Claude Code.

According to TechCrunch, the Codex cloud environments announced at OpenAI's DevDay on 29 September are more persistent and configurable and give the team a shared workspace with approved settings and permissions. Codex Security Cloud scans whole GitHub repositories on demand, on a schedule or on every new commit and prepares fixes in the cloud, with the laptop closed. OpenAI lists it as a research preview, limited to workspaces with access: it validates issues in an isolated environment when possible and asks users to review proposed patches before the pull request.

GitHub's summary of 1 October collects the September releases (VS Code 1.136 to 1.140). Automations run tasks hourly, daily, weekly or on demand. With agent merge the agent answers review comments, fixes failed checks and conflicts and reruns workflows. HydraFusion (research preview) chooses models and workflows automatically.

According to VS Code's documentation, agent merge is experimental and must be switched on; once configured, it can merge the pull request or add it to the merge queue. With the When Ready option it does so after the required checks and approvals have passed. In our view, on a production system a person's approval should always be required.

What the measurements say about time saved

METR, a non-profit research organisation, has measured how long experienced open-source developers take with and without AI. In its July 2025 study, 16 developers completed 246 real tasks in their own repositories, with AI allowed or banned at random: with AI they took 19% longer (confidence interval +2% to +39%), although they had expected to be 24% faster and believed afterwards that they had been 20% faster. METR describes those results as out of date.

The second study, started in August 2025 with 57 developers, puts time per task down 18% for the ten who were also in the first study (-38% to +9%) and down 4% for the new ones (-15% to +9%). In February 2026 METR judged it an unreliable signal: a growing number of developers declined to take part because they did not want to work without AI, which probably leads to underestimating the speedup. METR thought a larger speedup than in 2025 likely, but with very weak evidence, and announced changes to the study.

DORA's report on the return on AI investment, summarised by InfoQ on 11 May 2026, describes AI as an amplifier: the greatest returns come from the quality of the internal platform, the clarity of workflows and the alignment of teams, and without that foundation AI creates local pockets of productivity that are lost further downstream. It expects a temporary dip in productivity before the benefits (a J-curve), caused by learning, verifying generated code and adapting processes such as testing and approvals.

Six questions to ask the supplier

  1. Who reviews the code the agents write? According to InfoQ, DORA's report counts the verification of generated code among the causes of the initial dip in productivity and reports more instability in delivery with AI. We recommend that an identifiable person approves every change and that the supplier tells you how it measures defects after release.
  2. Can the agent merge changes on its own? It depends on the configuration and on the repository's rules. Ask to see the main branch's rules: which checks are mandatory and whether a person's approval is required.
  3. Where do the agents run and what can they reach? Since 7 October, local sandboxing for Copilot has been generally available and restricts the files, network and credentials that commands launched by the agent can reach. Ask which limits are active, where the source code ends up and who holds the credentials.
  4. How is security checked? Codex Security Cloud is a research preview and proposes fixes to be reviewed. The supplier should tell you which scans it uses and who decides on the fixes.
  5. What changes in the quote? The 40% from Anthropic is a cost of running the model. We think a promised reduction should come with how it was measured: ask how the supplier estimates the work, and for data from past projects.
  6. Which models do you use, and what happens if one changes? Opus 5.5 costs 20% less per token than Opus 5, and HydraFusion chooses models by itself. Ask which models the supplier works with, whether the workflow runs on others too, and whether the code stays in your repository, readable without the agent.

How we review code

For custom software we start from an analysis of the processes already in use and work in phases, with frequent, verifiable releases. Code is delivered tested, documented and structured so that another team can modify it years later: the standard applies to any code, whoever or whatever wrote it. For a client who wants rules on agents and review, we suggest writing them into the contract before starting.

We also have a review tool of our own. TrueUp is a GitHub app that says what a pull request makes possible and what it reimplements: it computes the effect profile of each unit of code (reads, writes, reachable calls), so a reviewer sees which capabilities the change introduces. It is at stable version 1.0.0 and is not yet on the GitHub Marketplace. In our view it shows where to look first in a change written by an agent. We cannot tell you in advance how much time an agent will save on your project and we do not promise a figure.

What we do not know yet

  • How much time an agent saves on a project like yours: METR rates the data of its second experiment unreliable, and the vendors' figures are their own claims.
  • How agent merge and Codex Security Cloud behave in production: GitHub and OpenAI give no error rates.
  • Who owns the generated code and how its licences and provenance are checked: the sources do not cover it and it depends on contracts and tools.

If you want us to look at your project, or at a supplier's rules on agents and review, start from our custom software page or get in touch.

Sources

Share

Got a project in mind?

Let us build it together

We can help you turn an idea into a digital product that is solid, fast and ready to grow.

Related articles