Two things landed within a week of each other, and they tell the same story. On November 12, the services firm Genpact published a survey of 2,005 senior executives: 79% said they have AI agents running in production, but only 15% have fully tied their AI spending to profit-and-loss results. Five days later, Cognition, the company behind the Devin coding agent, launched Devin Bots, agents built to own a goal instead of finishing a single task.
Put them side by side and you get a practical lesson for any business owner: starting an AI agent is now the easy part. Pointing it at the right job, setting its limits and checking its work is where the value is won or lost. This post pulls that lesson out of the week’s best material, with the numbers and the caveats, and turns it into something you can use on Monday.

From tasks to responsibilities
Most of us use AI the way we use a calculator: we hand it one question and get one answer. Cognition’s launch post describes the limit of that approach. As they put it, a coding agent can finish every task you give it and still leave you responsible for moving the project forward: finding the next problem and checking whether the last change helped.
Their answer is to change how the job is written. In their announcement, they contrast a task with a responsibility, using software examples. The right-hand column below is our own translation for a small business, not Cognition’s:
| A task to complete | A responsibility to own | Small-business version (our example) |
|---|---|---|
| Port this feature to Android. | Keep Android at parity with iOS. | Every new website inquiry gets a helpful reply within the hour. |
| Fix the layout shift. | Keep the web app smooth as it changes. | Keep our top service pages accurate as prices and hours change. |
| Fix this onboarding error. | Get new users through setup successfully. | Get every new customer from “yes” to a scheduled first appointment. |
“A task describes the change you want. A responsibility describes the outcome.”
Cognition, Introducing Devin Bots, Nov 17, 2026
Notice what makes the right-hand versions work: each one is a result you can see, not an activity. That is the first and most useful habit to copy, whether or not you ever use a product like Devin Bots. Cognition’s co-founder Walden Yan said on X that teams are using it “to automate work to a completely new level.” That is a vendor talking about its own launch, so treat the numbers below as a description of what is possible, not a forecast for your shop.
The human doesn’t leave. The human moves.
Here is the detail that matters most. In Cognition’s own example, one person on its team handed an Android app to a Bot and let it run for 44 days. By the company’s count it merged 1,134 pull requests, and he sent just 183 messages, about one for every six. On 22 of those days he sent none. Most of his messages answered something the Bot needed a human to decide, such as a change that needed a security review. The Bot collects those moments in a “Needs You” list of decisions, reviews and permission requests, according to its documentation.
Genpact’s numbers say the same thing from the other direction. Even as AI now leads or fully runs 40% of the decisions it measured, humans still decide or audit in at least 87% of cases at every decision stage, and 92% of respondents with agents in production kept their human workforce the same size or grew it. Fortune quoted Cognition CEO Scott Wu putting it this way: “it’s always up to the human where we’re driving to.”
So the goal isn’t to remove yourself. It’s to spend your attention only where your judgment is needed: money, promises to customers, anything public, anything you can’t undo.
Why “in production” isn’t the same as “paying off”
Look at the chart above again. Genpact reports that 79% have agents live, yet only 20% use them broadly across functions, and on average just 49% of AI initiatives funded in the last 24 months delivered the value expected. Only 9% have completed a significant overhaul of how they operate. The report’s explanation is that companies deployed agents into operating models designed for human-only work, so much of the expected value hasn’t arrived.
Two honest caveats. First, these are self-reported answers from executives at companies with $1 billion or more in revenue, and Genpact sells agentic operations services, so it has a stake in the story. Second, a small business is a different animal. The upside of being small is that you can tie an agent to a number you already watch: leads answered, quotes sent, hours saved, bookings made. A company with 50 departments struggles to do that. You can do it before lunch.
Try this today
Pick one job you do over and over. Write it as an outcome in one sentence, add the number you will watch, and note the first thing the agent must ask you before doing. If you can’t write those three lines, the job isn’t ready to hand off yet.
“Right most of the time” is not good enough
Agents are not like spreadsheets. Run the same job twice and you can get two different outcomes. Anthropic’s guide to evaluating AI agents shows what that does to reliability with a simple example: an agent that succeeds 75% of the time on any single try will succeed on three tries in a row only about 42% of the time. For anything customer-facing, where every customer is a new try, that second number is the one that counts.

The takeaway isn’t “don’t use agents.” It’s that reliability has to be measured, not assumed, and two simple habits from this week’s reading make that practical for a small team.
Test on your own work, not on a leaderboard
Elvis Saravia, who runs the DAIR.AI research community, posted a roundup of ten favorite evaluation papers on November 17. His own conclusion: public leaderboards are fine for a first look, but “for decisions that matter, you need private evals built on your own tasks and data.” For you, that can be ten real examples from your own inbox or order history, run through the agent before it touches a live customer.
Search consultant Doug Turnbull described a hands-on version of the same idea. Instead of grading results on paper, he lets AI agents use a copy of a client’s app as pretend customers and watches what they click. His write-up is candid about the limits: the traffic is not a replacement for real human behavior, an agent tends to click the same obvious result every time, and it still needs a person to steer it toward realistic scenarios. That’s a good model for a small business: let an agent play customer on your site, then judge it with a human eye.
Check the steps, not only the answer
The same roundup highlights a study called Claw-Eval. In it, grading only an agent’s final answer missed 44% of the safety violations that a fuller review caught, and 13% of the robustness failures. Those numbers come from 14 models on that study’s own tasks, so don’t expect the same figures in your business. The point survives: an agent can land on the right answer by taking a wrong action along the way. Keep the log of what it did, and spot-check it the way you would a new hire’s first weeks.
Check your data before you blame the agent
Mayfield’s 2026 survey of 266 technology leaders at large companies found that 58% named data readiness and quality as their number-one blocker, and 60% described their AI governance as early-stage or nonexistent. Different scale, same lesson. If your price list lives in three places and your customer notes live in someone’s head, an agent will faithfully work from the mess. An hour spent cleaning up the one source of truth is often the best-paid hour of the project.
A one-page job description for your agent
Before any agent gets a responsibility, write it down. Here is a checklist you can copy:
- The outcome. One sentence, in plain words, tied to a number you already track.
- What it can do alone. Low-risk, reversible work: drafting, sorting, looking things up, updating internal notes.
- What it must ask first. Spending money, promising a price or a date, posting anything public, deleting anything, messaging a customer who is upset.
- Where it asks. One inbox or one thread, so you answer questions in one batch instead of all day.
- How you will check. Ten real examples before launch, then a weekly spot-check of its actions, not only its results.
- Who owns it. One named person, a fixed review rhythm, and a plain way to switch it off.
The hard parts nobody puts in the demo
- Vendor numbers are marketing numbers. Cognition’s examples are its own, and they come from software teams. Your results will differ.
- Agents are inconsistent. Plan for the 42%, not the 75%, wherever a customer is on the other end.
- Someone has to read the “Needs You” list. Handing off work doesn’t remove the decisions. It concentrates them, and a pile of unanswered questions stalls the agent.
- Savings don’t capture themselves. Genpact’s point about the P&L gap applies at any size: if you don’t decide what the saved hours will be used for, they quietly disappear.
Where to start
Pick one responsibility, give it a number, set its limits and test it on your own examples. Then widen it a little at a time. The businesses that get value from agents in 2026 are likely to be the ones that treat them like new team members: clear job, clear boundaries, regular check-ins.
If you’d like to see this approach applied to a website, that’s the idea behind Purple AI: three AI agents, Forge to build your site, Hone to rank it and Lift to convert visitors, each starting at $500 and refined by real designers. You can read more at gopurple.ai.
Sources
- Genpact, 79% of Enterprises Report AI Agents in Production, but Only 15% Have Fully Tied AI Investment to P&L Outcomes (press release, Nov 12, 2026) and Autonomy in Practice (report page).
- Cognition, Introducing Devin Bots: from tasks to outcomes (Nov 17, 2026), and Devin Bots documentation.
- Fortune, Exclusive: Cognition launches Devin Bots (Nov 17, 2026).
- Anthropic, Demystifying evals for AI agents (Jan 9, 2026).
- Doug Turnbull, Prefer synthetic traffic over LLM-as-a-judge (Nov 12, 2026), and his post on X.
- Elvis Saravia (@omarsar0), article on ten evaluation papers, on X (Nov 17, 2026), summarizing the Claw-Eval study.
- Walden Yan (Cognition), post on X about Devin Bots (Nov 17, 2026).
- Mayfield, The Agentic Enterprise in 2026 (2026 CXO survey of 266 technology leaders).
