A regional insurance carrier hired a development firm on the strength of one demo. The portal looked sharp, the day rate came in under budget, and the first release shipped roughly on time. Eighteen months later, nobody inside the carrier could explain the authentication layer, and small changes to the rating logic took weeks.
That ending is far more common than vendor case studies suggest, and it almost never comes from hiring an incompetent shop. It comes from evaluating the wrong things. Procurement teams get very good at comparing rates and timelines, because those numbers line up neatly in a spreadsheet, while the factors that decide whether a system is still maintainable in year three hide inside architecture choices and contract clauses nobody reads twice.
The useful shift is to stop asking whether a partner can build the thing and start asking what your organization will actually own once the invoices stop. Six areas carry most of that weight: technical depth, security practice, scalability, integration capability, support, and long-term ownership. Each one has a question behind it that no pitch deck will answer on your behalf.
Every firm claims deep expertise in whatever stack you named in the RFP. The way to test the claim is to stop asking about successes and ask about a decision they got wrong. A team with real depth will tell you about the time they picked the wrong message queue, what it cost to unwind, and how the decision process changed afterward. A team selling you a logo wall will deflect into another case study.
Tooling deserves a direct question too, since the way a partner builds internal software now looks nothing like it did five years ago. Plenty of delivery teams prototype admin panels and workflow tools on an enterprise vibe coding platform and hand over a maintainable app in days rather than sprints. That is a real advantage when it is deliberate, and a liability when a vendor uses generated scaffolding without being able to explain a line of it.
Security is where buyer questions tend to go soft, usually because the answers are technical and the evaluator is not. The fix is to ask for evidence rather than assurances. A partner handling regulated data should be able to produce an independent attestation, and the SOC suite of services exists precisely so that buyers can assess outsourcing risk without auditing a vendor themselves. A Type 2 report covering a real observation window tells you considerably more than a security policy PDF written last quarter.
The same discipline applies on your side of the boundary. Access granted to an outside team needs a continuous evidence trail, so a client audit or an internal review can be answered from records rather than from memory. Build that expectation into the contract while you still hold leverage, because retrofitting it onto a live engagement is slow and awkward.
Scalability conversations collapse into buzzwords unless you anchor them to your own numbers. Bring the actual shape of your load: peak concurrent users, the quarterly batch that triples volume for six hours, the report that nobody can wait forty seconds for. Then ask what breaks first and what it costs to fix. A partner who has run systems at your scale will name a bottleneck immediately, because they have met it before.
Integration is usually the larger risk, and it gets the smaller share of the evaluation. Enterprise software rarely lives alone. It talks to an ERP that was customized in 2011, a CRM with three sources of truth, and a data warehouse that someone is mid way through replacing. Ask how the partner documents an interface before building against it, and whether they design the contract first or discover it by reading someone else's code.
A practical signal is whether the team works from machine readable specifications rather than tribal knowledge. The OpenAPI Specification exists so that an interface can be described, versioned, tested, and handed to another team without a tour guide, and partners who work that way tend to produce integrations your own engineers can pick up later. Partners who do not will leave you dependent on a conversation you can no longer have.
Support terms get skimmed because they feel like boilerplate, which is exactly why they deserve a slow read. Response time is only half of it. Ask what counts as a severity one incident, who makes that call, whether coverage follows your business hours or the vendor's, and what the escalation path looks like at two in the morning on a holiday weekend. Then ask what support costs in year two, since introductory terms have a habit of expiring.
Ownership is the clause that decides your future options. Confirm in writing that you own the source, the infrastructure definitions, the CI pipelines, the test suites, and the documentation, not just a compiled artifact and a license to use it. Ask for a software bill of materials, and check how third party components are licensed, because the OSI approved license list draws real distinctions between permissive and reciprocal terms that can constrain how you redistribute or commercialize what you paid for.
Then insist on a handover that is rehearsed rather than promised. The strongest arrangement I have seen is a scheduled dry run: a few months before the engagement winds down, your internal engineers ship a non trivial change with the vendor watching and not touching the keyboard. Whatever goes wrong in that exercise is a gap you still have time and budget to close, and you learn it while someone is still contractually obliged to help.
The carrier from the opening eventually recovered, though it took a second vendor, a six month remediation, and an uncomfortable board conversation about an asset nobody could value. The expensive part was never the build. It was the gap between what they thought they had bought and what they could actually operate, and that gap was fully visible in the proposal if anyone had known where to look.
Evaluate for the day after delivery, not the day of it. The partner who looks slightly more expensive because they insist on documentation, test coverage, and a real handover plan is usually the cheaper choice measured across the life of the system. That is the comparison worth making, and it is the one most evaluation scorecards still leave out.