# Twenty questions to ask any AI customer support vendor > Twenty questions to send an AI support vendor before you sign, with the answer that should end a bad evaluation for each one. - **Published:** September 1, 2026 - **Category:** Guides - **Author:** Udit Goenka - **URL:** https://communicate.so/blog/ai-support-vendor-questions --- > **TL;DR:** Most AI support vendor evaluations run on a feature comparison sheet, which is exactly what a polished sales deck is built to survive. A sharper evaluation runs on twenty specific questions, each with a wrong answer that should end the conversation on the spot. This guide lists all twenty, grouped into data and security, accuracy and grounding, compliance and audit, and pricing and lock-in, with the disqualifying answer named for each one. None of the questions are trick questions; every one of them has a plain, verifiable answer that a legitimate vendor can give in a sentence. A vendor that hedges, deflects to a sales engineer, or points you to a whitepaper instead of answering directly has just told you something worth knowing before you sign a year-long contract. --- A feature comparison sheet tells you what a vendor claims their product does. It does not tell you what happens when a customer disputes an answer, when your data protection team asks where conversation data lives, or when you want to leave after year one. Those are the moments that actually determine whether an AI support vendor was the right choice, and none of them show up on a features page. This guide is built around twenty direct questions, organized into four groups of five, that surface exactly those moments before you sign anything. Send them in writing, ask for written answers, and treat a vague or deflected answer as data in itself. If you want the deeper mechanics behind any single question, [security](/security) and [AI customer support pricing](/blog/ai-customer-support-pricing) cover the two topics this list draws on most. None of the twenty questions require a technical background to ask or to evaluate the answer to. They are written so a support lead, an operations owner, or a founder without an engineering team can send them directly and judge the responses without needing a specialist to translate. That is deliberate: a vendor evaluation gatekept behind technical jargon tends to favor whichever vendor talks the fastest, not the one that actually built the safer product. ## Why a questionnaire beats a feature comparison A feature comparison rewards whoever wrote the most convincing marketing copy. Every vendor in a competitive space claims accurate answers, strong security, and responsive support, and a checklist of claims cannot distinguish a vendor that has actually built the underlying system from one that is describing a roadmap item as if it already shipped. A direct question forces specificity in a way a feature list cannot. Asking whether an agent's retrieval sources are logged per response is a different question from asking whether the product is auditable, and the second version lets a vendor answer yes without committing to anything checkable. The gap between those two questions is where a lot of bad AI support purchases get made, a pattern also visible in how vague marketed deflection numbers compare with real enterprise medians, as [Lorikeet](https://www.lorikeetcx.ai/articles/resolution-rate-ai-customer-support-benchmarks-2026) documents. The pressure to buy fast makes this worse. [DigitalApplied reports](https://www.digitalapplied.com/blog/ai-customer-support-statistics-2026-adoption-roi-data) that 91 percent of CX leaders are under executive pressure to deploy AI, according to Gartner data, and a team moving on that timeline is exactly the team most likely to sign based on a confident demo rather than a verified answer. The twenty questions below are built to be answerable in a single sitting, so speed and rigor do not have to trade off against each other. ## How to use this list Send all twenty questions in writing before a sales call, not during one. A written response gives you something to hold the vendor to later, and it filters out the vendors unwilling to commit specifics to text, which is itself useful information. During the follow-up call, ask to see two or three of the answers demonstrated live rather than described, since a live demo of an audit log or an escalation path is much harder to fake than a slide describing one. Do not treat every hedge as automatically disqualifying. A vendor that says a specific retention window depends on your plan tier is being accurate, not evasive, as long as they follow up with the actual number for the tier you are evaluating. The disqualifying answers below are specifically the ones that avoid the substance of the question entirely. ## Data and security: questions one through five ![Checklist of five security questions beside a padlock icon representing customer data](https://communicate.so/blog/ai-support-vendor-questions-checklist-security-beside-padlock.webp) **1. Where is our conversation data stored, and can we choose the region.** A vendor that cannot name a specific data residency answer, or that says 'it depends' without following with an actual list of available regions, has not built region-aware storage. That is the disqualifying answer: an inability to name where your customers' data physically sits. **2. Is conversation data encrypted at rest and in transit, and with what standard.** Any credible vendor answers this in one sentence, typically AES-256 at rest and TLS in transit. A vendor that redirects to a general 'we take security seriously' statement instead of naming a standard has told you the standard is either weak or that nobody on the call actually knows the answer. **3. Do you have a current SOC 2 report, and can we see it under NDA.** A vendor actively pursuing or holding SOC 2 Type II attestation can produce the report or a bridge letter on request. A vendor that says SOC 2 is 'on the roadmap' with no committed date is telling you compliance was not a design constraint from the start, which tends to show up elsewhere in the product too. **4. Who at your company can access our customer data, and how is that access logged.** A specific answer names role-based access controls and an internal access log. The disqualifying answer is any version of 'our whole engineering team has access for debugging,' since that means your customer data has a much larger exposure surface than the security page implies. **5. What happens to our data if we cancel.** A vendor with a real answer names a deletion timeline, typically 30 to 90 days, and confirms it in the contract, not just verbally. The disqualifying answer is any version of 'we retain it indefinitely for product improvement' offered as a default with no opt-out. ## Accuracy and grounding: questions six through ten ![Five questions surrounding a chat bubble connected to a knowledge base document](https://communicate.so/blog/ai-support-vendor-questions-surrounding-chat-bubble-connected.webp) **6. What does the agent do when it cannot find a confident answer in our content.** A grounded system escalates to a human or says it does not know. The disqualifying answer is any version of 'it generates its best guess,' which is a direct admission that the product will hallucinate rather than defer, the exact failure pattern covered in [reducing AI hallucinations in support](/blog/reduce-ai-hallucinations-support). **7. Can the agent cite the specific source document it used for an answer.** A grounded, [RAG-based](/blog/rag-for-customer-support) architecture can point to the passage it pulled from. A vendor that cannot demonstrate source citation live, and instead describes it as a future feature, is asking you to trust an answer you cannot independently verify. **8. How often is our content re-indexed after we update our help center.** A specific number, whether that is near real time, hourly, or nightly, tells you how stale an answer can get after you fix a documentation error. 'We recommend re-indexing periodically' with no default schedule is the disqualifying answer, because it means outdated content will keep answering questions until someone remembers to intervene manually. **9. What is your measured resolution or deflection rate, and how is it defined.** A vendor with a real number defines it precisely, since [Zendesk enterprise medians](https://www.lorikeetcx.ai/articles/resolution-rate-ai-customer-support-benchmarks-2026) sit around 41.2 percent while some vendors market claims as high as 80 percent using a looser definition of resolution. The disqualifying answer is a headline number with no definition of what counts as resolved attached to it. **10. Can we test the agent against our own historical tickets before signing.** A vendor confident in their retrieval and grounding will support a pilot against real, anonymized historical conversations. A vendor that insists on a generic sandbox demo instead of your own data, or delays a pilot indefinitely, is avoiding a test they expect to fail. | Question group | Ask in writing first | Verify live in a demo | | --- | --- | --- | | Data and security (1-5) | ✓ | ✓ access logs and encryption settings | | Accuracy and grounding (6-10) | ✓ | ✓ live source citation on a real query | | Compliance and audit (11-15) | ✓ | ✓ export a sample audit log | | Pricing and lock-in (16-20) | ✓ | ✗ contract terms, not a live demo item | ## Compliance and audit: questions eleven through fifteen ![Five questions beside a document icon labeled audit log](https://communicate.so/blog/ai-support-vendor-questions-beside-document-icon-audit.webp) **11. Do you log which model version generated each response.** Model and prompt versions change over time, and a vendor that cannot tie a historical answer to the exact configuration that produced it cannot help you explain a disputed response months later, the exact gap covered in [the AI support audit trail](/blog/ai-support-audit-trail). 'We log the conversation, not the model version' is the disqualifying answer. **12. Can we export the full audit trail, not just the transcript.** A real export includes the retrieved sources, model version, and any human review alongside the transcript. An export limited to plain conversation text is a transcript export, not an audit trail export, and the difference matters the first time a regulator or a customer disputes an answer. **13. How does the product support disclosure requirements like the EU AI Act.** A vendor operating in this space should be able to name a configurable disclosure message shown at the start of a conversation. 'We have not looked into that' from a vendor selling into the EU is a disqualifying answer, since it means regulatory obligations were not part of the product's design brief. **14. How long are audit logs retained by default, and is that configurable.** A specific default, commonly a year or more, plus the ability to extend it for regulated customers, is the answer you want. A vague 'we keep logs for a while' with no number attached means you cannot plan a compliance retention policy around the vendor's actual behavior. **15. What happens during a human handoff, and is that transition logged.** A well-built handoff logs exactly when and why a conversation moved from AI to a person, tied to the same conversation record. If handoffs are not logged as discrete events, you lose the ability to measure how often the AI actually resolves things versus quietly passing the hard cases along. ## Pricing and lock-in: questions sixteen through twenty ![Five questions surrounding a contract document and a pricing tag](https://communicate.so/blog/ai-support-vendor-questions-surrounding-contract-document-pricing.webp) **16. Is pricing based on seats, conversations, resolutions, or a flat platform fee.** Every model is legitimate, but the vendor should be able to state it plainly and show how it maps to your expected volume. 'It varies, let us build you a custom quote' before answering the basic pricing model is a stalling tactic more than an answer, a pattern the [AI customer support pricing](/blog/ai-customer-support-pricing) guide breaks down by model. **17. What is the all-in cost at our expected volume, including any overage fees.** A specific number, not a range with no floor or ceiling, is what a serious vendor gives once they know your rough volume. Overage fees buried in a separate schedule, revealed only after signing, are one of the most common sources of [AI customer support cost](/blog/ai-customer-support-cost) surprises reported by teams that skipped this question. **18. Can we export our conversation history and knowledge base configuration if we leave.** A vendor with nothing to hide supports a clean export in a standard format. A vendor that says data export is 'handled case by case' or requires a professional services engagement to retrieve your own data is describing a lock-in mechanism, not a limitation. **19. What is the minimum contract term, and what is the penalty for leaving early.** Month-to-month or a clearly stated annual term with a named early-termination cost are both fair answers. A vendor that cannot state the term without pulling up a contract mid-call likely has terms they would rather you discover after signing than before. **20. Who owns the AI agent's configuration and prompts, us or you.** You want a direct answer that the configuration, prompts, and knowledge base mappings are yours to export and reuse elsewhere. A vendor that treats the configuration as proprietary intellectual property you cannot take with you is building a moat out of your own setup work. ## Red flags that should end the evaluation immediately Three patterns across all twenty questions matter more than any single answer. The first is any answer that redirects to a sales engineer instead of committing to a number in writing, since the follow-up rarely arrives with more specificity than the original dodge. The second is a security or compliance answer that references a roadmap item as if it were shipped, which is the single most common way teams discover a gap only after signing, a risk covered further in [AI agent guardrails](/blog/ai-agent-guardrails). The third is a live demo that only works on a curated, pre-tested example. Ask to run one of your own real, messy support questions through the agent during the demo, not a question the sales team has clearly rehearsed. A vendor that resists that request, or steers you back to their prepared script, is protecting a weaker product than the demo suggests. Watch, too, for enthusiasm that outruns the specifics. A vendor that answers every question with energy but no numbers, no named standards, and no committed dates is optimizing for how the call feels rather than what it proves. The written answers exist precisely to strip that layer away, since a document either states a retention period or it does not, and a strong [AI customer support software](/blog/ai-customer-support-software) evaluation should end up favoring the vendor with the more boring, more specific paperwork over the one with the better call energy. ## How to run the evaluation call once you have the written answers Structure the call around verifying, not re-asking. Pick five of the twenty questions where the written answer was vague, specific to your industry, or otherwise worth confirming live, and ask the vendor to demonstrate rather than restate them. A vendor comfortable with scrutiny will welcome this; a vendor uncomfortable with it will try to steer the conversation back toward a scripted walkthrough, which is itself useful signal heading into a [shared inbox](/shared-inbox) and [AI agent](/ai-agents) evaluation that will determine how your team works for the next year. Bring someone from outside the buying team to that call if you can, ideally whoever will own the day-to-day relationship once the contract is signed. Sales conversations optimize for the person with signing authority, and a support lead sitting in on the technical portion of the call often catches a vague answer that a procurement-focused buyer would let pass, simply because they know which details actually matter once the tool is live. Close the evaluation with a written summary of where each finalist stood on all twenty questions, not just a gut impression from the last call. Memory of a good demo fades faster than the specifics that made it good, and a written scorecard, revisited a week later with a clear head, tends to surface a different winner than the one who felt best in the room, particularly once [pricing](/pricing) and lock-in terms are weighed against the security and grounding answers rather than considered separately. ## Frequently asked questions ### How many of the twenty questions should a good vendor answer without hesitation All twenty, in writing, within a few business days. None of these are trick questions or edge cases; every one has a plain, checkable answer that a vendor who built the product correctly can give without consulting engineering. Repeated hesitation across multiple questions is a stronger signal than a single vague answer. ### Is it reasonable to ask for a pilot against our own historical tickets Yes, and most credible vendors will support it, since it is the fastest way for both sides to see real performance rather than a marketing number. A vendor unwilling to run a pilot against your own anonymized data is asking you to take their resolution claims on faith, which is exactly what this questionnaire is designed to avoid. ### Should pricing questions come before or after the security and accuracy questions After. A vendor that fails the security or accuracy questions is disqualified regardless of price, so front-loading pricing wastes evaluation time on a vendor you were never going to select. Save the pricing and lock-in group for vendors that already cleared the first fifteen questions cleanly. ### What if a vendor answers all twenty questions well but the demo still feels weak Trust the demo. Written answers describe intent; a live demo on your own questions shows execution, and a mismatch between the two is common enough to take seriously. Run a few real support questions through the agent yourself rather than relying on the vendor's chosen examples, the same discipline covered in [how to build an AI customer support agent](/blog/how-to-build-an-ai-customer-support-agent) for evaluating your own build. ### Does SOC 2 attestation alone mean the security answers can be trusted SOC 2 is a meaningful signal, but it attests to controls at a point in time, not to every specific answer in this list. Ask to see the report and still ask the specific questions above, since a SOC 2 report can cover data handling broadly without confirming the exact access-control details a support-specific evaluation needs. ### What is the biggest mistake teams make when evaluating an AI support vendor Comparing feature lists instead of asking direct, falsifiable questions. A feature list rewards the vendor with the best marketing copy, not the one with the strongest underlying product, and the gap between the two is exactly where a bad twelve-month contract gets signed. The complaint patterns most often cited afterward, hallucinated answers, missing escalation, and poor context awareness, per [Twig](https://www.twig.so/blog/most-common-complaints-ai-customer-support-tools), are precisely the failures this questionnaire is built to catch before signing. ### Should we send these questions to every vendor on our shortlist at once Yes, in the same written format, so the answers are directly comparable. Sending different or informal versions of the questions to different vendors makes the responses harder to compare fairly, and a consistent written request also makes it easier to spot which vendor is stalling relative to the others. ### How long should a vendor take to answer all twenty questions in writing A vendor with a mature, well-documented product should return written answers within three to five business days. A delay stretching past two weeks, especially on the security and compliance questions, is itself a data point about how the vendor's internal processes work, not just a scheduling issue. ### Is it fair to disqualify a vendor for one vague answer among twenty It depends on which question. A vague answer on data residency or access controls is a harder disqualifier than a vague answer on contract term flexibility, since the first touches customer data risk and the second is a negotiable business term. Weight the twenty questions by risk rather than treating every vague answer identically. ### Do these questions apply to evaluating an in-house build versus a vendor Most of them do. Swap the vendor framing for an internal design review, and questions six through fifteen in particular, on grounding, escalation, and audit logging, apply just as directly to a team building its own agent as to one buying a vendor's. The [launch your first AI agent](/blog/launch-your-first-ai-agent) guide covers the build-side version of several of these same decisions. ### What should we do if a vendor refuses to put security answers in writing Treat that refusal as the answer. A legitimate vendor has no reason to withhold a written statement about encryption standards, data residency, or access controls, since none of that is competitively sensitive information. A refusal usually means the honest answer is weaker than the vendor wants documented. ### Can pricing based on resolutions instead of conversations hide the real cost It can, if 'resolution' is loosely defined, since a vendor billing per resolution has an incentive to count more conversations as resolved. Pair question nine, on how resolution is defined, with question seventeen, on all-in cost at your expected volume, to catch a pricing model that looks cheap per unit but expensive in aggregate, the kind of gap covered in [AI customer support cost](/blog/ai-customer-support-cost). ### Should we ask these questions again when renewing an existing vendor contract Yes. A vendor's security posture, pricing model, and audit capabilities can all change materially between an initial contract and a renewal, sometimes for the better and sometimes not. Re-running the same twenty questions at renewal catches drift in either direction before you commit to another term. ### What is the fastest single question to filter out an unqualified vendor Question six, on what the agent does when it lacks a confident answer. A vendor that admits the system generates a best guess rather than escalating or declining has told you, in one answer, that the product will hallucinate under pressure, which tends to predict how the rest of the evaluation will go. ### Do smaller AI support vendors answer these questions differently than large ones Not systematically. Size does not reliably predict the quality of an answer; a smaller vendor with a genuinely well-built product often answers these questions faster and more specifically than a larger one with more layers between the sales team and the engineers who actually know the answer. ### Is it reasonable to expect a demo on our own real support questions during a first call Reasonable to request, though not always available on a first call if the vendor needs a connected data source configured first. Ask for it as a condition of moving to a second call rather than the first one, and treat repeated deferral of this specific request as a signal worth weighing alongside the written answers, the same due diligence covered in [AI support agent implementation](/blog/ai-support-agent-implementation). ### What should the final decision weigh most heavily among all twenty questions The security and grounding groups, questions one through ten, should carry the most weight, since a failure there creates risk that pricing or contract flexibility cannot offset. A vendor that is expensive but grounded and secure is a manageable trade-off; a vendor that is cheap but hallucinates or mishandles data is not. ### Should we ask these questions before or after a vendor demo Before. Sending the list ahead of a scheduled demo gives the vendor time to prepare accurate answers rather than being caught off guard mid-call, and it also tells you before the demo starts whether the vendor engages with specifics or stalls, which changes what you watch for once the demo begins. ### What if two vendors answer all twenty questions equally well When written answers are close, the tie usually breaks on the live demo and the pilot, not on the questionnaire itself. Run both vendors against the same set of your own real support questions, covered in question ten, and compare how each handles the cases where the right answer is not obviously in the knowledge base, since that is where the underlying [AI agent](/ai-agents) quality actually shows. ### Do these questions still apply if we already have a vendor and are just adding a new feature Yes, scoped to the new feature. Adding voice support, a new channel, or a new integration to an existing vendor relationship is effectively a new purchase decision for that capability, and the accuracy, security, and audit questions apply to it just as they would to a first-time evaluation, even if the vendor relationship itself is established.