On this page
Key takeaways
- The best evaluation questions test boundaries, not features.
- Ask what the vendor cannot do to you: move funds, see keys, change behaviour without notice.
- Inspect the event contract, the failure modes, and the upgrade path before the demo.
- A vendor that answers 'not implemented yet' honestly is safer than one that says yes to everything.
Feature checklists are the wrong way to evaluate software that can move value. Every gateway supports payments; the differences are in the boundaries, the failure behaviour, and how honest the vendor is about limits. This checklist is organised around those.
Use it on any gateway, including ours. If a vendor will not answer a question in writing, that is an answer.
How to use this list
Boundaries and control
Ask these first
- Where do production signing keys live, and who can use them?
- Can the vendor move funds on my behalf, even in an emergency?
- Does anything require the vendor to be reachable for payments to work?
- Which parts of the payment record can I query directly?
- Is there any hosted fallback or 'convenience' mode that involves custody?
Why these come first
If the boundary is unclear, everything else — security, compliance, incident response — is built on sand.
The event contract
Inspect the webhooks
- Are events signed, and can I verify without calling the vendor?
- Are event names stable and versioned, with a deprecation policy?
- Is delivery at-least-once, with a documented retry schedule?
- Can I replay a missed event without forging one?
- What does the dashboard show when a delivery has failed twice?
Networks, tokens, and honesty
Scope and limits
- Which networks and tokens are implemented and tested — not planned?
- How is confirmation policy configured, and can it vary by amount?
- What happens with underpayments, overpayments, and wrong-network payments?
- How are expired payments that later receive value handled?
- Which RPC providers are supported, and can I run my own?
Operations
Running it at 2am
- Is there a runbook for the ugly cases, or only an install guide?
- How are patch releases announced, and how do I apply them safely?
- What are the backup and restore steps for the database?
- Can I rebuild the gateway from backups without the vendor?
- What metrics and logs can I export to my own tooling?
Commercial terms
The fine print
- Is the licence one-time or recurring, and what does it include?
- How long are patch releases included?
- What is out of scope: on-call, legal, compliance, custody?
- What happens to my deployment if the vendor stops trading?
- Who owns customisations built for me?
Red flags and green flags
Red flags
- Yes to every network you ask about
- Vague answers about who holds keys
- No replay path for missed events
- Demo-only screenshots, no runbook
- Pressure to skip staging
Green flags
- A clear 'not implemented yet' where true
- The boundary stated in one plain sentence
- Signed, versioned, replayable events
- A runbook and a fire-drill offer
- Encouragement to test in staging first
Score it
| Area | Weight (1–5) | Vendor A | Vendor B |
|---|---|---|---|
| Boundaries and control | _ | _ | _ |
| Event contract | _ | _ | _ |
| Networks and honesty | _ | _ | _ |
| Operations | _ | _ | _ |
| Commercial terms | _ | _ | _ |
A two-week proof of concept plan
A demo shows the happy path. A proof of concept should try to break the product. Two weeks is enough to learn most of what you need if you plan the tests before you start.
| Days | Focus | What you do | What you record |
|---|---|---|---|
| 1–2 | Deploy | Install in staging using only the documentation | Where the docs were wrong or missing |
| 3–4 | Happy path | Create, pay, confirm, receive events | Latency and the shape of every event |
| 5–6 | Failure injection | Fail your webhook, expire a link, block the provider | Whether states and alerts match the docs |
| 7–8 | Security review | Walk the boundary questions; inspect key handling | Any hidden custody or fallback |
| 9 | Restore | Restore a backup into a clean environment | Time to recover and steps missing |
| 10 | Upgrade | Apply a patch release in staging | Downtime and migration behaviour |
The rule of the PoC
Use only the documentation you were given. Every time you have to ask a question that the docs should have answered, write it down. That list is a preview of your support experience.
Threat-modelling questions to ask the vendor
A structured lens helps. Walk each element of the system and ask what an attacker could do, using the OWASP API Security Top 10 [1] and application security verification standard [2] as prompts.
| Surface | Question | What a good answer sounds like |
|---|---|---|
| API authentication | How are API credentials issued, scoped, and revoked? | Scoped keys, rotation, and an audit trail |
| Webhook endpoints | How do we verify authenticity and freshness? | Signed raw body, timestamp tolerance, dual secrets |
| Object-level access | Can one tenant or key read another's payments? | Enforced authorisation on every object |
| Rate limiting | What stops abuse of create endpoints? | Documented limits and idempotency |
| Secrets | Where do signing keys and secrets live at rest? | A secret store; never in the image or repository |
| Supply chain | How are dependencies and images updated and verified? | Signed releases, dependency scanning, changelog |
Load and failure tests worth scripting
#!/usr/bin/env bash
KEY="poc-$(date +%s)"
for i in $(seq 1 25); do
curl -s -X POST "$GATEWAY/v1/payment-intents" \
-H "content-type: application/json" \
-H "idempotency-key: $KEY" \
-d '{"reference":"poc-order-1","amount":"10.00"}' &
done
wait
# Expect: exactly one intent for reference poc-order-1 in the dashboard.- Concurrency. Create the same payment many times in parallel and count the results.
- Endpoint failure. Return 500 from your webhook receiver for an hour, then recover and check nothing was lost.
- Duplicate and reordered events. Replay events out of order and check your handler's final state.
- Provider failure. Kill the RPC provider and watch the alerts, lag indicator, and recovery.
How to run reference calls
References are most useful when you ask operators, not sponsors, and when the questions are specific. Ask to speak to the person who was on call.
Reference call script
- How long from purchase to first live payment, and what slowed it down?
- What surprised you in the first month of operating it?
- Tell me about your worst incident. How did the runbook hold up?
- How have upgrades gone? Any that hurt?
- What would you ask the vendor to change?
- Would you buy it again?
Contract clauses worth negotiating
| Clause | Why it matters | What to look for |
|---|---|---|
| Scope of support | Sets expectations for incidents | Response windows and channels in writing |
| Patch entitlement | Software ages | Length of included patch releases |
| Escrow or continuity | Vendor risk | What happens to you if the vendor stops trading |
| Custody exclusion | Legal clarity | A clear statement that the vendor does not hold funds or keys |
| Customisation ownership | Future flexibility | Who owns bespoke work and who may maintain it |
StablePay is early-access, self-hosted payment gateway software. We would rather you hold us to this list than take our word for any of it — and the product page states plainly what the software is not.
References & further reading
- 1OWASP API Security Top 10 (2023) — OWASP
- 2
- 3Secrets Management Cheat Sheet — OWASP Cheat Sheet Series
The product behind this post
StablePay
Self-hosted stablecoin payment infrastructure.
From $4,800 one-time license
About the authors
Omar Farouk
Engagement Lead, Services
Scopes and leads selected client builds, and helps startups and agencies decide what to build first.
Daniel Whitfield
Staff Engineer, Platform & Reliability
Owns webhooks, PostgreSQL, deployment, and the operational habits that keep self-hosted software boring.
Product behaviour described here reflects what is implemented and tested; anything else is marked as planned. Code samples are illustrative.
All writing