Price Testing: Running Controlled Experiments on What You Charge

Price testing is running a controlled experiment on what you charge. It is the highest-leverage test most companies never run, and the one with the most ways to go wrong that have nothing to do with statistics.

The mechanics are ordinary: assign visitors to variations, show different prices, measure revenue. The difficulty is everywhere else — in the metric that has to change, in what happens when two customers compare notes, and in the rules about what you are allowed to vary and on what basis.

What price testing answers

A price test estimates the shape of demand near your current price. It does not find "the optimal price"; it tells you which direction to move and roughly how far, for one segment, at one moment.

Three questions are worth the effort:

• Is the current price leaving money on the table? A higher price that loses fewer customers than it gains in margin.

• Does a lower price expand the market enough to pay for itself? Rarely, in practice, but it is answerable.

• Does packaging beat price? Changing what is included at the same number is often a larger effect than changing the number, and it avoids most of the problems below.

What a price test does not answer is what a customer will pay in two years, how a competitor will respond, or how the change interacts with a sales team's discounting. Those are strategy questions the test informs rather than settles.

Three things that make price different

One: the metric flips. Conversion rate is the wrong outcome. A higher price will reduce conversion and may still be the better price. The outcome has to be revenue per visitor — total revenue divided by everyone assigned, converters and non-converters alike. Reading a price test on conversion rate reliably concludes that the lowest price is best, which is only true if the goal is to give the product away.

Two: the metric is heavy-tailed. Revenue per visitor is dominated by a small number of large orders, so its variance is far higher than a binary conversion rate. That means a price test needs substantially more traffic than a button-colour test to reach the same confidence, and a single enterprise order arriving in one arm can move the readout on its own. The guide to effect size in A/B testing covers the arithmetic, and variance reduction with CUPED is worth applying here more than almost anywhere else, because pre-period spend is strongly predictive of post-period spend.

Three: the treatment is visible and permanent-feeling. A customer who sees $49 and later sees $59 does not experience a variation. They experience a price rise, or a suspicion that they were charged more than someone else. Nothing else in experimentation carries that consequence.

Fairness, disclosure and the rules

Before designing anything, be clear on what is being varied and on what basis. There are two very different practices sharing one name.

Randomised price experimentation varies the price by a random assignment that has nothing to do with who the person is. Everyone is equally likely to see each price. This is a normal experiment.

Personalised pricing varies the price by attributes of the individual — their location, their device, their inferred willingness to pay, their history. This is a different activity with a different risk profile, and in several jurisdictions it carries disclosure obligations. In the EU, the Consumer Rights Directive as amended by the Omnibus Directive requires traders to inform consumers when a price has been personalised through automated decision-making. Pricing that varies by protected characteristics, or by proxies for them, raises discrimination exposure regardless of intent.

This article covers the first practice. If you are considering the second, the design question is a legal one first, and the answer should come from counsel rather than from a testing guide.

Two operational rules apply either way. Honour the price a customer was shown — for the session and for any renewal implied by it — because the reputational cost of not doing so exceeds any experiment's value. And keep the assignment stable per user, not per session, so nobody watches the price change by reloading.

Designs that avoid showing two prices to one person

flowchart TD
  A{Do users see each<br/>other's prices?} -->|Yes, public or shared| B[Do not randomise price.<br/>Test packaging or<br/>run a time-split]
  A -->|No| C{Is there a stable<br/>logged-in identity?}
  C -->|Yes| D[User-level assignment,<br/>sticky across sessions]
  C -->|No| E[Cookie-level assignment<br/>plus honour-the-price rule]
  D --> F[Measure revenue<br/>per assigned user]
  E --> F

Sticky user-level assignment is the default. Bucket on a durable identifier so the same person always sees the same price. In Optimizely, that means bucketing on a user id rather than an anonymous visitor id wherever a login exists, and keeping the bucketing key stable across devices if the account spans them.

New-customer-only tests avoid the worst of the fairness problem by excluding anyone who has already seen a price. Existing customers stay on their current price and are excluded from the analysis entirely — not silently assigned to control, which would dilute the estimate.

Packaging and framing tests vary what is included, the billing period, or how the price is displayed, while the amount charged for a given bundle stays the same for everyone. These carry none of the fairness problems and are often where the larger effects are.

Time-split tests run price A for a period, then price B, and are the fallback when nothing else is possible — a public price list, a marketplace, a product where customers talk. They are not randomised, so the comparison is confounded by everything seasonal, and they need a quasi-experimental analysis rather than a t-test.

Running a price test with Optimizely

The price itself belongs in the flag configuration, not in the code. That way the variation can be changed without a deploy, and the analysis reads the same value the customer saw.

// Bucket on the account id so the price is stable for the customer across
// devices and sessions. Read the amount from the flag rather than branching on
// the variation key, so a new price point is a config change, not a release.
const decision = optimizelyClient.decide('checkout_price_2026q3', [], accountId, {
  plan: account.plan,
  is_new_customer: account.orderCount === 0,
})

const priceCents = decision.variables.price_cents
renderCheckout({ priceCents, priceKey: decision.variationKey })

// Track revenue on every assigned user. Non-converters contribute a zero, and
// leaving them out is what turns revenue per visitor back into a rate.
optimizelyClient.track('checkout_completed', accountId, {}, {
  revenue: order.totalCents,
  price_key: decision.variationKey,
})

The flag configuration keeps the price points and the guardrails together, so the test is legible to somebody who was not in the room when it was designed.

{
  "flagKey": "checkout_price_2026q3",
  "bucketing": "account_id",
  "audience": "new customers only",
  "variations": {
    "control": { "price_cents": 4900 },
    "up_20": { "price_cents": 5900 },
    "down_20": { "price_cents": 3900 }
  },
  "primaryMetric": "revenue_per_assigned_user",
  "guardrails": ["refund_rate", "support_tickets_per_order", "trial_to_paid_rate"],
  "minimumRuntimeDays": 28,
  "honourShownPriceForDays": 365
}

Note the runtime. Price tests need to cover at least one full billing cycle plus the refund window, because the early readout is systematically optimistic: cancellations and refunds arrive after the purchases they belong to. The guide to how long to run an A/B test covers the general case; price is the case where stopping early is most tempting and most misleading.

Reading a price test

Report revenue per assigned user with its interval, and report the components next to it — conversion rate, average order value, refund rate — so the mechanism is visible.

A worked example. Control converts 4.0% at $49, giving $1.96 per visitor. The higher price converts 3.5% at $59, giving $2.07 per visitor. Conversion fell by an eighth and revenue per visitor rose by about 6%. Whether that is a real effect depends entirely on the interval around it, and on whether the refund rate moved.

Check three things before acting.

The interval, not the point. Revenue per visitor is noisy enough that a 6% observed difference can easily be consistent with no effect at all.

The guardrails. A price rise that increases refunds, support load or cancellations is buying revenue from the next quarter.

The segments you pre-registered. Price sensitivity varies by market and by plan more than almost any other treatment effect, so a single average can hide a variation that is right for one segment and wrong for another. Pre-register the cuts; do not go looking afterwards. The guide to segmentation and heterogeneous treatment effects covers why the after-the-fact version is unreliable, and reading results without inventing cohorts covers the same trap in a longitudinal form.

Frequently asked questions

Is price testing legal?

Randomised price experimentation is a normal commercial practice in most jurisdictions. Personalised pricing — varying the price using attributes of the individual — is a different activity that can carry disclosure obligations, and pricing that varies by protected characteristics or their proxies carries discrimination exposure. Get advice on the specific design rather than relying on a general answer.

What metric should a price test use?

Revenue per assigned user, computed across everyone in the experiment including non-converters. Conversion rate alone will always favour the lowest price, and average order value alone will always favour the highest.

How long should a price test run?

At least one full billing cycle plus the refund window, fixed in advance. Early readouts are biased upward because refunds and cancellations lag the purchases they cancel.

What if customers notice different prices?

Honour the price each customer was shown, for the session and any renewal it implies. If the product is one where customers routinely compare prices with each other, do not randomise the amount at all — test packaging, framing or billing period instead.

Can I test more than two price points at once?

Yes, and it is often worth it because the shape of the demand curve matters more than a single comparison. Budget for it: each additional arm splits the traffic and adds a comparison, so plan the multiple-comparison correction before the test rather than after.