How to Prioritize A/B Test Ideas: ICE, PIE, and PXL

You have thirty test ideas in a spreadsheet and enough traffic to run maybe three or four good experiments this quarter. Some ideas came from analytics, some from a sales call, one from the CEO. Every one of them "feels important." The question that actually matters is not which idea is good — most of them are plausible — it is which do I run first, given that I can only run a few.

That is a prioritization problem, and it has a well-established answer: score every idea against the same criteria, rank by the score, and work down the list. This guide covers the three frameworks practitioners actually use — ICE, PIE, and PXL — where each comes from, exactly how to score with it, and how to keep the scores honest. It ends with the caveat that most prioritization advice skips: a framework ranks your backlog, but it cannot tell you whether the top-ranked test is even runnable on the traffic you have.

You can score your own backlog while you read using OptiPilot's free Test Prioritization Scorer, which ranks your ideas by ICE or PIE and exports the ordered list.

Why a framework beats a meeting

Without a system, prioritization defaults to whoever argues hardest or outranks everyone else. That produces a backlog ordered by politics, not by expected value, and it makes the ordering impossible to defend six months later when someone asks why you ran the checkout test before the pricing test.

A scoring framework does three useful things. It forces every idea through the same set of questions, so a homepage headline tweak and a full checkout redesign are compared on shared terms. It turns a vague debate into a number you can sort, filter, and revisit. And it creates a paper trail: the score records what you believed at the time, which is exactly what you want when you audit your hit rate later.

None of the frameworks is "correct." They are lenses, and they trade speed against objectivity in different ways. Pick one your team will actually use every week rather than the most rigorous one that gets abandoned after a month.

How ICE, PIE, and PXL work

The three mainstream frameworks share a structure — score each idea on a few factors, combine the factors into one number, rank descending — but they differ in what they measure and how much they try to remove human bias.

ICE: Impact, Confidence, Ease

ICE was created by Sean Ellis, the founder of GrowthHackers and the person who coined "growth hacking," to triage the high volume of experiments growth teams run. You score each idea from 1 to 10 on three factors:

• Impact — how much this will move the target metric if it works.

• Confidence — how sure you are that the impact estimate is real, based on the evidence behind the idea.

• Ease — how little effort it takes to build and ship (higher score = easier).

You then combine the three, either by multiplying them or by averaging them, and rank on the result.

ICE (average) = (Impact + Confidence + Ease) / 3
ICE (multiply) = Impact x Confidence x Ease

Example, Impact 8, Confidence 4, Ease 9:
  average  = (8 + 4 + 9) / 3 = 7.0
  multiply = 8 x 4 x 9        = 288  (range 1-1000)

The two methods can rank the same backlog differently. Multiplying punishes a low score on any single factor much harder — a 2 anywhere drags the product down sharply — which is usually what you want, because a brilliant idea you have no confidence in should not sit near the top. Averaging is more forgiving and easier to read. Pick one and apply it consistently across the whole backlog; mixing them makes scores incomparable.

ICE's strength is speed: three quick judgments and you have a ranked list. Its weakness is that Impact and Confidence are guesses. Confidence is the built-in safeguard — it exists precisely so that a shaky Impact estimate gets marked down rather than treated as fact — but the whole score still rests on human estimation.

PIE: Potential, Importance, Ease

PIE comes from Chris Goward at WiderFunnel and is aimed squarely at conversion optimization. You score each idea 1 to 10 on three factors and average them:

• Potential — how much room for improvement the page or flow has (a bad page has high potential).

• Importance — how valuable the traffic hitting it is; a high-traffic, high-cost-to-acquire page scores higher.

• Ease — how simple the change is to implement, technically and organizationally.

PIE = (Potential + Importance + Ease) / 3

Example, Potential 7, Importance 9, Ease 5:
  PIE = (7 + 9 + 5) / 3 = 7.0

PIE's distinctive move is Importance, which forces you to weight by page value. It stops teams from over-investing in a clever test on a page nobody visits. Its blind spot is that Potential and Importance are still subjective, and "Importance" can overlap with the traffic reality you need for powering — a point the last section returns to.

PXL: objective, criteria-based scoring

PXL was built by Peep Laja and the ConversionXL (CXL) team specifically to reduce the subjectivity in ICE and PIE. Instead of asking "how impactful is this, 1 to 10?", PXL asks a fixed list of roughly ten concrete, mostly yes/no questions and scores each as points. The standard template includes questions like:

• Is the change above the fold? (1 / 0)

• Is it noticeable within 5 seconds? (2 / 0)

• Does it add or remove an element rather than just tweak one? (2 / 0)

• Is it designed to increase user motivation? (1 / 0)

• Is it on a high-traffic page? (1 / 0)

• Is it backed by user testing? (1 / 0)

• Is it backed by qualitative feedback (surveys, support)? (1 / 0)

• Is it backed by digital analytics? (1 / 0)

• Is it supported by heatmaps, mouse tracking, or eye tracking? (1 / 0)

• How easy is it to implement? (graded scale — more dev time earns fewer points)

You add up the points and rank by the total. The exact point values and the ease scale vary between published versions of the template, and PXL is explicitly meant to be customized — swap in the evidence sources and page criteria that matter to your business.

The payoff is objectivity and consistency: two people scoring the same idea will usually land on the same total, because most answers are matters of fact rather than opinion. The cost is that PXL is heavier — it presumes you have the research (user testing, analytics, heatmaps) to answer the evidence questions, and it says less about raw expected uplift than ICE's Impact score does. PXL rewards well-evidenced, safely-scoped tests, which is a deliberate bias toward reducing wasted experiments.

ICE vs PIE vs PXL: which framework should you use

There is no universally right answer; there is a right answer for your situation:

flowchart TD
  A[30+ test ideas in a backlog] --> B{How much time<br/>for scoring?}
  B -->|Minutes, rough triage| C{Do the scorers<br/>share context?}
  B -->|We have research data| D[PXL: objective criteria]
  C -->|Yes, one tight team| E[ICE: Impact, Confidence, Ease]
  C -->|No, mixed stakeholders| F[PIE: adds page Importance]
  E --> G[Rank by score]
  F --> G
  D --> G
  G --> H{Can you power the<br/>top test at your MDE?}
  H -->|Yes| I[Queue it]
  H -->|No| J[Re-scope, widen the MDE,<br/>or drop to the next idea]

A practical default: use ICE to triage a long, messy backlog down to a shortlist quickly, then run PXL on the shortlist before you commit engineering time. You get ICE's speed where you have many ideas and PXL's objectivity where the decision is expensive.

How to reduce bias in your scores

Every framework is only as good as the numbers you feed it, and the fast ones (ICE, PIE) are easy to game — consciously or not. Four habits keep the scores honest.

Define anchors for each score before you start. "Impact 8" means nothing until the team agrees what an 8 looks like. Write a short rubric: Impact 9-10 = expected to move the primary metric by a large, business-changing margin; 5-6 = a modest lift on a secondary metric; 1-2 = cosmetic. Anchored scales are the single biggest improvement you can make to ICE and PIE.

Tie Confidence and Potential to evidence, not enthusiasm. A high Confidence score should require a reason: prior test results, a documented user-research finding, a clear analytics signal. If the only support is "I think users will like it," Confidence is a 3, not an 8. This is exactly the discipline PXL bakes in structurally — you can borrow it inside ICE and PIE by refusing to let anyone score Confidence high without naming the evidence.

Have more than one person score. Independent scoring by two or three people, then a reconciliation of the gaps, surfaces the ideas where people are secretly guessing. Wide disagreement on a score is a signal, not noise — it usually means the idea is under-specified.

Score the hypothesis, not the pet project. A well-formed hypothesis ("changing X will cause Y because Z, measured by metric M") is far easier to score consistently than "let's try a new hero image." If you cannot state the hypothesis, the idea is not ready to be prioritized — it is ready to be researched.

Prioritization does not make a test runnable

Here is the caveat that ranks above the frameworks themselves: a prioritization score tells you what is worth running, not what you are able to run. A test can top your ICE list and still be impossible to execute this quarter, because ranking says nothing about statistical power.

Every A/B test needs a minimum sample size to reliably detect a given effect. That sample size is driven by your baseline conversion rate, your chosen minimum detectable effect (MDE), and your significance and power thresholds. The smaller the effect you want to catch, the more traffic you need — and the relationship is steep. A high-ICE idea on a low-traffic page, aiming to detect a 2% relative lift, might need months of traffic you do not have. On paper it is your best idea; in practice it is un-runnable.

So bolt a feasibility gate onto the end of whichever framework you choose:

1. Take the top-ranked idea.

2. Estimate its required sample size and runtime at a realistic MDE using the Traffic Estimator.

3. If you can reach significance within an acceptable window (typically two to six weeks, covering full business cycles), queue it.

4. If you cannot, do one of three things: widen the MDE (only run it if you would act on a bigger effect), broaden the scope so the change is bolder and the expected effect larger, or set it aside and drop to the next idea.

This is why "Importance" in PIE and "high-traffic page" in PXL are not just about business value — traffic is also what makes a test powerable. Two ideas with identical scores are not equal if one runs on a page with ten times the traffic; that one reaches significance far sooner and should jump the queue on feasibility grounds even when the framework rates them the same.

Prioritization and powering are two gates in sequence, not one. The framework answers "is this worth my time?" Sample-size math answers "can I get a trustworthy answer in a reasonable window?" A test has to pass both.

Put it into practice

Prioritization is not about finding the one true framework. It is about replacing "whoever argues hardest wins" with a repeatable process: score every idea against consistent criteria, keep the scoring honest with anchors and evidence, rank the backlog, then check that the top item is actually powerable before you build it.

Start simple. Run your current backlog through ICE this week to get a ranked shortlist, then pressure-test the top few with PXL's evidence questions and a sample-size check. If you want a running head start, score and rank your ideas in the Test Prioritization Scorer, size the winners with the Traffic Estimator, and QA them against the Experiment QA Checklist before launch. For sector-specific ideas worth adding to the backlog in the first place, see the industry experimentation playbooks.

The teams that compound wins are rarely the ones with the best individual ideas. They are the ones with a boring, consistent way of deciding what to run next — and the discipline to only queue the tests they can actually finish.