Sticky bucketing

Sticky bucketing checks whether a user has already experienced a variant of your experiment. If so, Experiment assigns the current value of the user property (the last variant they saw) to the user. This helps keep variants bucketed into their original variants unless you change the parameters of the experiment.

Sticky bucketing is often used as a defense mechanism against variant jumping which is when a user experiences two (2) or more variants for a single flag or experiment. However, simply enabling sticky bucketing doesn't guarantee that users will never experience variant jumping. Variant jumping may occur if your experiment includes both a logged-out and a logged-in experience because a user can have different logged in and logged out Amplitude IDs.

How sticky bucketing works

To turn sticky bucketing on or off, open your experiment and navigate to your experiment and then click the pencil icon and then go to Experiment Design > Advanced (Optional) > Bucketing Options.

When sticky bucketing is enabled, Experiment checks whether a user already has a value for the user property associated with the experiment. If so, the user Experiment assigns the current value of the user property (the last variant they saw). Otherwise, the user is re-evaluated.

Users aren't sticky bucketed to the Off variant. Learn more about evaluation and exposure with these Amplitude resources: evaluation flow chart, and local evaluation targeting capabilities.

When to use sticky bucketing, and when not to

This section provides examples of when to enable sticky bucketing versus when not to. This isn't intended to be an exhaustive list. There are also cases where the results would be the same, regardless of whether sticky bucketing was on or off. An example might be an experiment where you’re targeting everyone who views your home page, and you don't touch any of the experiment controls while the experiment is running. 

When to enable sticky bucketing

• You want to give the user a consistent experience, even if the user property you’re targeting changes. For example, if you’re running an experiment only in the United States, enabling sticky bucketing would ensure your users would receive the same variant if they happened to travel outside the country.
• You want to decrease the percentage rollout of an experiment where the treatment group isn't performing well. But you don’t want to move users from either the treatment or the control to the group that never saw either variant (also known as the 'Off' variant). Enabling sticky bucketing keeps users in their assigned groups even after you change the percentage rollout. Increasing or decreasing percentage rollout doesn't cause users to move between treatment and control.
• You want to target users for a specified duration (for example, two weeks) and then stop targeting new users, while maintaining the original assignments for any users that have already been bucketed. (You might do this if you want to study the long-term effects of a particular treatment on user behavior, or if you are unsure of the quality of one or more of the experiment’s treatments.) Enable sticky bucketing at the beginning of the experiment, with a 50/50 split. Then, after the duration passes, change the rollout percentage to zero.
• You want to sunset a failed experiment, but ensure the users bucketed into an experience still get that experience. Enable sticky bucketing and set the rollout percentages to zero.

When not to enable sticky bucketing

• You want the user’s experience to change as the targeted user property changes. To continue an example from the previous list, if you‘re running an experiment in the United States, you may not want users to have the same experience if they’re traveling abroad.
• Your experiment intends to drive free users to becoming paid users, and relies on earning rewards. After these users convert, you no longer need to offer a reward. If sticky bucketing were enabled here, those users would receive the free experience even after upgrading to paid.
• You want to enforce a “cool down” period between giving discounts. If you want to limit the frequency of discounts for each user to one time every seven days, you can add a seven-day filter to the targeting criteria; if a user received a discount within that period, the flag would evaluate to [off]. This prevents the user from collecting another discount before you want them to.
• You're rolling out or rolling back a variant. When sticky bucketing is enabled and you change the traffic allocation, you get a weighted average between the old and new allocation (since the users who were already bucketed stay in their bucket). It takes some time for your experiment to achieve allocations you want.

Verify if sticky bucketing was enabled for a specific user

Follow these steps to understand if a user was subject to sticky bucketing:

1. Check the Experiment Assignment events in the user's event stream. You can do this if you haven't blocked the Experiment Assignment events in Data.
2. Find the event property with .details that corresponds to the experiment flag key of interest. This shows the version of the evaluated flag, and which targeting rule applies to the user. This can also be helpful for debugging assignment issues.

For example, the properties above show that the user is assigned to off for the lp-app-downloads flag because the device family wasn't iOS. From the above example, it's also understood that this is the 21st version of the flag. The name of the rule applied to them is non-iOS users and the user failed the first rule-based targeting filter, so they went to the second one instead.

In this example, sticky bucketing is enabled and the user was bucketed to the 14th version of the signup-ux-updates flag, where the experiment served them the phone-number-removed variant. Having the flag version helps with debugging when the flag changes. Remember that Experiment shows the assignment event for all active flags in that project, but shows the exposure event on a per-flag basis. If you don’t find an event property corresponding to the flag you’re interested in, check the [Experiment] Environment Name field and make sure it matches the deployment your flag belongs to.