How to experiment with your risk profiles

Introduction to Risk Experiments

Risk experiments let you compare any two risk profiles to decide which performs better. You can utilize experiments when transitioning to Protect to compare the performance of your existing score based risk profile against the ML powered Protect risk profile. Note: Experiments are only available when using the premium version of Protect. 

As part of the migration process, we offer a new risk profile named 'Basic/Premium Action Rules & ML', which you can customize and use directly for A/B experiments or copy to create new risk profiles tailored to your needs.

When you run a risk experiment, a portion of your live traffic is randomly assigned to either the merchant account’s assigned risk profile and compared against a newly created risk profile (B-Profile). 

Before starting your experiment, it’s essential to clearly define your goal and establish criteria for success. If your goal is to reduce chargebacks, then success will be determined by a decrease in chargebacks on the B profile. On the other hand, if your goal is to improve authorizations, then success will be determined by an increase in authorization of the B profile. 

When transitioning from the old RevenueProtect to the new Protect, your primary goal may be to demonstrate that the new version performs as effectively as the old one, giving you confidence in making the switch. Success in this context would be defined by consistent metrics across both risk  profiles.

Our experiments give you data on authorizations, how many transactions are blocked by the risk tool versus blocked by the issuers, as well as chargeback and NoFs.

How to start a risk experiment

1. First you must set up and create your B (new) risk profile. At the company account level in the customer area select Revenue & Risk>Risk Profiles>Create new profile based on “Basic/Premium Action Rules & ML”. Once your B (new) profile has been created you can make any adjustments that you need prior to starting the experiment. For example, adjusting the risk threshold or adding custom rules.

2. Once you are satisfied with both your B profile and the risk profile currently assigned to the merchant account that you are going to run the experiment on it is time to start. Select Revenue& risk>Experiments>Create new experiment (Note that experiments can only be created while on the company account level). 

3. First you will need to give the experiment a name. - Next you will select the merchant account where the experiment will be taking place. - Only one experiment can be run on a merchant account at a time. - You will see that the base risk profile is the risk profile that is currently assigned to your chosen merchant account. Then select the variant (B profile) that you would like to compare against. - Finally, choose the percentage of live traffic you want to allocate to the experiment. For example, selecting 50% for the experiment means that 25% of transactions will be routed to each risk profile, while the remaining 50% of transactions in your merchant account will remain unaffected and unevaluated. Once you click 'Create,' the experiment will begin immediately.

In order to view active and past experiments you select Revenue&Risk > Experiments in the customer area. You will be able to see all of the aforementioned data points and how they compare between the two risk profiles. 

FAQs & Things to Know

• How much traffic should I send to an experiment? Adyen recommends a bare minimum of 10% traffic being run to the new risk profile to ensure the data segment is large enough to drive well-informed decisions. Conducting 50/50 tests, where 100% of traffic is included in the experiment, is recommended for the most reliable results.

• How long do I run my experiment for? We usually recommend running an experiment for around a month before making the switch or increasing traffic. How long the experiments run may also depend on the volume being sent to the experiment or what your goal is. For example, you might notice changes in authorizations over a couple of days. Since transactions may take up to 100 days to potentially result in chargebacks, you may want to extend the experiment duration to accurately assess its success.

• Do I need to reassign my accounts to a risk profile when running an experiment? No, keep your merchant account assigned to the old or “base” risk profile, the experiment will route the assigned traffic amount between the two.

• If I receive chargebacks on an experiment after it has finished, will the metrics update accordingly? Yes, any chargebacks or notifications of fraud resulting from transactions evaluated during the experiment will automatically update the experiment’s metrics in the Customer Area. 

• How long are old/finished experiments viewable in the customer area? Experiments in the customer area will remain viewable for 60 days after you end them. Be sure to copy or screenshot the Experiment Overview to refer back to if you might need to review after 60 days.  *Additional tip: take note of and save the experiment reference ID (Displayed directly under the Experiment name). This will allow you to report on experiments after the 60 days has passed or to view all transactions in an experiment within the payments view in your customer area.

• Can I make changes to risk profiles and rules when an experiment is running? Yes. However, to ensure accurate experiment results, any changes made to a risk rule within a risk profile that is part of an experiment will automatically stop the experiment. You will need to manually restart the experiment.