He Spent $1M on A/B Tests. Here's What Won. transcript

40% of people who install this app from an ad end up paying for it. This astrology app spent over a million dollars on AB tests and made $20 million with no investors. All the common advice we hear about onboarding, pay wall, pricing, and even giving free trials, all of it failed those tests. So, I asked the founder to come on the channel to share all his experiments that won and failed in the last 6 years. The usual advice is to keep onboarding short, but based on what we found in our previous episode, studying over a thousand onboarding flows, most of the top crossing apps have one of the longest onboarding flows.

Moonly has a similar approach. VR on boarding is 30 screens long. Over the last 6 years, we had three major iterations of onboardings. We focused mostly on [music] features, but not the users problem. The latest version shifted the focus to users jobs to be done or the pain behind it. And we offer an activational insight. So we give you small bite-sized information that unlocks some insight such as [music] how lunar cycles can affect your mood.

And finally, we introduce the feature by showing how it solves your specific problem. [music] Since the app has 40 features and not everyone uses each of these features, they tailor the onboarding flow based on the ad you came from. >> We just finished developing our own attribution engine. Today we can connect user from ad network to specific custom product page on app store and then to specific [music] custom onboarding flow and even a specific app experience.

Someone who simply search only will [music] see the standard page but someone who search specific feature like to they will see fully customized page and set of screenshot. We now have around 10 different onboarding flows in production each built around a specific user job even anxiety or ADHD. >> By customizing their onboarding flows their customers lifetime value doubled which means their customers are worth twice more than before.

This led them to double their ad spend in a single month. >> Our cos are now profitable from day zero just because custom on boardings. [music] During the onboarding, we collect as much information as we can including birth details and personalize the first session. >> So your user finishes onboarding and lands on the homepage for the first time. What do they see? >> They see a daily guidance card, moonface, personalized daily program.

Plus, they provide something called personal transits, >> which works as a perfect hook because [music] you keep thinking about your transits and you remember about this and you expecting when your transit will finish, when the next transit will start. Quick assess buttons on main screen are also customized [music] based on their on boarding answers. >> Customizing their whole experience worked wonders for their app retention numbers.

Our day one retention now is 41%. And day 7 37%. >> If you're trying to grow an app, you would want to be featured as app of the day. Only has been featured on the app store multiple times. Did this move the needle for you? >> It worked greatly 15 years ago, but in reality, being feature stopped delivering the same benefits today. It tends to bring a lot of irrelevant traffic. Irrelevant traffic shows worse conversion rates.

So it just hurts your statistics. The app store algorithms analyze it and may rank you lower in search result. It's a problem sometimes actually. So featuring can still work well for universal generic apps like calculators or flashlights, [music] but it isn't particularly useful for niche apps. >> When designing the onboarding flow, we always debate when to introduce the login. Is it before the customer arrives at a homepage or after?

On Mobin only 12% of apps have no sign up or login step anywhere only has 10 million users and no login screen at all. >> I always recommend to start with question why why do we need this? [music] It's easy to go on afterpilot. You see same elements and dozens of other apps and assume we need it too. [music] Sometimes login even requires an SMS which doesn't always arrive in [music] seconds. Just like that you create a UX nightmare.

So they asked why do they even need login at all? >> Was it made to let users keep their data? No, because their data was already stored in Apple keychain and [music] even after installing the app on a new iPhone, you will be instantly locked in. >> Was it to collect emails? >> Entering a real email creates significant cognitive friction [music] and distracts user from what they actually came to do. Besides, users are used to signing in with Apple and hiding their real email address. >> Was it for email marketing? >> But building an [music] effective email marketing is a huge and complex challenge.

And we haven't yet managed to make it work efficiently despite the fact we spent years on this. Even with strong open rates, sending more than a million emails over months cost significant [music] amount of money. So why did we need login at all? it was better to remove it and reduce the cognitive load on our users. >> When designing pay walls, we also believe that we should show the right payw wall at the right moment.

[music] We call this contextual pay walls. Moonly used to do this as well. They would show users the relevant payw wall that matches what they're doing. >> For instance, taro feature taro payw wall bus chart screen chart payw [music] wall. >> In theory, this sounded like a great idea, but every single contextual payw wall failed their AB test. [music] When people see this pay walls one by one, they have perception that you have to buy all these features separately to use the product. >> So in the end, you went back to the single consistent payw wall. >> Yep. >> Orbit's payw wall makes it really clear that if you pay today, you unlock all features.

There are two types of pay [music] walls. The soft payw wall is a payw wall where you can close it and keep using the app. Whereas the hard payw wall, there's no way past it without paying. You've tested both soft payw wall and a hard pay wall. Which pay wall won? >> Across all our tests, the soft pay wall won. The close button only appears after 5 seconds, but we call it soft pay wall. Anyway, in the US, users are much more accepting of aggressive monetization. >> Soft payw wall globally whereas hard payw walls won in the US.

But out of all the paywall experiments, what moved their numbers the most? I feel like images are the biggest lever. We run hundreds of experiments on every element on the pay wall. [music] Just a few days, a single winning image delivered us 2x uplift. And the winner was not the image that we bet on. >> Could you guess which payw wall image won the AB test? >> The woman [music] on the sunset. Yeah, >> that was the first image that was in realistic style that won the AB test.

For some reason, we fell in love with Disney style, but we failed to consider that the audience most willing to pay for our product is over 35 and they couldn't recognize themselves in cartoonish 3D characters. They need something more realistic. >> Since realism worked, they pushed it even further. >> We once decided to [music] personalize users avatar based on their zodiac signs. It makes product feel more personal.

But then we decided to take it further and personalized every image with user's [music] own face around 50 images in the product which is very complex and expensive but we build it anyway. The result aren't always consistent [music] and some of them are simply cringe so people are sensitive to anything involving their own face. [music] In the end thousands of users avoiding open the app so we completely removed this feature.

They also tested eight variants of titles and subtitles on the pay wall. [music] >> When we only change the words, conversion increased by 12%. And I recommend to always add social proof like number of downloads, reviews. >> They had a similar finding with our previous guests. >> These social accolades are really nice to kind of show credibility and authority. >> Even mentioning a fine print. [music] >> This no commitment cancel anytime subtitle always seems to bump things up incrementally.

If you compare our pay walls year by year, [music] it might look like we launched only a few major versions over last six years. But what you see today is the result of small incremental improvements made almost every day. We started [music] with a 10% conversion rate in 2020. It was our baseline and today we reached 40% conversion on paid [music] traffic. >> Every element on the payw wall earned its place. What about the price?

[music] Pricing tests are the most important experiment you can run in a product. It comes down [music] to the price and perceived value. >> In the past year, Moonly ran 40 [music] pricing experiments. >> For example, a highriced lifetime plan can increase the product perceived value and niche users toward ano over weekly plan. So, you don't mean to sell a lifetime plan. >> We call this price anchoring. [music] On Mobin, 4% of iOS apps offer a lifetime plan and it's usually priced at about twice the annual plan.

How do you decide which pricing test to run next? >> People usually don't compare prices to value, they compare prices to each other. [music] So, we kept the number and changed the units. For instance, 8 bucks [music] per month became 8 bucks a week. 30 bucks per year became 30 bucks per month. After we rebalance for weekly [music] payments, money now arrive weekly instead of monthly. You recover your ad spend faster.

You can scale acquisition with a shorter feedback loop. >> Did the users who converted at $8 a week churn harder? >> Yes, but the final interview is bigger as you have more rebuilds before the scoreboard will die. >> There are also a lot of advice around free trials. They had one. It worked for a year. >> But most of the time the unit economics were much stronger without them. People make purchase and then in few seconds they cancel it and it's now a very common pattern. >> The birth chart feature is one of the most valuable features in the app and trial users got it for free.

After that they feel that they already received 90% of value from our product and they are not interesting to pay it >> and that's why they removed the free trial completely. On Mobin 59% of iOS paywall screens do not offer a free trial. So instead of a free trial moonly dropped a one-time offer. They have different mechanics in place but in this case it's usually on day three. If price is the blocker, we don't lose this user.

The single payable lifted revenue between day three and day 30 about 15%. >> Now, here are four things that works really well for Moonly. The first one is sharing in one tap. >> People don't share features, they share insights about themselves. If someone takes a screenshot in Moonly, we offer a readym made flow that lets you share this in one tab. 23% of users share something from the app and as a result they got millions of free views.

Apart from 40 features in their app, the one that moves the needle was their AI astrologer Luna. >> We now have about 1 million conversations after feature was launched 6 months ago and it grows exponentially. [music] We experimented extensively with her style, vibe, tone of voice, visual identity. They tested cartoon graphics, a Disney-like style, and a faceless approach. But none of these appealed to their users. >> When people receive meaningful personal guidance, they need the entity to feel trustworthy.

So, realism works better here. >> On top of subscriptions, they offer add-ons. >> We even developed our own viculations engine. We've created around 10 in-depth astrology reports that go far beyond the core experience like soulmate report, career, astrotography, numerology. This report is also added to the AI astrologers context. We just started few months ago. We already generate around 500k per year on adons and it doesn't subscription [music] >> to sell add-ons.

They built a smart system for inapp announcements. It analyzes each user's context and anticipates what they might need next. The system maintains a dynamic queue of offers that updates every second. >> The AI astrologer also recommends add-ons to the [music] users >> but only when a specific one can really help user at the moment. And when it comes to payments >> with native payments, 50% of cancellation happened in 3 minutes from purchase because iOS made cancelling a onetop habit. >> In the US, they moved payments of Apple entirely. >> So we added Stripe payments as an alternative in the US and our LTV doubled overnight. >> Now about 80% of their US customers pay with [music] Stripe. >> We provide small discounts for Stripe and it's a win-win.

I was very skeptical of in the beginning, but now on iOS, it works absolutely great. >> Does that work on Android too? >> You can only offer alternative billing with a Google native SDK, which means you don't control anything. The interface is designed to scare users away and it also doesn't support Google Pay, which kills almost all conversions. [music] No one wants to enter the cards details manually. So I don't recommend to try alternative building conundrate >> since they run hundreds of AB tests and [music] experiments each month.

How do they manage it? >> There is no way we could manage it manually. [music] We use Coror and Codex for this. AI helps us connect the dots and design [music] experiments around areas of inefficiency. AI automatically matches numbers across all our data analytic sources like post hog adaptive drive [music] and delivers a clear recommendations like to ship not to ship or collect more data. >> They tested things that people would probably not even question in the first place. >> Every element, every icon, every text [music] earned its place by winning a test.

Uh in 2026, it's very hard to get positive unit economics in application. >> [music] >> The common mistake is to spend millions of dollars trying to build amazing product only after that realize that market didn't exist for this product or your approach is incorrect. [music] >> And he recommends this instead. >> Start with a web funnel because you have much more freedom and you will instantly get the signals from the market.

You can do it alone with AI agents or maybe with one front- end developer. It's better to be as lean as you can so you can go step by step. I recommend to move slowly and cheap. >> So, six years, $1 million worth of AB tests. In our previous episodes, we studied thousands of onboarding flows, dashboard designs, pay walls to find out what works and what doesn't. If you're into design databacked [music] breakdowns on apps and websites that have already been shipped, this channel is for you.

Thank you for watching and I'll see you in the next one.