Introduction
Initially, this reward platform was like many other platforms, treating all users the same and displaying rewarded tasks indiscriminately. This one-size-fits-all approach was clearly suboptimal, leaving both users and the platform in a state of untapped potential. Recognizing the opportunity, we embarked on a journey to develop a recommendation system tailored to the unique dynamics of the platform.
Join me as we delve into this transformative journey, exploring how we developed and refined our AI recommender to drive a significant revenue surge. This article chronicles the challenges, breakthroughs, and learnings that defined our project.
The initial situation
The platform allows users to earn rewards through various activities. Key features of the site include:
- Play to earn: Users can earn money by playing games.
- Online Testing: Users can start as testers for websites and games, providing feedback and earning rewards for their participation.
- Surveys: The platform offers numerous surveys in which users can share their opinions and earn money.
- Cashback on Online Shopping
Initially, the strategy was straightforward yet limited: manually selecting and regularly updating the display of tasks based on their perceived revenue potential. This method, while functional, presented several inherent limitations.
Firstly, the manual selection of tasks for display inherently lacked optimization for maximum revenue generation. This approach, governed more by intuition than a behavioral prediction, was not finely tuned to capitalize on the full revenue potential of various tasks.
Secondly, the manual system did not take into account a wealth of available data that could significantly enrich the process. Important demographic data, as well as external data sources, were missing. Such data could have offered a more advanced understanding of user behaviors and preferences, which is critical in tailoring offerings to individual users.
Thirdly, and perhaps most critically, this approach failed to leverage interaction data. User interaction data is a veritable goldmine, providing deep insights into how users engage with the platform. This data includes how users navigate the site, which tasks they are naturally inclined towards, and their behavior patterns even before they start generating revenue. This oversight meant missing out on vital clues that could guide more effective and personalized task recommendations.
In summary, while the initial approach to display revenue-generating tasks was operational, it was far from optimal.
Requirements check
Before embarking on any AI/ML project, certain prerequisites must be met, primarily focusing on data, which is crucial for understanding and predicting user behavior. Additionally, infrastructure aspects need to be checked to ensure a smooth and secure process.
Data requirements
- Variety
A wide range of data points related to user behavior is essential. This includes interaction data such as page views, clicks, task selection, task initiation, and revenue generated per task. Additionally, demographic data like the user’s origin (which platform they came from), gender, employment status, income, and year of birth should be incorporated. - Relevance
The key assumption is that these data are implicit signals helping to predict user behavior and preferences. To make accurate predictions, it’s crucial to have data correlated with the behavior we aim to predict. Identifying these relevant data points is a vital, business-oriented part of the process. - Quantity
The amount of data is also critical. For example, having 20,000 users but only one page view per user is insufficient for prediction. The necessary quantity of data depends on the user base size and the type of behavior to predict. A useful approach is to consider how many product interactions a human would need to observe to predict a specific user’s preferences. This method offers a general guideline for estimating the data volume needed for AI/ML processes.
Real-time Data Streaming Architecture
Implementing a robust architecture for real-time data streaming is crucial. This ensures that the data used for AI/ML processes is up-to-date, allowing for more accurate and timely predictions.
Plan for A/B Testing
Establishing a plan for A/B testing is essential. This allows for testing different models and approaches in a controlled environment, helping to identify the most effective strategies for predicting user behavior.
These elements form the foundation of a successful AI/ML approach, ensuring that the data is comprehensive, relevant, and adequate in quantity, supported by the necessary infrastructure for efficient processing and testing.
Managing multi-type and multi-tier rewards
Since the platform consolidates various types of rewards (including cashback, play-to-earn, and product testing), we must implement a method for processing interactions across these diverse reward categories. This approach will also enable our recommendation system to learn from behaviors across different reward types, rather than solely within the same reward category.
Additionally, some reward programs offer earnings through a multi-tier system. For instance, if you begin a task such as app testing, you’ll receive a cash reward at each step of the testing process as you complete it.
To manage these multi-type and multi-tier rewards effectively, we have standardized each reward by mapping it to a unique format with consistent types of interactions.
Results
Baseline version: +12% in revenues
The initial version of our recommendation system, implemented with specific data types and a Neural Collaborative Filtering model (He et al. in 2017), resulted in a notable increase in revenue. In this version, we utilized user data without demographic details, and interaction data focusing on tasks initiated and revenue generated, but not on impressions. The product data was limited to just identifiers, without any additional metadata. Despite these limitations, the system successfully enhanced our revenue by 12%.
Revenue vs. Conversion Trade-off
Higher conversions often indicate a successful alignment between user preferences and the recommended products. However, focusing solely on conversions doesn’t always align with maximizing revenue.
Revenue generation, on the other hand, emphasizes the monetary value of conversions. A recommendation system might lead to fewer conversions but higher revenue if it successfully prompts users to purchase more expensive or profitable items. In contrast, a system optimized for conversions might encourage more frequent purchases of lower-priced items, potentially leading to lower overall revenue.
Balancing these two aspects is key. A system overly focused on high-value items might alienate users seeking affordable options, reducing overall engagement and potential conversions. Conversely, prioritizing conversions without regard to product value can lead to increased transaction numbers but lower profitability.
Tuning the emphasis on revenue: +28% in revenues
After shifting our focus to prioritize revenue as the primary objective in our recommendation system, we undertook a strategic tuning of related hyperparameters. This recalibration was aimed at optimizing the system’s performance with a stronger emphasis on revenue generation. By adjusting these key parameters, which influence how the system selects and ranks product recommendations, we were able to align our recommendations more effectively with higher-value products or those likely to generate more significant revenue.
The results of this adjustment were striking. We observed a substantial increase in revenue, quantified at a remarkable 28% growth.
Conclusion
In conclusion, the journey of implementing and optimizing an AI/ML-driven recommendation system showcases the significant impact that data-driven strategies can have on online platforms. Starting with a basic model that relied on limited data types and yielded a 12% increase in revenue, the platform progressively refined its approach. By incorporating a wider range of data, including user demographics and interaction patterns, and focusing on real-time data streaming, the system became more sophisticated. The crucial pivot came with the strategic decision to prioritize revenue generation over mere conversion rates. This shift, backed by a meticulous tuning of hyperparameters, led to a remarkable 28% growth in revenue. This case study underscores the importance of a well-rounded data strategy, the necessity of balancing different business objectives, and the power of AI/ML in driving business success in the digital age.
