Tutorial
7 min read

Data online generation for event stream processing

In a lot of business cases that we solve at Getindata when working with our clients, we need to analyze sessions: a series of related events of actors, to find correlations or patterns of the actors' behaviors . Some of the cases include:

  • Analysis of user web sessions for web analytics.
  • Funnel analysis, cohort analysis and many more,to understand how product users interact with the product, behave, convert or drop out.
  • Fraud/anomaly or threat detection to prevent them quickly.
  • Product recommendations, suggestions and many more…

This is generally speaking, wherever behavioral analytics are being applied.

For the use cases to be able to research and develop applications, we need to create test data for experimentation, functional tests, performance tests, demos and so on.

Randomly generated events produced by most data generators are not enough. The data that is produced is unrelated. There are no actor sessions. However,  there are a few projects that have tried to tackle this problem. One of them is GitHub - Interana/eventsim which was sadly abandoned a few years ago.

So, what can a solution that supports mentioned use cases look like? We can start with modeling actor behavior - what the actor can do and how it can evolve over time. Something  like this can be expressed as a state machine.

There are still few things missing thou. For start we don't want to just have a state machine and then invoke transitions by writing lines of code, we would land back where we started. We want our state machine to transition automatically based on some probability.

Stream processing use case

Let's consider an example we are working on pattern recognition. We are trying to find banking application users that were thinking about taking a loan but did not took it. We are trying to target users with low balance that opened a loan screen and exited without taking a loan. Model of such interaction with application could look like that:

data-online-generation-for-event-stream-processing-diagram

User can log into an application and when they are Online they can open a loan screen or exit application. From Loan screen they can ether take a loan or exit.

But what 0.1% probability mean? This is a probability that our state machine will automatically do that transition in each step. If we assign some real-world duration of a step, for example 1 minute. 0.1% probability would mean that on average user will log in once in 17 hours. Or from other perspective in each minute 1 of 1000 users will log in.

Using above we can generate click stream of user actions. But what about balance? It cannot be easily expressed in finite-state machine. We could think about adding new set of states Offline low balance, Online low balance,
Loan Screen low balance but it looks like a duplication and it gets even more messy when we think about adding something additional like for example a loan balance.

data-online-generation-for-event-stream-processing-diagram2

Because balance does not affect set of possible actions it should not be modeled as a state. It is just an attribute of particular user that is traversing our state machine which value can be emitted when we are generating events during transitions. Using attributes we can go back to our simpler state machinebut this time enriched with new actions that won't be changing state but will change value of attributes. How can we generate data suitable for stream processing?

data-online-generation-for-event-stream-processing-diagram3

Data Online Generator

We have create a tool that can do all of the above Data Online Generator GitHub - getindata/doge-datagen it can be used to simulate user, system or other actor behavior bases on probabilistic model. It is a state machine which is traversed by multiple subjects automatically based on probability defined for each possible transition.

To express above state machine we need to define list of states, initial state, a factory to create initial state of users, number of users, tick length and number of ticks to simulate. Then we need to define each transition, its name, beginning and ending state, probability of transition and optional callback and sinks. Callback will be invoked during transition and allows for user attributes change. It can also be used to implement conditional transitions returning False will block state machine from transitioning. For example we can block Spending transition if balance is zero. Event sinks are used to generate events during a transition. Currently DOGE-datagen ships with sinks for Database and Kafka.

datagen = DataOnlineGenerator(['offline', 'online', 'loan_screen'], 'offline', UserFactory(), 10, 60000, 1000, 1644549708000)
datagen.add_transition('income', 'offline', 'offline', 0.01,
                       action_callback=income_callback, event_sinks=[balance_sink])
datagen.add_transition('spending', 'offline', 'offline', 0.1,
                       action_callback=spending_callback, event_sinks=[trx_sink, balance_sink])
datagen.add_transition('login', 'offline', 'online', 0.1, event_sinks=[clickstream_sink])
datagen.add_transition('logout', 'online', 'offline', 70, event_sinks=[])
datagen.add_transition('open_loan_screen ', 'online', 'loan_screen', 30, event_sinks=[clickstream_sink])
datagen.add_transition('close_loan_screen', 'loan_screen', 'online', 40, event_sinks=[clickstream_sink])
datagen.add_transition('take_loan', 'loan_screen', 'online', 10,
                       action_callback=take_loan_callback, event_sinks=[clickstream_sink, loan_sink, balance_sink])
datagen.start()

Example output:

$ kafka-avro-console-consumer --topic clickstream --consumer.config ~/.confluent/local.conf --bootstrap-server localhost:9092 --from-beginning
{"user_id":"7","event":"login","timestamp":1644549768000}
{"user_id":"7","event":"open_loan_screen ","timestamp":1644549828000}
{"user_id":"7","event":"take_loan","timestamp":1644550008000}
{"user_id":"1","event":"login","timestamp":1644556668000}
{"user_id":"1","event":"open_loan_screen ","timestamp":1644556728000}
{"user_id":"1","event":"close_loan_screen","timestamp":1644556968000}
{"user_id":"4","event":"login","timestamp":1644559668000}
{"user_id":"4","event":"open_loan_screen ","timestamp":1644559728000}
{"user_id":"4","event":"close_loan_screen","timestamp":1644559788000}

$ kafka-avro-console-consumer --topic trx --consumer.config ~/.confluent/local.conf --bootstrap-server localhost:9092 --from-beginning
{"user_id":"8","amount":75,"timestamp":1644553008000}
{"user_id":"6","amount":47,"timestamp":1644558708000}
{"user_id":"3","amount":42,"timestamp":1644563808000}
{"user_id":"0","amount":85,"timestamp":1644583128000}
{"user_id":"2","amount":40,"timestamp":1644597228000}
{"user_id":"5","amount":18,"timestamp":1644601308000}

postgres=# select * from balance;
 timestamp     | user_id | balance 
---------------+---------+---------
 1644550008000 |       7 |   10933
 1644553008000 |       8 |     409
 1644558708000 |       6 |     320
 1644563808000 |       3 |     577
 1644583128000 |       0 |     259
 1644597228000 |       2 |     697
 1644601308000 |       5 |     702

postgres=# select * from loan;
 timestamp     | user_id | loan  
---------------+---------+-------
 1644550008000 |       7 | 10000

Conclusion

Data Online Generator can be used not only during development and experimentation, but can also be a part of automated test pipelines. It is designed to be flexible to support many use cases and is also extendable and allows for easy implementation of new event sinks. Visit our GitHub, to see examples on how to work with the data online generator.

If you're interested in stream processing, check our CEP Platform

Did you like this blog post? Check out our other blogs and sign up for our newsletter to stay up to date!

streaming
stream processing
data online generator
Data Generator
15 March 2022

Want more? Check our articles

getindata blog big data knowledge sharing it jobs

How do we apply knowledge sharing in our teams? GetInData’s internal initiatives

Knowledge sharing is one of our main missions. We regularly speak at international conferences, we contribute to open-source technologies, organize…

Read more
getindata nifi ingestion universe made out flow files nifi architecture big data
Tutorial

NiFi Ingestion Blog Series. PART IV - Universe made out of flow files - NiFi architecture

Apache NiFi, a big data processing engine with graphical WebUI, was created to give non-programmers the ability to swiftly and codelessly create data…

Read more
getindata big data blog ml model mleap
Tutorial

Online ML Model serving using MLeap

Training ML models and using them in online prediction on production is not an easy task. Fortunately, there are more and more tools and libs that can…

Read more
16kxTuxGkZjskytKJLQKsJg
Tech News

Two BI companies in play. Tableau acquired by Salesforce and Looker by Google.

The two recently announced acquisitions by Google and Salesforce in the thriving business analytics market appear to be strategic moves to remain…

Read more
blogobszar roboczy 1
Success Stories

How we built a Modern Data Platform in 4 months for Volt.io, a FinTech scale-up

Money transfers from one account to another within one second, wherever you are? Volt.io is building the world’s first global real-time payment…

Read more
getindata 1000 followers

5 reasons to follow us on Linkedin. Celebrating 1,000 followers on our profile!

We are excited to announce that we recently hit the 1,000+ followers on our profile on Linkedin. We would like to send a special THANK YOU :) to…

Read more

Contact us

Interested in our solutions?
Contact us!

Together, we will select the best Big Data solutions for your organization and build a project that will have a real impact on your organization.

The administrator of your personal data is GetInData Sp. z o.o. Sp.k with its registered seat in Warsaw (02-508), 39/20 Pulawska St. Your data is processed for the purpose of provision of electronic services in accordance with the  Terms & Conditions. For more information on personal data processing and your rights please see  Privacy Policy.

By submitting this form, you agree to our Terms & Conditions and Privacy Policy