In the previous architecture, multiple problems were faced by the organization like:
1). If you have 5 source systems and 8 target systems we need to write the 40 integration to work upon.
2). Each integration comes with difficulties around:
-Protocol : how the data is transported among the systems(http,https,ftp,sftp,TCP etc..)
-Data format : how the data is parsed(Json, XML)
3). As the systems grow source will have an increased load from the different channels.
This is where the apache Kafka can be used to decouple the source and target systems.source and target would only have to be connected with the Kafka for the sending and receiving of the message.
Source systems could be like website Events, Financial transactions, etc.
Target System could be like Analytics, Database, etc.
Advantage of Kafka:
1). It is distributed,fault-tolerant, and resilient.
2). This can scale to the 100s of brokers and can scale to millions of messages per second.
3). High performance(latency of less than 10ms).
1). Messaging system.
2). De-coupling of system dependencies.
3). Integration with Hadoop and many other Big data technologies.
1). Netflix uses Kafka to apply recommendations in real-time while you are watching on it.
2). Uber uses Kafka to gather user, taxi, and trip data in real-time to compute and forecast demand and pricing in real-time.
Note: Kafka is only used as the transportation mechanism.