However using Blob storage is the equivalent of dumping data into files on a file share on-premise There are more sophisticated, functionality richer services such as Azure Data Lake or Azure CosmosDB that would be better choices. Passing the data into Blob Storage enables the data to be picked up by other Azure Data Services for additional “cold path” analytical processing or on-premise analytical systems.It also ensures consistency of business rules for KPIs across multiple reports as the rules are stored in the DB and not the report. This minimises the complexity of queries and the workload of Power BI, significantly decreasing the refresh time of reports which is critical in near-realtime reporting. In-memory columnar storage can be utilised in SQL DB, with triggers used to calculate additional measures.Data can also be streamed directly into Power BI to provide real-time reporting, rather than near real-time reporting.
However, Stream Analytics can also perform aggregations over multiple criteria and time windows and push the results in SQL DB. It can also stream a raw feed (with or without enrichment) into Blob storage.
While it’s possible to add code buffering logic and time delays into Logic Apps to handle high volume tweets, other components are available. Logic Apps is exposed directly to the Twitter API.An example PowerBI dashboard is shown below (this is available as a Power Bi Solution Template): These calculations should be done in the DB, so that a consistent set of rules can be applied across all reports. This has obvious data governance and quality concerns. Aggregate generation in Power BI allows for rapid rule changes but doesn’t allow those rules to be shared across multiple reports, requiring the rules to be recreated in each report.However, for a production architecture it has weaknesses: This is an awesome architecture for rapidly prototyping a solution and allows for business rules/logic to be changed rapidly. Cognitive Services provides a variety of machine learning services which are extremely useful for social media analytics such as sentiment analysis, moderation (less blushes if the profanity is automatically removed), language translation and intent (whether they asking a question or making a complaint). LogicApp is used to pull the data from Twitter periodically using the inbuilt Twitter connector, the Cognitive Services Text API can be called from within LogicApp and the enriched data inserted into SQL DB, and Power BI can be used to generate aggregates/measures/KPIs and to visualise the data. One simple pattern for twitter analytics is this: However, there are also a number of PAAS services that can be utilised to achieve the same result, with much less coding, configuration and setup effort and improved maintenance and HA/DR capabilities. There are numerous technologies that can be utilised to stream, analyse and visualise tweets on Azure, including traditional Big Data technologies such as Hadoop, Kafka, Spark, Storm, No SQL DBs and more.