Sources of big data: Where does it come from?

Over the last five years, there has been a growing understanding of the role that Big Data can play in delivering priceless insights to an organization, revealing strengths and weaknesses and empowering companies to improve their practices. Big data has no agenda, is non-judgmental and non-partisan – it simply reveals a snapshot of activity.

Yet while many organizations understand the importance of data, very few are yet seeing the impact of it. A new study entitled Broken Links: Why analytics have yet to pay off makes the claim that 70% of business executives acknowledge the importance of sales and marketing analytics, yet only 2% say that their analytics have achieved a broad, positive impact. This finding points to the need for Big Data to be handled by outsourced firms who specialize in analyzing the data generated by companies and who can offer real, actionable insights. In the foreword to his report, Dan Weatherill writes that “Our survey and follow-up interviews with nearly 450 U.S-based senior executives from industries including pharmaceuticals, medical devices, IT, financial services, telecoms and travel and hospitality confirmed one thing that we already knew: few organizations have been able to get it right and to generate the kind of business impact that they had hoped for.”

So, What is Big Data and where does it come from?


The term is an all-inclusive one and is used to describe the huge amount of data that is generated by organizations in today’s business environment. The thinking around big data collection has been focused on the 3V’s – that is to say the volume, velocity and variety of data entering a system. For many years, this was enough but as companies move and more and more processes online, this definition has been expanded to include variability — the increase in the range of values typical of a large data set — and value, which addresses the need for valuation of enterprise data.”

The Sources of Big Data

The bulk of big data generated comes from three primary sources: social data, machine data and transactional data. In addition, companies need to make the distinction between data which is generated internally, that is to say it resides behind a company’s firewall, and externally data generated which needs to be imported into a system.
Whether data is unstructured or structured is also an important factor. Unstructured data does not have a pre-defined data model and therefore requires more resources to make sense of it.


The three primary sources of  Big Data


Social data comes from the Likes, Tweets & Retweets, Comments, Video Uploads, and general media that are uploaded and shared via the world’s favorite social media platforms. This kind of data provides invaluable insights into consumer behavior and sentiment and can be enormously influential in marketing analytics. The public web is another good source of social data, and tools like Google Trends can be used to good effect to increase the volume of big data.

Machine data is defined as information which is generated by industrial equipment, sensors that are installed in machinery, and even web logs which track user behavior. This type of data is expected to grow exponentially as the internet of things grows ever more pervasive and expands around the world. Sensors such as medical devices, smart meters, road cameras, satellites, games and the rapidly growing Internet Of Things will deliver high velocity, value, volume and variety of data in the very near future.

Transactional data is generated from all the daily transactions that take place both online and offline. Invoices, payment orders, storage records, delivery receipts – all are characterized as transactional data yet data alone is almost meaningless, and most organizations struggle to make sense of the data that they are generating and how it can be put to good use.

Unlocking real value from data

Real business value comes from an ability to combine this data in ways to generate insights, decisions and actions. CloudMoyo helps companies develop a comprehensive, cohesive and sustainable analytics strategy, which gives them the tools to differentiate themselves via actionable insights and supports employees and the business itself. A number of factors point to the value of the niche that companies like CloudMoyo are fulfilling. A recent study found that two-thirds of companies with the most advanced technology in this area cannot hire enough people to run these capabilities. Added to that, analytics is resource-intensive.

Large companies struggle to allocate enough resources, but for smaller companies, it’s inconceivable that they can dedicate all that is needed for effective analysis. In both these cases, outsourcing is an invaluable advantage to have.

While there is a generally acknowledged understanding that big data can provide a competitive advantage, those who are partnering with sophisticated third-party providers stand a much better chance of benefitting from high-quality, affordable insights. The era of big data is well and truly upon us, and it’s no longer a question of whether enterprises should engage with big data, but how. Technology giant Cisco predicts that the amount of data produced in 2020 will be 50 times what it is today. No wonder then that companies feel overwhelmed and desperately in need of solid advice from specialists who understand their business and can combine it with technology to deliver results.

Being proactive is key

Traditional reporting & BI is giving way to Advanced Analytics. It’s no longer enough to retro-actively analyze what happened and why. Instead, systems and partnerships need to be put in place which leverage high quality data and interpret the data to make predictions around what is likely to happen next, with concrete evidence to back up the claims.

Organizations can address business needs across the full range of analytics requirements with Cloud-based Big Data as a Service — from data delivery and management to data usage. By developing a comprehensive cloud-based big data strategy, they can define an insight framework and optimize the total value of enterprise data. However, cloud-based big data analytics is not a one size-fits-all solution and an expert IT partner like CloudMoyo can help you on this journey.

Sign up for a consultation today and explore more about our services offering