The Origins of Big Data

Every day, new data pours in from documents, images, cell phones, emails, social media, sensors and multiple other sources at an unprecedented rate. The rate at which this data comes in, and the sheer size of data available for organizations and companies to analyze, is where the term “big data” comes from.

Here are the “Three V’s” that are generally used to define big data, originally explained by industry analyst Doug Laney:


The vast amount of data generated at any given time. Thanks to the Internet and the mobile revolution, the amount of data available to analyze has grown — and will continue to grow – exponentially with every mouse click. According to the International Data Corporation (IDC), the data we create and copy annually will reach 44 zettabytes, or 44 trillion gigabytes, by 2020. As the volume of data continues to increase and more information becomes available to analyze, data scientists are becoming better-equipped to make behavioral predictions and recognize patterns.


The speed at which data streams in and out. Similar to how the volume of data grows at an exponential rate, the speed at which this data is being generated is almost incomprehensible. Check out the following statistics to get an idea of just how quickly data is moved around today:

According to the IDC, by the year 2020, “about 1.7 megabytes of new information will be created every second for every human being on the planet.”

In March 2015, statistics from Facebook showed that users sent an average of 31.25 million messages and viewed 2.77 million videos every minute.

An average of 80 million photos are uploaded every single day on Instagram.


The various sources from which data is collected and analyzed. Today, data is collected in a wide range of formats, both traditional (medical records, paper archives, insurance forms, financial transactions) and non-traditional (social media conversations, emails, texts, photos, videos). In the past, most data was structured and could neatly be placed in databases. Much of the data received today is text-heavy and unstructured, and requires a different approach to be stored and analyzed.

The data that was collected about a decade ago is minuscule compared to the amount of data that is collected daily today. The databases that were once used became outdated as the digital revolution introduced endless possibilities for rapid data streaming. Sophisticated computing technology — such as IBM’s supercomputer, Watson, which becomes more “intelligent” as it is used — and changes to IT infrastructures have made it possible to meet the demands of the evolving big data industry. Cloud computing, grid computing, the declining cost of data storage, and data storage technologies such as Hadoop have also significantly eased the burden of big data analysis.

The importance of big data is immeasurable. Analyzing datasets can help organizations make better business decisions, improve business processes and procedures, provide insight and predictions for customer behavior, boost security, and can be applied to nearly every aspect of an organization in any industry. Netflix, for example, uses big data to understand a user’s viewing history and suggest similar TV shows or movies based on what the user watched. Big data alone isn’t useful, but the ability to pull meaning from the data and recognize patterns is where the true value lies. Any industry — from healthcare and government, to sports and entertainment — can make use of the enormous sets of data streaming in every second of every day.


The Evolving Industry of Big Data Analytics and Future Insights

The amount of data that organizations produced in a month ten years ago is dwarfed by the amount of data that organizations currently produce daily. Rapid expansion of data over the years has been made possible thanks to huge movements such as the Internet of Things, the mobile revolution, social media and technological improvements.

The Internet of Things is the concept of connecting any device — from smartphones and wearable devices, to refrigerators and jet engines — to the Internet. To handle the massive streams of data rolling in from devices, sensors and apps, many organizations and IT professionals are making the move to cloud-based storage solutions rather than handle data storage on their own infrastructure. With social media, smartphones and tablets, people around the world are also connected to the Internet and capable of moving incredible amounts of data around in a matter of seconds — and it’s not slowing down. According to Ericsson, 6.1 billion smartphones will be in circulation by 2020, representing 70 percent of the global population.

Unstructured and text-heavy data — such as Facebook and other social media posts, or emails — can’t be accurately measured in numeric databases, but it accounts for as much as 80 percent of an organization’s data today. Databases and IT infrastructures that were once used to store and analyze structured data crumpled, unable to store such large volumes of unstructured data streaming in from various sources so quickly. Better data storage technology, such as Apache Hadoop, made it possible for enormous amounts of unstructured data to be stored in its original format by not having to “structure” the data first.

Although big data is useful for industries across-the-board, here’s a look at three fields that have evolved and will continue to benefit from big data analysis:


Patient medical records, prescriptions, doctors’ notes, insurance claims — even data from smartphone apps such as calorie counters or pedometers — can be analyzed by healthcare providers to make improvements to the healthcare industry. Data analysis can help predict disease outbreaks, produce better drugs, detect warning signs of patient illness, prevent disease and lower healthcare costs as a result of improving quality of life.


Government and law enforcement agencies are able to analyze big data to increase cyber security, fight crime and lower government spending. The CIA can use analytical software to foil terrorist attacks, as well as prevent cyber fraud by searching for patterns in online transactions. Government agencies, at the federal and state level, can also use data to monitor budgets and prevent unnecessary spending.


Data from testing scores, GPAs, notes on student behavior and online education programs have given educators the opportunity to improve student results. Test results and assignment grades are the current indicators of student performance, but big data can help educators to adjust curriculums, ensure adequate student progress and provide a better evaluation process for educators.

Competitive Skills for Big Data Jobs

For those in the market for a career in big data, it takes more than an aptitude for computers and software to land the job. Job requirements may vary by industry, but the most competitive big data analysts excel in:

Knowledge of Big Data Platforms and Databases

Fundamental knowledge of industry tools and databases, and how they can be managed, is beneficial for aspiring big data analysts. One of the platforms most commonly used to reduce constraints for storing and analyzing data is Apache Hadoop. Hadoop expertise — and knowledge of its processing component, MapReduce — allows big data analysts to spend more time analyzing and less time collecting data.

Data Mining and Analytical Prowess

Data mining, or the process of digging through data to find meaningful patterns, is the bridge that connects data collection to improving business processes. An analytical mindset and a knack for problem-solving are essential for successful data mining. Big data analysts must be able to view data from all angles, categorize the information and create data-backed strategies to improve products, predict outcomes and improve decision-making.

Business Mindset

To find valuable insights from data, big data analysts must first understand all business goals and what factors can improve or hinder business strategy. Strong communication skills, and the ability to explain complicated results in a clear and concise way, are highly sought after by business leaders. The fast-paced and swiftly changing nature of data also calls for analysts who are quick to change directions per data results.


Landing the Job: Tips for Acing Your Big Data Job Interview

Similar to the overall job requirements, the interview process for big data analyst positions will vary by industry and organization. Before the interview, applicants can set up for success by researching the industry, the hiring company, its analytics processes and the role being filled. Prepare to answer common interview questions regarding educational background, strengths and weaknesses, as well as technical questions about tools and platforms. Knowledge of the company’s goals, and the big data tools and processes used to achieve those goals, may help applicants determine what skills and achievements they can discuss during the interview.


High School Preparation for Big Data Analytics Program Success

Exposure to big data concepts in high school can help pave the path to career success in big data analytics. High school students who enjoy working with computers and are curious about technology may want to consider:

Specialized Schools

Students with an expressed interest in computers and data science may benefit most from a magnet school education. A magnet school bases its curriculum on a specific academic focus, ranging from technology to the performing arts. For those who may want to pursue a career in big data post-graduation, magnet schools can provide better academic achievement, innovative and hands-on experience, and more opportunities to foster analytical and problem-solving skills.

Advanced Placement Program and Related Courses

As early as the ninth grade, students may choose to take courses based on personal interests. For freshmen and sophomores, technology and computer-related electives are great options to explore a potential interest in the big data industry. Students in their junior or senior year may also discuss Advanced Placement (AP) course options with their guidance counselors. If available, AP courses in computer science, statistics and calculus can help students prepare for higher education in big data analytics and earn college credit.


Keys to Finding the Right Big Data Analytics Degree Program

Finding the best program to pursue a big data analytics degree requires careful consideration of degree requirements, available resources and learning environments. Degree programs for big data analytics vary, and selecting the right program largely depends on personal preference. Some students may prefer to build analytical skills through intensive coursework, while others may prefer more hands-on experience and interaction with the faculty. A lower student-to-faculty ratio gives students the opportunity to receive more individual help and boost academic performance. Resources and available equipment may also help students determine the degree program that would be the best fit. Students should carefully consider programs that use the most common big data tools and platforms — Hadoop, R, Python — or advanced industry equipment, such as the IBM supercomputer, to address real-world challenges.


Common Big Data Analytics University Courses

Big data degree programs may vary in terms of learning styles and resources for students, but many of the courses remain the same across the board. Prospective students may need to take courses in the following specialties:

Data Visualization

Courses in data visualization explore visualization software to teach students how to turn data analysis into 3-D models and graphs for a more clear understanding.

Data Mining

Data mining courses harness analytical skills from students, pushing them to sort and categorize data to extract meaningful information.

Data Structures and Organizations

Students can learn about the various data structures, such as structured and unstructured data, and the databases that can be used to extract, store and help sort datasets.


Why Pursuing a Big Data Analytics Degree Pays Off

Students who pursue a big data analytics degree have a promising job market to look forward to. According to the Society of Human Resource Management, nearly 4.4 million big data positions will be open in 2016. The demand to fill big data positions also leads way to more competitive salaries, which according to Forbes, median salaries for experienced big data professionals frequent the six-figure range.

Big data grows bigger by the minute, and the industry needs more analysts to sort through and find the value. Today, the rising cost of doing business means that companies feel more pressure to be efficient. Big data analysts can help companies, both for-profit and not-for-profit, become more efficient, find business opportunities that may have been overlooked, and improve business procedures or services.