Big data and Hadoop facts

Big data means a lot of data. The experts say, big data fits one or more of four Vs of big data, namely, volume, velocity, veracity and variety. We are living in the age of big data and the factors mentioned ahead prove this fact to some extent.

Over 90% of all the data in the world was created in the past 2 years. And, it is expected that by the year 2020 the amount of digital information in existence will have grown from 3.2 zettabytes to 40 zettabytes. The total amount of data being captured and stored by industry doubles every 1.2 years. In two days we create as much information as we did from the beginning of time until 2003.

So, all of these trending threats about big data gave birth to the requirement of having a system which can handle big-data and analyze it at a fast rate. And, this is how Hadoop came into existence, although there were many system/frameworks which were being used or are still used for handling big data.

Big Data has been around for a long time, in fact, you can handle high volumes of data with massively parallel-processing (MPP) databases, such as those offered by Greenplum, Aster Data and Vertica. And, they’re incorporating Hadoop into these platforms.

Hadoop is the distributed file system which is nothing but the way to create clustered or distributed storage and can run on any server. HDFS is fast, secure, and fault tolerant.

MapReduce is actually the core of Hadoop which can put all the data nodes to process the data locally, and is fast and very powerful.

Hadoop is not actually an analytic platform; it can be used with traditional analytic platform or a common way to analyze the data we use R programming language to write our MapReduce jobs.

Hadoop can also be used for archiving and for ETL that stands for extracting, transform, and load. Moreover, Hadoop can also be used for filtering. The Hadoop platform provides many opportunities for transforming and extracting the data and processing.

Scaling of data is the major concern in the data world. The Hadoop system uses Accumulo for scaling the data. Accumulo is actually inspired from Google big table design and is built on the top of Hadoop. It comes with a few improvements in big table, for example, it provides cell-based access control and a server side programming. Also, in Accumulo the key-value pair at the various points can be modified in the process of data management.

Components of Hadoop

Hive: Apache Hive is a data warehouse application and provides high level language for expressing data analysis programs. It provides SQL like environment

PIG: Apache PIG provides high level language for expressing large datasets. PIG’s language consist of textual language called Pig Latin.

Click here to know more about Big Data Hadoop Training Course

Top 10 Repositories of Big Data

Repositories is a term which is used for storage location where we can store things retrieve things. The term repositories of big data means where the big data is stored and we can use that big data for big data analytics, mining etc. SO, there is no need to build their own massive data repositories before starting with big data analytics. So, these are the best data sources available or we can say these are the top 10 repositories for big data.

Google trends http://www.google.com/trends/explore

Google Trends is a public web facility provided by Google Inc., based on Google Search. Google Trends shows how often a specific search-term is entered relative to the total search-volume across various regions of the world. Google trends shows the result in the form of graph. The horizontal axis of the graph represents time, and the vertical axis represents how often a term is searched for relative to the total number of searches.

Data.gov http://data.gov

It is a U.S. government website for getting dataset, launched in late May 2009 by Vivek Kundra, the then Federal Chief Information Officer (CIO) of the United States. Data.gov stores all sorts of amazing information on everything like climate, business, education, agriculture etc.

Healthdata.gov https://www.healthdata.gov/

Healthdata.gov provides health-related data free. You can get comprehensive catalog of health-related data sets relevant to all aspects of health, available for free.

Facebook Graph https://developers.facebook.com/docs/graph-api

In most of the cases, the Facebook profile of any user is public. Facebook provide the Graph API as a way of querying the huge amount of information that the users want to share with the world.

Google Finance https://www.google.com/finance

Google Finance is a website launched on March 21, 2006 by Google Inc. based on Google Search. Google Finance provides updated real time stock data. Google Finance also aggregates Google News and Google Blog Search articles about each corporation.

New York Times http://developer.nytimes.com/docs

New York Times is a big data repository which provides searchable, indexed archive of news articles. It is an open source big data repository.

DBPedia http://wiki.dbpedia.org

Wikipedia provides millions of pieces of data on every subject under which exists in the world. DBPedia is a project to create a public, freely distributable database and anyone can analyze this data.

Google Books Ngrams http://storage.googleapis.com/books/ngrams/books/datasetsv2.html

When you enter words into the Google Books Ngram, it search and analyze the full text of any of the millions of books digitized as a part of the Google Books project.

Amazon Web Services public datasets http://aws.amazon.com/datasets

Amazon Web Services hosts various public data sets that anyone can access for free. The data sets on Amazon Web Services are hosted in these two possible formats, Amazon Elastic Block Store snapshots and/or Amazon Simple Storage Service buckets.

The CIA World Factbook https://www.cia.gov/library/publications/the-world-factbook/

The CIA world Factbook is a factbook that provides the facts on the history, population, economy, government, infrastructure and military of 267 countries.

 

Click here to know more about Big Data Hadoop Training Course