From the Beginnings of Spark and Hadoop
Doug Cutting and Mike Cafarella created Hadoop back in 2005 in order to construct a distributed computing infrastructure for a Java-based free software search engine called Nutch. Nutch was based on publications from Jeff Dean and Sanjay Gemawat at Google and their computing concepts of MapReduce.
It was a part-time project for Doug and Mike. IT wasn’t until 2008 that Yahoo released a search engine based on 10,000 processor cores operated by Hadoop. Just one month before, Apache Software Foundation made Hadoop a high priority project, and just a few months later, Hadoop became the fastest at sorting data with 1TB being processed in 309 seconds. Hadoop has been used for a number of tech companies including Last.fm, Facebook, The New York Times.
From 2010 onwards, Hadoop converted into a critical technology for Big Data, frequently used for mass-parallel data processing, in conjunction with r Cloudera, startups that have promoted the commercialization of Hadoop.
Spark was released by Matei Zaharia AMPLab in 2009. In 2010, the Spark code became open under the BSD license, but this was changed when Apache Software Foundation changed the license to Apache 2.0. In 2014, Spark set the world record for the speed of sorting large amounts of data.
Spark is one of the most active Apache projects and even one of the biggest projects for large open-source data. In 2015, there were more than 1,000 project participants.
What is Distributed Computing Concept
Before you choose one of these technologies, you need to have a better knowledge of distributed computation, also known as distributed data processing. This is a method used to resolve large computer tasks with at least two computers on the same network. A special type of parallel computing. One task can be carried out by various processors; however, the task must be broken down into “mini” tasks in order to compute them in parallel.
A Summary and Comparison of Hadoop and Spark
Spark
As an open-source cluster computing platform, Spark has a framework similar to Hadoop yet with practical features for solving certain types of tasks. Here are some of the key tasks suited to Spark:
- Interactive queries
- Disbursed data sets in memory
- Enhancing the outcomes of iterative tasks/implicit data parallelism
- Fault tolerance
- Batch applications
- Iterative algorithms
- Streaming workloads
Spark has been woven into Scala for application development. Spark and Scala are integrated closely, allowing Scala to simply control distributed data sets as local collective objects, something that Hadoop can’t boast.
Spark and Hadoop can actually work together. Furthermore, Spark now has an RDD, resilient distributed database, a gathering of immutable objects distributed over a set of nodes. They support two types of operations; Transformation, creating a fresh RDD from a current one, passing the dataset to the function and then returning it as a new dataset. Action is when the final result is returned to the driver program or it is written to an external data store.
Below are some of the principal operations that are performed by applications, or drivers:
- Spark Core- the kernel of Spark
- Spark SQL- for the running of SQL/HQL queries
- Spark Streaming- for building data analytics apps with live streaming data
- Spark MLlib- machine learning library containing high-performance algorithms for scaling apps
- Spark GraphX- the graph computation engine
- SparkR- the R package for light-weight frontend use
While the above list defines the Ecosystem of Apache, R, SQL, Python, Scala, and Java make up the Apache Spark Core.
Hadoop
The software framework, Hadoop MapReduce is a great solution for writing apps that have a massive number of records with little effort. These records (multi-terabyte data-sets) are in-parallel on huge clusters in fault-tolerant mode. It will process significant quantities of data from file systems. Hadoop Distributed File System was designed particularly for high-performance and to read, process, and save data on a set of multiple computers.
It will normally separate the input data into separate chunks which are then processed by mapping tasks in parallel, with inputs and outputs stored in a file-system. It is also dedicated to scheduling tasks, controlling and re-executing any failed tasks automatically.
Often, the compute nodes are identical to the storage nodes so that the framework and productively schedule tasks in between nodes.
There are many other components within the Hadoop ecosystem, for example:
- MapReduce- for data processing
- Yarn- for the management of clusters
- HDFS- the high-performance Hadoop Distributed File System
- Pig- for scripting
- Hive- for SQL Query
- Mahout- a Distributed Linear Algebra Framework
- HBase- a columnar store
- Oozie- for workflow management
Spark and Hadoop Uses and Cloud Systems
If you are looking for a fast, and simple cloud service, Cloud Dataproc is a cloud service for Apache Spark and Apache Hadoop cluster clusters with the advantage of ease and value for money. It can be integrated with other Google Cloud Platforms services and is successful at data processing, analytics, and learning.
You can take advantage of many features such as resizable clusters, integration, switching between Apache Spark and Apache Hadoop versions, amazing developer tools, automatic or manual configuration, and flexible virtual machines.
Amazon EMR
Amazon EMR includes built-in assistance for Apache Spark. You have the means to create managed Apache Spark clusters via AWS Management Console, AWS Line Interface, or Amazon EMR API. Amazon ENR has various other features that make the whole process fast and straightforward.
Our Conclusions About Spark and Hadoop
Spark, created in Scala, is approximately 100 times faster than Hadoop. It uses an in-memory model to transfer data between RDD. It is ideal for iterative algorithms to process streaming data, real-time analysis, ML algorithms, and large graph processing.
On the other hand, Hadoop, created in Java, is still faster than non-distributed systems. It uses HDFS to read, process and store huge amounts of data, and this is its forte. It is also a good solution for that time is not so crucial, for step-by-step processing of large databases, and for massive data storage.
Like many software solutions, the first thing you will need to do is understand the nature of your project and then understand which of the two is more suitable for you.
