What is the most important thing in the era of big data?

With the advent of the cloud era, big data has attracted more and more attention. Zhucloud's team of analysts believes that big data is usually used to describe a large amount of unstructured and semi-structured data created by a company, and it will take too much time and money when it is downloaded to a relational database for analysis. Big data analysis is often associated with cloud computing, because real-time analysis of large data sets requires a framework such as MapReduce to distribute work to dozens, hundreds or even thousands of computers.

In short, the ability to quickly obtain valuable information from all kinds of data is big data technology. It is important to understand this point, and it is this point that makes this technology have the potential to go to many enterprises. The four "V's" or features of big data have four levels: First, the amount of data is huge. Jump from TB level to PB level; Second, there are many data types. The aforementioned blogs, videos, pictures, geographic information and so on. Third, the value density is low. Take video as an example, in the process of continuous monitoring, the data that may be useful is only one or two seconds. Fourth, the processing speed is fast. 1 the second law. This last point is also crucial for traditional data mining technology.

The difference. The industry classifies it into four "V"-volume, variety, value and speed. Internet of Things, cloud computing, mobile Internet, car networking, mobile phones, tablet computers, PCs, and various sensors all over the world are all sources or ways of carrying data, Zhucloud.

Examples include weblogs, RFID, sensor networks, social networks, social data (due to the society of data revolution), Internet texts and files; Internet search index; Call detailed records, astronomy, atmospheric science, genomics, biogeochemistry, biology and other complex and/or interdisciplinary scientific research, military reconnaissance and medical records; Video files of photographic archives; And large-scale e-commerce.

Big data requires special technology to effectively process large amounts of data within tolerance time. Technologies suitable for big data include MPP database, data mining power grid, distributed file system, distributed database, cloud computing platform, Internet, extensible storage system and so on.

Some (but not all) MPP relational databases have the ability to store and manage PB data. Implicit loading, monitoring, backup and optimization of large data tables are used in RDBMS.

Sloan Digital Sky Survey collected more data in the first few weeks than all the data in the history of astronomy as early as 2000. Since then, it has accumulated 140 terabytes of information. The successor of this telescope, the Great Survey Telescope, will go online on 20 16 to obtain data. Every five days, Wal-Mart handles transactions with more than1100,000 customers per hour. Conversely, the import database estimates that more than 2.5 PB is equivalent to the Library of Congress 167 books.

FACEBOOK processes 40 billion photos from its user base. The decoding of the original human genome, which originally required 10 years, can now be realized within one week.

The influence of "big data" has increased the demand for information management experts. Oracle Bone Inscriptions, IBM, Microsoft and SAP spent more than $65.438+0.5 billion on software intelligent data management and analysis. The value of this industry itself exceeds $654.38+000 billion, twice a year, with an increase of nearly 654.38+00%, which is probably due to the rapid development of the overall software business. Big data comes into being because we live in a society with more things. There are 4.6 billion mobile phone users in the world, with US$ 6,543.8 billion and 2 billion people accessing the Internet.

Basically, people interact with data or information more than ever before. From 1990 to 2005, more than 1 100 million people in the world entered the middle class, which means that more and more people benefit from this money, which in turn will lead to more literacy information growth. Cisco predicts that by 20 13, the traffic flowing on the Internet will reach 667 per year.

The first institution to propose that the era of "big data" has arrived is McKinsey, a world-renowned consulting company. Mckinsey pointed out in the research report that data has penetrated into every industry and business function field and gradually become an important factor of production; And people's use of massive data will herald a new wave of productivity growth and consumer surplus.

"After the release of McKinsey's report, big data quickly became a hot concept in the computer industry and attracted great attention from the financial community." With the continuous development of Internet technology, data itself is an asset, which has been recognized in the industry. "If cloud computing provides places and channels for data assets to be saved and accessed, then how to revitalize data assets and make them serve national governance, corporate decision-making and even personal life is the core issue of big data, and it is also the inner soul and inevitable upgrade direction of cloud computing."

In fact, global Internet giants have realized the importance of data in the era of "big data". Global IT giants, including EMC, Hewlett-Packard (Weibo), IBM and Microsoft (Weibo), have acquired "big data" related vendors to realize technology integration, which also shows their emphasis on "big data".

As a relatively new concept, "Big Data" has not been directly put forward by the China Municipal Government as a proper term to give policy support. However, in the 12th Five-Year Plan of the Internet of Things issued by the Ministry of Industry and Information Technology on February 8th, 65438, information processing technology was put forward as one of the four key technological innovation projects, including massive data storage, data mining and intelligent analysis of images and videos, which are all important components of big data. The other three key technological innovations, such as information perception technology, information transmission technology and information security technology, are also closely related to "big data".