What does a big data engineer do?

What does a big data engineer do? What abilities are required?

Big data is a very fashionable technical term at the moment. At the same time, it has naturally spawned some professions related to big data processing, which influence the business decisions of enterprises through data mining and analysis.

This group of people is called data scientists abroad. This title was first proposed by D.J. Pati and Jeff Hammerbacher in 2008. They later became the data science teams of LinkedIn and Facebook respectively. person in charge. The position of data scientist has now begun to create value in traditional U.S. telecommunications, retail, finance, manufacturing, logistics, medical, education and other industries.

However, in China, the application of big data has just begun, and the talent market is not yet so mature. “It is difficult for you to expect a generalist to complete all links in the entire chain. More companies will base their efforts on Use existing resources and shortcomings to recruit talents who can complement the existing team," Wang Yuyao, director of business analysis and strategy at LinkedIn, told China Business Weekly.

So each company has different requirements for big data work: some emphasize database programming, some highlight applied mathematics and statistics knowledge, and some require experience in consulting companies or investment banks. Some hope to find application-oriented talents who understand products and markets. Because of this, many companies will give these people who deal with big data some new titles and definitions based on their business types and team divisions: data mining engineers, big data experts, data researchers, user analysis experts, etc. are often We refer to the titles that appear in domestic companies as "big data engineers".

Wang Yuyao believes that in a mature data-driven company, "big data engineers" are often a team, which means everything from data collection, presentation, analysis and business insights to market transformation. The whole process. This team may include roles such as data engineers, analysts, product specialists, marketing specialists, and business decision-makers to complete the conversion from raw data to business value - in a nutshell, this is a team that supports the company in making business decisions. , discover important groups of business models.

Since domestic big data work is still at a stage to be developed, how much value can be mined depends entirely on the personal abilities of engineers. Experts who are already in this industry have given some general frameworks for talent needs, including computer coding ability, mathematics and statistics-related backgrounds. Of course, if you can have a deeper understanding of some specific fields or industries, you can make quick judgments about them. And it is more helpful to grasp the key factors.

Although for some large companies, company employees with master's and doctoral degrees are a better choice, Alibaba Group researcher Xue Guirong emphasized that academic qualifications are not the most important factor as long as they can process data on a large scale. Experience and curiosity about treasure hunting in the ocean of data will make you more suitable for this job.

In addition, an excellent big data engineer must have certain logical analysis capabilities and be able to quickly locate the key attributes and determinants of a business problem. "He has to know what is relevant, which is important, what kind of data is the most valuable, and how to quickly find the core needs of each business." said Shen Zhiyong, a data scientist at the United Nations Baidu Big Data Joint Laboratory. Learning ability can help big data engineers quickly adapt to different projects and become data experts in this field in a short time; communication skills can make their work go more smoothly, because big data engineers work mainly in two ways : Driven by the marketing department and driven by the data analysis department, the former needs to often understand the development needs from the product manager, while the latter needs to find the operations department to understand the actual transformation of the data model.

You can regard the above requirements as the direction of efforts to become a big data engineer, because according to the observation of Yan Liping, managing partner of Marlboro Ruihua, this is a big talent gap.

At present, domestic big data applications are mostly concentrated in the Internet field, and more than 56% of companies are preparing to develop big data research. "In the next five years, 94% of companies will need data scientists." Yan Liping said. Therefore, she also suggested that some people in companies who were originally engaged in data-related work should consider transformation.

This issue of "First Financial Weekly" interviewed three domestic Internet companies, BAT, and human resources experts in related fields. They explained to us how to become a big data engineer and the benefits of such positions from a workplace perspective. Current status of the workplace.

A What does a big data engineer do?

In the words of Xue Guirong, a researcher at Alibaba Group, big data engineers are a group of people who "play with data", playing with the commercial value of data and turning data into productivity. The biggest difference between big data and traditional data is that it is online, real-time, massive in scale, irregular in form, and unsystematic, so people who can "play" with this data are very important.

Shen Zhiyong believes that if big data is imagined as a mine that keeps accumulating, then the job of a big data engineer is, “The first step is to locate and extract the data set where the information is located, which is equivalent to prospecting and mining. The second step is to turn it into information that can be directly used for judgment, which is equivalent to smelting. The last step is to visualize the data. ”

So analyzing history, predicting the future, and optimizing choices are big data. The three most important tasks for engineers when "playing with data". Through these three lines of work, they help companies make better business decisions.

Find out the characteristics of past events

A very important job of big data engineers is to find out the characteristics of past events by analyzing data. For example, Tencent's data team is building a data warehouse to sort out the large and irregular data information on all the company's network platforms and summarize the characteristics that can be queried to support the data needs of the company's various businesses, including advertising. placement, game development, social networking, etc.

The biggest role of finding out the characteristics of past events is to help companies better understand consumers. By analyzing the user's past behavior trajectory, you can understand this person and predict his behavior. "You can know what kind of person he is, his age, hobbies, whether he is a paying Internet user, what type of games he likes to play, and what he usually likes to do online." General Manager of Beijing R&D Center of Tencent Cloud Computing Co., Ltd. Zheng Lifeng told China Business Weekly. The next step is to reach the business level, where relevant services can be recommended for various groups of people, such as mobile games, or new business models can be derived based on different characteristics and needs, such as WeChat's movie ticket business.

Predict what may happen in the future

By introducing key factors, big data engineers can predict future consumer trends. On Alimama’s marketing platform, engineers are trying to help Taobao sellers do business by introducing weather data. "For example, if this summer is not hot, it is very likely that some products will not sell as well as last year. In addition to air conditioners and fans, vests, swimsuits, etc. may be affected. Then we will establish the relationship between meteorological data and sales data and find Related categories will warn sellers in advance about inventory turnover,” Xue Guirong said.

At Baidu, Shen Zhiyong supports the model development of some products of "Baidu Prediction", trying to use big data to serve a wider range of people. Already online include World Cup predictions, college entrance examination predictions, scenic spot predictions, etc. Taking Baidu attraction prediction as an example, big data engineers need to collect all the key factors that may affect the flow of people in attractions over a period of time to predict, and grade the future congestion of various attractions across the country - in the next few days, what will it be? Smooth, crowded, or generally crowded?

Find the optimal results

Based on the business nature of different enterprises, big data engineers can achieve different purposes through data analysis.

For Tencent, Zheng Lifeng believes that the simplest and most direct example that can reflect the work of big data engineers is option testing (AB Test), which helps product managers make decisions between alternatives A and B. choose.

In the past, decision makers could only make judgments based on experience, but now big data engineers can conduct real-time tests on a large scale—for example, in the example of social network products, let half of the users see interface A and the other half use interface B, and observe Statistics of click-through rates and conversion rates over a period of time will help the marketing department make the final choice.

As an e-commerce company, Alibaba hopes to use big data to target precise groups of people and help sellers do better marketing. "What we look forward to more is that you can find a group of people who are more interested in the product than existing users." Xue Guirong said. An example from Taobao is that a certain ginseng seller’s original target group for promotion was pregnant women, but after mining the correlation between the data, engineers discovered that the marketing targeted at pregnant women had a higher conversion rate.

B Required abilities

Mathematics and statistics related background

As far as the three major Internet companies of BAT we interviewed, for big data engineers The requirements are all for a master's or doctoral degree with a background in statistics and mathematics. Shen Zhiyong believes that data workers who lack theoretical background are more likely to enter a technical danger zone (Danger Zone) - a bunch of numbers, according to different data models and algorithms, you can always figure out some results, but if you don’t know What that means is not a truly meaningful result, and such a result can easily mislead you. "Only with certain theoretical knowledge can we understand models, reuse models and even innovate models to solve practical problems." Shen Zhiyong said.

Computer coding ability

Practical development capabilities and large-scale data processing capabilities are some necessary elements for a big data engineer. "Because the value of many data comes from the mining process, you have to do it yourself to discover the value of gold." Zheng Lifeng said.

For example, many records generated by people on social networks are unstructured data. How to extract meaningful information from these clueless texts, voices, images and even videos is a challenge. Big data engineers need to dig it themselves. Even in some teams, big data engineers' responsibilities are mainly business analysis, but they must also be familiar with the way computers process big data.

Knowledge of specific application fields or industries

In Yan Liping’s view, a very important point in the role of big data engineer is that it cannot be separated from the market, because big data can only be related to specific fields. Applications are combined to create value. Therefore, experience in one or more vertical industries can help candidates accumulate knowledge of the industry, which will be of great help in becoming a big data engineer in the future. Therefore, this is also a more convincing bonus when applying for this position.

“He must not only understand data, but also have business acumen. Whether it is retail, medicine, gaming, tourism or other industries, he must have a certain understanding of some of these fields. It is best to do business with the company. In the same direction," Xue Guirong also made an analogy, "In the past, we said that some luxury store clerks were snobbish and could tell at a glance whether they could afford something or not, but this group of people is precisely perceptive. We think they are Experts in this industry. Another example is a person who knows the medical industry. When he considers medical insurance business, he will not only consider people’s hospital records, but also diet data, which are all based on their understanding of this field. "

C Career development of big data engineers

How to become a big data engineer

Due to the current shortage of big data talents, it is difficult for companies to recruit suitable Talent - you must have a high degree of education, and preferably also have experience in large-scale data processing. Therefore, many companies will dig internally.

In August this year, Alibaba held a big data competition. It took out the data on the Tmall platform, removed sensitive issues, and then put it on the cloud computing platform and handed it over to more than 7,000 teams for competition. The competition is divided into internal competition and external competition. “In this way, we can motivate internal employees, while also discovering external talents, so that big data engineers in various industries can emerge.

”

Yan Liping suggested that people who have been engaged in database management, mining, and programming for a long time include traditional quantitative analysts, Hadoop engineers, and any managers who need to use data to make judgments and decisions at work. Anyone, such as operations managers in certain fields, can try this position, and experts in various fields can also become big data engineers as long as they learn to use data.

Salary and benefits

As the "giant panda" among IT professions, the income and benefits of big data engineers can be said to be at the top of the category. According to Yan Liping's observation, 10% of domestic IT, communications, and industry recruitment are related to big data. , and the proportion is still rising. Yan Liping said, "The arrival of the big data era is sudden. The development momentum in the country is radical, but the talent is very limited. Now it is completely in short supply. "In the United States, the average annual salary of big data engineers is as high as 175,000 US dollars. It is understood that in top domestic Internet companies, the salary of big data engineers at the same level may be 20% to 30% higher than other positions, and it is quite popular among enterprises. Attention.

Career development path

Due to the small number of big data talents, the data departments of most companies generally have a flat hierarchical model, which is roughly divided into data analysts, There are 3 levels: senior researcher and department director. Large companies may divide different teams according to the dimension of application field, while in small companies, some Internet companies that place special emphasis on big data strategy will set up separate top positions - such as Chief Data Officer of Alibaba. “Most people in this position will develop in the direction of research and become important data strategy talents. Yan Liping said. On the other hand, big data engineers understand business and products as well as business department employees, so they can also move to the product department or marketing department, or even rise to the company's senior management.