What is a reptile?

Crawler technology is an automatic program.

Crawler is an automatic program, which can grab data information from web pages and save it. Its principle is to simulate a browser sending a network request, accepting the request and responding, and then automatically grabbing Internet data according to certain rules.

Search engines use these crawlers to crawl from one website to another, track links in web pages, and visit more web pages. This process is called crawling, and these new websites will be stored in the database for searching. In short, the crawler keeps visiting the Internet, and then gets the information you specify from it and returns it to you. On our Internet, there are countless reptiles that grab data at any time and return it to users.

The role of reptile technology

1, get the webpage.

Getting a web page can be simply understood as sending a web request to the server of the web page, and then the server returns the source code of the web page to us. The underlying principle of communication is complicated. Python has packaged the URL library and the requests library for us, so that we can send all kinds of requests very simply.

Step 2 extract information

The obtained web page source code contains a lot of information. If we want to extract the information we need, we need to further filter the source code. You can choose the re library in python to extract information through regular matching, or you can use the BeautifulSoup library (bs4) to analyze the source code. Besides the advantages of automatic coding, bs4 library can also output source code information in a structured way, which is easier to understand and use.

Step 3 save the data

After extracting the useful information we need, we need to save it in Python. You can use the built-in function open to save as text data, or you can use a third-party library to save as other forms of data. For example, you can save as common xlsx data through Panda Library, and if you have unstructured data such as pictures, you can also save as an unstructured database through pymongo Library.