How to write a crawler program using python

How to use Python to write a crawler program:

1. First analyze the website content. The red part is the website article content div.

2. Just open a div and look at it. You can see that the blue part has no useful information except an article title. Pay attention to the place I outlined in the red part. You can know that it points to The hyperlink of the article address, then the crawler only needs to capture this address.

3. The next problem is the page turning problem. You can see that this is different from most websites. There is no page number label at the bottom, but view more.

4. However, when viewing the source file, there is a hyperlink. After testing, it points to the next page. Then by changing its last value, you can locate the corresponding page number.

The code is as follows: