博客

1.某网站爬取抓包

2.请求：

category_id = str(i['category_id'])
req_body = '{"attributes":"","keyword":"","categoryId":%s,"productId":0,"manufacturerId":0,"searchProductTypeEnum":2,"sort":"[{\"RankScore\":\"desc\"}]","searchRelation":{"clientType":4},"pageSize":12,"pageIndex":%s}'%(category_id,page)
yield scrapy.Request(url=self.url, method="POST", body=req_body, headers=self.headers,
                     meta={"category": i},
                     callback=self.get_detail, dont_filter=True)

3.报错：

4.解决：将请求参数转换成json

page = 1
category_id = str(i['category_id'])
formdata = {
    'attributes':'',
    'categoryId':category_id,
    'keyword':'',
    'manufacturerId':'0',
    'pageIndex':str(page),
    'pageSize':'12',
    'productId':'0',
    'searchProductTypeEnum':'2',
    'searchRelation: {clientType':'4}',
    'sort':'"[{\"RankScore\":\"desc\"}]"',
}
req_body = json.dumps(formdata)
yield scrapy.Request(url=self.url, method="POST", body=req_body, headers=self.headers,
                     meta={"category": i},
                     callback=self.get_detail, dont_filter=True)

5.成功响应