博客

Yield

yield是返回一个生成器对象，通过next()方法来得到结果

import time


def func():
    start_time = time.time()
    num_list = []
    for i in range(1, 100000000):
        num_list.append(i)
    end_time = time.time()
    cost_time = end_time - start_time
    print(cost_time)
    yield num_list


print(func()) 
func()
next(func())
-------------------------------------------------
<generator object func at 0x00000293CB694138>
8.696713209152222

直接调用func()得不到结果，通过next()方法，得到所用时间

scrapy中的yield

1. yield后面跟回调函数。通过 callback 参数为yield请求添加回调函数，在请求完成之后会将响应作为参数传递给回调函数。

class Douban(scrapy.Spider):
    name = 'douban250'

    def start_requests(self):
        urls = ['https://movie.douban.com/top250']
        for url in urls:
            yield scrapy.Request(url, headers=headers,dont_filter=True,callback=self.parse)

2. yield后面跟scrapy.Item 对象，scrapy框架会将这个对象传递给 pipelines.py做进一步处理

    def get_data(self, response):
        result = json.loads(response.text)
        result = dict(result)
        result = result["list"]
        ids = [result[id]["cover"] for id in range(0, len(result))]
        names = [result[id]["name"] for id in range(0, len(result))]
        hero_ids = [result[id]['hero_id'] for id in range(0, len(result))]
        self.item['image_urls'] = ids
        self.item['images'] = names
        yield self.item

Return

return代表结束函数，return 后面的代码块不会执行，返回该函数的执行结果

import time


def func():
    start_time = time.time()
    num_list = []
    for i in range(1, 1000):
        num_list.append(i)
        return num_list
    end_time = time.time()
    cost_time = end_time - start_time
    print(cost_time)

print(func())
------------------------------------------------------------------
[1]