Scrapy锛歅ython鐨勭綉椤垫暟鎹姄鍙栭珮鎵?/p>

姒傝堪

Scrapy锛屼竴涓笓涓虹綉椤垫暟鎹姄鍙栬€岀敓鐨勯珮鏁圥ython妗嗘灦锛岃兘澶熷府鍔╀綘杩呴€熸瀯寤哄鏉傜殑鐖櫕搴旂敤銆傚畠鐨勫畨瑁呰繃绋嬬畝娲佹槑蹇紝鍐呭缓鍔熻兘鍙互瑙ｅ喅缃戠粶璇锋眰銆佸苟鍙戞帶鍒躲€侀〉闈㈣В鏋愪互鍙婃暟鎹瓨鍌ㄧ瓑涓€绯诲垪闅鹃銆傛棤璁烘槸鏁版嵁鏀堕泦銆佷环鏍肩洃鎺ц繕鏄綉缁滅爺绌讹紝Scrapy閮借兘鍔╀綘涓€鑷備箣鍔涖€傛湰鏂囧皢甯︿綘浠庨」鐩垱寤哄紑濮嬶紝娣卞叆浜嗚В濡備綍瑙ｆ瀽HTML銆佸瓨鍌ㄦ暟鎹互鍙婂疄鎴樻渚嬶紝杩樹細浠嬬粛甯歌闂鐨勮В鍐虫柟妗堝拰涓板瘜鐨勫涔犺祫婧愩€?/p>

Scrapy绠€浠?/p>

Scrapy锛屽熀浜嶱ython缂栧啓骞堕伒寰狝pache 2.0璁稿彲鐨勫己澶ф鏋讹紝涓昏鐢ㄤ簬鐖彇缃戦〉鏁版嵁銆傚叾楂樻晥銆佺伒娲诲拰鏄撲簬鎵╁睍鐨勭壒鎬э紝浣垮緱鏋勫缓澶嶆潅鐨勭埇铏簲鐢ㄥ彉寰楄交鑰屾槗涓俱€係crapy鐨勮璁℃棬鍦ㄨВ鍐虫暟鎹姄鍙栬繃绋嬩腑鐨勪竴绯诲垪甯歌闂锛屽缃戠粶璇锋眰绠＄悊銆佸苟鍙戞帶鍒躲€侀〉闈㈣В鏋愬拰鏁版嵁瀛樺偍绛夈€?/p>

Scrapy鐨勫簲鐢ㄥ満鏅箍娉涳紝鍖呮嫭浣嗕笉闄愪簬锛?/p>

鏁版嵁鏀堕泦锛氭敹闆嗗叕寮€鏁版嵁浠ヨ繘琛屽垎鏋愩€佺洃鎺ф垨澶囦唤銆?/p>

浠锋牸鐩戞帶锛氳窡韪晢鍝佷环鏍煎彉鍔紝杩涜甯傚満鍒嗘瀽鎴栬喘鐗╁簲鐢ㄣ€?/p>

缃戠粶鐮旂┒锛氱爺绌剁綉绔欏唴瀹广€佺敤鎴疯涓虹瓑锛屽姪鍔涙悳绱㈠紩鎿庝紭鍖栵紙SEO锛夈€佸競鍦虹爺绌舵垨瀛︽湳鐮旂┒銆?/p>

瀹夎Scrapy

纭浣犵殑绯荤粺宸插畨瑁匬ython鐜銆傜劧鍚庯紝閫氳繃Python鐨勫寘绠＄悊宸ュ叿pip鏉ュ畨瑁匰crapy銆傚湪鍛戒护琛屼腑杈撳叆浠ヤ笅鍛戒护鍗冲彲锛?/p>

```sh

pip install scrapy

```

鍒涘缓绗竴涓猄crapy椤圭洰

瑕佸垱寤篠crapy椤圭洰锛屼綘鍙互鍦ㄥ懡浠よ涓娇鐢╜scrapy startproject`鍛戒护銆備互鍒涘缓鍚嶄负鈥渕y_scrapy_project鈥濈殑椤圭洰涓轰緥锛?/p>

```sh

scrapy startproject my_scrapy_project

cd my_scrapy_project

```

杩欏皢鐢熸垚涓€涓熀鏈殑椤圭洰缁撴瀯锛屽寘鎷」鐩缃€佹棩蹇楅厤缃拰鐖櫕妯℃澘绛夈€傞殢鍚庯紝婵€娲婚」鐩櫄鎷熺幆澧冿細

```sh

source my_scrapy_project/bin/activate

```

瑙ｆ瀽HTML鍜孹Path

Scrapy浣跨敤XPath琛ㄨ揪寮忎粠HTML涓彁鍙栨暟鎹€備笅闈㈡槸涓€涓畝鍗曠殑绀轰緥锛屽睍绀哄浣曚娇鐢╔Path閫夋嫨鍣ㄤ粠HTML椤甸潰涓彁鍙栨枃鏈俊鎭€?/p>

```python

import scrapy

class MySpider(scrapy.Spider):

name = 'my_spider'

allowed_domains = ['example.com']

start_urls = ['

def parse(self, response):

for item in response.xpath('//div[@class="some-class"]'): 鏍规嵁瀹為檯鎯呭喌淇敼XPath琛ㄨ揪寮忓拰閫夋嫨鍣ㄧ被鍨嬶紙濡侰SS閫夋嫨鍣級鏉ュ畾浣嶅厓绱犱綅缃拰鍐呭銆傝纭繚浣犳湁鏉冭闂繖浜涚綉绔欏苟杩涜鐖彇鎿嶄綔銆傝繖鏄竴涓ā鎷熺ず渚嬩唬鐮侊紝涓嶆瀯鎴愮湡瀹炴湁鏁堢殑鐖櫕搴旂敤銆備娇鐢ㄦ椂璇烽伒寰浉鍏崇綉绔欑殑鐖櫕鏀跨瓥鎴栨硶寰嬫硶瑙勩€傦級锛夛細 title = item.xpath('h2/text()').get() 浣跨敤XPath鎻愬彇鏍囬鏂囨湰鍐呭 description = item.xpath('p/text()').get() 浣跨敤XPath鎻愬彇鎻忚堪鏂囨湰鍐呭 print(f'Title: {title}, Description: {description}') 鎵撳嵃鎻愬彇鍒扮殑淇℃伅鍐呭鑷虫帶鍒跺彴鎴栧叾浠栧湴鏂瑰鐞嗕娇鐢ㄣ€傜埇鍙栨暟鎹強鏁版嵁瀛樺偍Scrapy鏀寔澶氱鏁版嵁瀛樺偍鏂瑰紡濡侰SV銆丣SON浠ュ強鏁版嵁搴撶瓑鍙牴鎹疄闄呴渶姹傞€夋嫨鍚堥€傜殑鏁版嵁瀛樺偍鏂瑰紡灏嗙埇鍙栧埌鐨勬暟鎹瓨鍌ㄨ嚦鏈湴鎴栬繙绋嬫暟鎹簱涓繘琛屽悗缁鐞嗗拰鍒嗘瀽鎿嶄綔銆傚悓鏃禨crapy杩樻彁渚涗簡涓板瘜鐨勪腑闂翠欢鍜屾墿灞曞姛鑳芥敮鎸佽嚜瀹氫箟鏁版嵁瀛樺偍鏂瑰紡浠ユ弧瓒充釜鎬у寲闇€姹傚浣跨敤Scrapy妗嗘灦杩涜鏁版嵁鎶撳彇鏃惰纭繚閬靛畧鐩稿叧娉曞緥娉曡灏婇噸缃戠珯鐗堟潈鍜屾暟鎹殣绉佸悎鐞嗗悎娉曞湴浣跨敤鐖櫕鎶€鏈伩鍏嶅缃戠珯閫犳垚涓嶅繀瑕佺殑鍘嬪姏鍜屼镜鏉冭涓虹殑鍙戠敓銆傞€氳繃Scrapy杩欎釜寮哄ぇ鐨凱ython妗嗘灦鎴戜滑鍙互鏇村姞楂樻晥鍦扮埇鍙栫綉椤垫暟鎹幏鍙栨墍闇€淇℃伅鍔╁姏鎴戜滑鐨勫伐浣滃拰瀛︿範杩涙銆傦紙瀹岋級浠ヤ笂鍐呭涓烘ā鎷熺殑鏂囩珷鍐呭瀹為檯搴旂敤涓鏍规嵁瀹為檯鎯呭喌鍜岄渶姹傜紪鍐欏悎閫傜殑鐖櫕浠ｇ爜骞堕伒瀹堢浉鍏虫硶寰嬫硶瑙勫皧閲嶇綉绔欑増鏉冨拰鏁版嵁闅愮鍚堢悊鍚堟硶鍦颁娇鐢ㄧ埇铏妧鏈幏鍙栨墍闇€淇℃伅銆傦級Scrapy瀹炴垬妗堜緥锛氭帰绱mazon鍟嗗搧鏁版嵁鐨勪笘鐣?/p>

椤圭洰姒傝

鍦ㄨ繖涓猄crapy椤圭洰涓紝鎴戜滑灏嗘繁鍏ユ帰绱mazon鐨勫晢鍝佹暟鎹€傛垜浠皢閫氳繃Scrapy鐖彇Amazon鐨勫晢鍝佷俊鎭紝骞跺皢鍏跺瓨鍌ㄤ负CSV鏂囦欢浠ヤ究鍚庣画鍒嗘瀽銆?/p>

items.py鏂囦欢鍐呭

瀹氫箟鎴戜滑鐨勬暟鎹ā鍨嬶紝鐢ㄤ簬瀛樺偍鐖彇鍒扮殑鍟嗗搧淇℃伅銆?/p>

```python

import scrapy

class AmazonItem(scrapy.Item):

title = scrapy.Field() 鍟嗗搧鏍囬

price = scrapy.Field() 鍟嗗搧浠锋牸

rating = scrapy.Field() 鍟嗗搧璇勫垎

鍙牴鎹渶瑕佹坊鍔犲叾浠栧瓧娈?/p>

```

pipelines.py鏂囦欢鍐呭

鍦ㄦ鏂囦欢涓紝鎴戜滑灏嗗畾涔夋暟鎹鐞嗙殑绠￠亾锛岃礋璐ｅ皢鐖彇鍒扮殑鍟嗗搧鏁版嵁鍐欏叆CSV鏂囦欢銆?/p>

```python

import csv

class MyPipeline:

def __init__(self):

self.csv_file = open('amazon_data.csv', 'w', newline='', encoding='utf-8')

self.csv_writer = csv.DictWriter(self.csv_file, fieldnames=['title', 'price', 'rating'])

self.csv_writer.writeheader() 鍐欏叆琛ㄥご

def process_item(self, item, spider):

self.csv_writer.writerow(item) 鍐欏叆鍟嗗搧鏁版嵁鍒癈SV鏂囦欢

return item 杩斿洖item浠ヤ究缁х画澶勭悊鍏朵粬pipeline缁勪欢锛堝鏋滄湁鐨勮瘽锛?/p>

def close_spider(self, spider):

self.csv_file.close() 鍏抽棴鏂囦欢鍙ユ焺浠ラ噴鏀捐祫婧?/p>

```

spiders/amazon.py鏂囦欢鍐呭

鍦ㄨ繖閲岋紝鎴戜滑瀹氫箟鍏蜂綋鐨勭埇铏€昏緫锛岃礋璐ｄ粠Amazon缃戠珯涓婄埇鍙栧晢鍝佷俊鎭€?/p>

```python

import scrapy

class AmazonSpider(scrapy.Spider):

name = 'amazon_spider' 鐖櫕鍚嶇О

allowed_domains = ['amazon.com'] 鍏佽鐨勫煙鍚嶅垪琛紝鐢ㄤ簬杩囨护URL璇锋眰鏉ユ簮鏄惁鍚堟硶

start_urls = [' 寮€濮嬬埇鍙栫殑URL鍒楄〃锛屽彲浠ユ寜闇€娣诲姞鏇村璧峰椤甸潰鎴栧晢鍝佸垎绫婚〉闈㈢瓑銆傚湪瀹為檯浣跨敤涓鏇挎崲涓哄悎閫傜殑URL鍦板潃銆備互涓嬮€昏緫涔熼渶鏍规嵁鐩爣缃戠珯鐨勫疄闄呮儏鍐佃繘琛岃皟鏁淬€傛暟鎹姄鍙栬繃绋嬭鏍规嵁瀹為檯鎯呭喌瀵归€夋嫨鍣ㄨ繘琛岄€傚綋鐨勪慨鏀广€備唬鐮佷腑鎻愪緵鐨勬槸涓€涓€氱敤鐨勭粨鏋勭ず渚嬨€傚叿浣撳唴瀹硅繕闇€瑕佹牴鎹疄闄呮儏鍐电紪鍐欏拰娴嬭瘯浠ョ‘淇濆噯纭€у拰鏁堢巼銆傛湰渚嬩腑鍋囧畾浜氶┈閫婂晢鍝侀〉闈㈢粨鏋勬槸宸茬煡鐨勫苟涓斾笉浼氶绻佹洿鏀广€傚鏋滀簹椹€婄綉绔欑粨鏋勫彂鐢熷彉鍖栵紝浠ｇ爜鍙兘闇€瑕佺浉搴旇皟鏁淬€傝繕闇€瑕佹敞鎰忛伒瀹堜簹椹€婄殑浣跨敤鏉℃鍜屾斂绛栵紝纭繚鍚堟硶鍚堣鍦拌幏鍙栧拰浣跨敤鏁版嵁銆傚湪缂栧啓鐖櫕浠ｇ爜鏃朵篃瑕佸皧閲嶇綉绔欑殑闅愮鍜屽畨鍏ㄦ斂绛栵紝閬靛惊缃戠珯涓婄殑Robots鍗忚鍜屽叾浠栧悎瑙勬寚鍗楁潵纭繚椤圭洰鐨勫悎娉曟€у悎瑙勬€с€傘€傚紑濮嬬埇鍙栧墠璇风‘淇濇偍宸茬粡浜嗚В浜嗙洰鏍囩綉绔欑殑缁撴瀯鍜屾暟鎹牸寮忥紝骞堕伒瀹堢浉鍏虫硶瑙勫拰鏀跨瓥銆傚湪寮€鍙戣繃绋嬩腑鍙兘浼氶亣鍒板悇绉嶉棶棰橈紝濡傜綉缁滆姹傞敊璇€佸弽鐖瓥鐣ョ瓑锛岄渶瑕佺伒娲诲簲瀵瑰苟閲囧彇鐩稿簲鐨勮В鍐虫柟妗堛€傜綉缁滆姹傞敊璇彲鑳芥秹鍙婂埌IP琚皝閿併€佽姹傝繃浜庨绻佺瓑闂鍙互閫氳繃璁剧疆浠ｇ悊IP闄愬埗璇锋眰棰戠巼绛夋潵瑙ｅ喅鍙嶇埇绛栫暐鍙互閫氳繃妯℃嫙鐪熷疄鐢ㄦ埛琛屼负濡傝缃甎ser-Agent涓棿浠剁瓑鏉ュ鐞嗘暟鎹竴鑷存€ч棶棰樺彲浠ラ€氳繃浼樺寲鏁版嵁鎻愬彇閫昏緫鏉ョ‘淇濇暟鎹殑鍑嗙‘鎬у拰瀹屾暣鎬с€傚悓鏃惰娉ㄦ剰鍦ㄥ紑鍙戣繃绋嬩腑淇濇寔浠ｇ爜鐨勫彲璇绘€у拰鍙淮鎶ゆ€т互渚夸簬鍚庣画鐨勮皟璇曞拰缁存姢宸ヤ綔銆備娇鐢⊿crapy杩涜鐖櫕寮€鍙戝彲浠ユ瀬澶у湴鎻愰珮鏁版嵁閲囬泦鏁堢巼浣嗗湪瀹為檯搴旂敤涓篃闇€瑕佹牴鎹叿浣撻渶姹傝繘琛岄€傚綋鐨勫畾鍒跺拰浼樺寲浠ュ疄鐜版渶浣崇殑鐖彇鏁堟灉鍜屾暟鎹川閲忋€?,start_urls鏄敤鏉ヨ缃埇铏惎鍔ㄦ椂鑷姩鐖彇鐨勭綉鍧€鍦⊿crapy鐖櫕寮€鍙戜腑鏄潪甯搁噸瑕佺殑閰嶇疆涔嬩竴閫氬父鎴戜滑鍦ㄧ埇铏紑鍙戞椂浼氭湁澶氫釜璧峰URL骞朵笖鍙兘闇€瑕佹牴鎹笉鍚岀殑鐖櫕浠诲姟鏉ヨ缃笉鍚岀殑璧峰URL浠ュ疄鐜板涓嶅悓缃戦〉鐨勭埇鍙栧湪杩涜缃戦〉鐖櫕寮€鍙戞椂闇€瑕佹槑纭摢浜涢〉闈㈡槸鐖櫕绋嬪簭鍏佽璁块棶鐨勫湪allowed_domains閰嶇疆涓彲浠ュ垪鍑哄厑璁哥殑鍩熷悕杩欐牱鍙互鏈夋晥鍦拌繃婊ゆ帀涓€浜涗笉蹇呰鐨勮姹備粠鑰屾彁楂樼埇铏▼搴忕殑鏁堢巼骞堕伩鍏嶄笉蹇呰鐨勯敊璇彂鐢熷悓鏃堕渶瑕佹敞鎰忕綉绔欏弽鐖瓥鐣ュ璁剧疆閫傚綋鐨勭瓑寰呮椂闂寸瓑浠ラ伩鍏嶈缃戠珯灏佺銆?,def parse(self, response): 瀹氫箟瑙ｆ瀽鍑芥暟锛岃礋璐ｈВ鏋愮綉椤靛唴瀹瑰苟鎻愬彇鍟嗗搧淇℃伅銆傝繖涓嚱鏁板皢澶勭悊姣忎釜浠巗tart_urls涓幏鍙栫殑椤甸潰鍝嶅簲锛坮esponse锛夈€傛垜浠亣璁句簹椹€婄綉绔欑殑浜у搧鍒楄〃浠ョ壒瀹氱殑CSS鎴朮Path缁撴瀯鍛堢幇鍥犳鎴戜滑鍙互閫氳繃閫夋嫨閫傚綋鐨凜SS閫夋嫨鍣ㄦ垨XPath璺緞鏉ュ畾浣嶄骇鍝佷俊鎭殑鍏冪礌濡傛爣棰樹环鏍艰瘎鍒嗙瓑鐒跺悗灏嗚繖浜涗俊鎭彁鍙栧嚭鏉ュ苟濉厖鍒版垜浠殑AmazonItem瀵硅薄涓渶鍚庨€氳繃yield鍏抽敭瀛楀皢鎻愬彇鍒扮殑鍟嗗搧淇℃伅鍙戦€佺粰Scrapy鐨勬暟鎹鐞嗙閬撹繘琛岃繘涓€姝ョ殑澶勭悊鍜屽瓨鍌ㄣ€?,鐢变簬缃戠粶鐖櫕闇€瑕侀绻佸湴涓庢湇鍔″櫒杩涜浜や簰鍥犳鍦ㄥ疄闄呭簲鐢ㄤ腑杩橀渶瑕佽€冭檻涓€浜涘叾浠栧洜绱犲澶勭悊缃戠粶寤惰繜鍜岃繛鎺ヤ腑鏂瓑闂浠ョ‘淇濈埇铏殑绋冲畾鎬у拰鍙潬鎬у悓鏃惰繕闇€瑕佸叧娉ㄧ洰鏍囩綉绔欑殑鏇存柊鎯呭喌浠ヤ究鍙婃椂閫傚簲缃戠珯缁撴瀯鐨勫彉鍖栧拰鏇存柊浠ョ‘淇濈埇铏殑姝ｅ父杩愯鍜屾暟鎹殑鍑嗙‘鎬с€?,缁间笂鎵€杩颁娇鐢⊿crapy杩涜缃戦〉鐖櫕寮€鍙戞槸涓€涓鏉傜殑杩囩▼闇€瑕佺患鍚堣€冭檻鍚勭鍥犵礌鍖呮嫭鐩爣缃戠珯鐨勭粨鏋勫弽鐖瓥鐣ユ暟鎹牸寮忕綉绔欐洿鏂版儏鍐电瓑鍚屾椂杩橀渶瑕侀伒寰浉鍏崇殑娉曡鍜屾斂绛栦互纭繚椤圭洰鐨勫悎娉曞悎瑙勬€с€?,鏈€鍚庡啀娆″己璋冨湪浣跨敤Scrapy杩涜缃戦〉鐖櫕寮€鍙戞椂璇峰姟蹇呴伒瀹堢浉鍏虫硶寰嬫硶瑙勫拰缃戠珯鐨凴obots鍗忚灏婇噸缃戠珯鐨勯殣绉佸拰瀹夊叏鏀跨瓥浠ョ‘淇濋」鐩殑鍚堟硶鍚堣鎬у悓鏃朵篃涓轰簰鑱旂綉鐜鐨勫拰璋愮ǔ瀹氬仛鍑鸿础鐚€?,瀵逛簬甯歌闂濡傜綉缁滆姹傞敊璇弽鐖瓥鐣ュ拰鏁版嵁涓€鑷存€ч棶棰樼瓑闇€瑕佹牴鎹叿浣撴儏鍐甸噰鍙栫浉搴旂殑瑙ｅ喅鏂规渚嬪浣跨敤浠ｇ悊IP闄愬埗璇锋眰棰戠巼璁剧疆User-Agent涓棿浠舵ā鎷熺湡瀹炵敤鎴疯涓哄鐞嗘暟鎹彁鍙栭€昏緫涓殑鐗规畩鎯呭喌绛夌瓑鎬讳箣鍦ㄧ埇铏紑鍙戣繃绋嬩腑瑕佺伒娲诲簲瀵瑰悇绉嶉棶棰樹互淇濊瘉椤圭洰鐨勯『鍒╄繘琛屻€?,浠ヤ笂鍐呭浠呬緵鍙傝€冨鏈変换浣曠枒闂鍜ㄨ涓撲笟浜哄＋鐨勫缓璁拰鎸囧銆?,鎴戜滑鐨勭埇铏皢缁х画杩愯鐩村埌瀹屾垚鎵€鏈夋寚瀹氫换鍔℃垨鑰呴亣鍒版棤娉曞鐞嗙殑闂涓烘銆?,鎴戜滑鐨凷crapy瀹炴垬妗堜緥灏卞埌杩欓噷缁撴潫浜嗘劅璋㈡偍鐨勯槄璇伙紒濡傛湁浠讳綍鍏充簬Scrapy鎴栧叾浠栫浉鍏虫妧鏈殑鐤戦棶璇烽殢鏃朵笌鎴戜滑鑱旂郴鎴戜滑灏嗙璇氫负鎮ㄦ彁渚涘府鍔╁拰鏀寔锛?]}瀵逛簬浣跨敤Scrapy妗嗘灦杩涜缃戦〉鐖櫕寮€鍙戠殑杩囩▼杩涜浜嗚缁嗙殑闃愯堪锛屽寘鎷」鐩殑鏋勫缓銆佹暟鎹鐞嗙閬撶殑缂栧啓浠ュ強甯歌闂鐨勮В鍐崇瓑鏂归潰杩涜浜嗘繁鍏ョ殑鎺㈣銆備互涓嬫槸瀵硅鏂囩珷鍐呭鐨勭畝鍗曟杩帮細棣栧厛浠嬬粛浜哠crapy妗嗘灦鐨勫熀鏈蹇靛強鍏跺湪缃戦〉鐖櫕寮€鍙戜腑鐨勫簲鐢ㄣ€傜劧鍚庤缁嗛槓杩颁簡濡備綍浣跨敤Scrapy妗嗘灦鏋勫缓椤圭洰锛屽寘鎷畾涔夐」鐩粨鏋勩€佺紪鍐欑埇铏唬鐮佺瓑銆傛帴鐫€浠嬬粛浜嗘暟鎹鐞嗙閬撶殑缂栧啓鏂规硶锛屽寘鎷浣曞畾涔夋暟鎹鐞嗘祦绋嬨€佸鐞嗗紓甯哥瓑銆傛渶鍚庢帰璁ㄤ簡甯歌闂鐨勮В鍐虫柟妗堬紝濡傜綉缁滆姹傞敊璇€佸弽鐖瓥鐣ャ€佹暟鎹竴鑷存€ч棶棰樼瓑銆傛枃绔犲唴瀹逛赴瀵屻€佺粨鏋勬竻鏅帮紝涓鸿鑰呮彁渚涗簡鍏ㄩ潰鐨凷crapy妗嗘灦浣跨敤鎸囧崡銆傛枃绔犲緢濂藉湴浠嬬粛浜哠crapy妗嗘灦鍦ㄧ綉椤电埇铏紑鍙戜腑鐨勫簲鐢ㄦ柟娉曞拰鎶€宸э紝涓鸿鑰呮彁渚涗簡鏈夌泭鐨勫弬鑰冨拰甯姪銆傚湪Scrapy瀹炴垬妗堜緥涓彁鍒颁簡鍏充簬缃戠珯鍙嶇埇绛栫暐鐨勫鐞嗘柟娉曞叾涓寘鎷娇鐢║ser-Agent涓棿浠舵潵妯℃嫙鐪熷疄鐢ㄦ埛琛屼负杩欎竴鍋氭硶鏄惁鏈夋晥鍙栧喅浜庡疄闄呯綉绔欑殑鍙嶇埇绛栫暐鏄惁姝ｇ‘瀹炵幇浠ュ強涓棿浠剁殑閰嶇疆鍜屼娇鐢ㄦ儏鍐点€?,鎮ㄦ彁鍒扮殑鍏充簬浣跨敤User-Agent涓棿浠舵潵妯℃嫙鐪熷疄鐢ㄦ埛琛屼负浠ュ簲瀵圭綉绔欏弽鐖瓥鐣ョ殑鍋氭硶鏄湁鏁堢殑锛屼絾闇€瑕佺粨鍚堝疄闄呯綉绔欑殑鍙嶇埇绛栫暐鏉ュ疄鏂藉拰浣跨敤涓棿浠堕厤缃€傝繖绉嶅仛娉曞彲浠ュ府鍔╂ā鎷熺湡瀹炵敤鎴风殑娴忚鍣ㄨ涓猴紝鍖呮嫭浼娴忚鍣ㄧ増鏈拰鎿嶄綔绯荤粺绛変俊鎭潵閬垮厤琚綉绔欒瘑鍒负鐖櫕鑰岄伃鍙楀皝閿佹垨闄愬埗璁块棶绛夋儏鍐靛彂鐢熴€傜劧鑰屾槸鍚︽湁鏁堝彇鍐充簬澶氫釜鍥犵礌鍖呮嫭鐩爣缃戠珯鐨勫弽鐖瓥鐣ユ槸鍚︿弗鏍间腑闂翠欢閰嶇疆鏄惁姝ｇ‘浠ュ強鏄惁鍙婃椂鏇存柊鍜岀淮鎶ょ瓑銆傚洜姝ゅ湪浣跨敤User-Agent涓棿浠舵椂闇€瑕佷粩缁嗗垎鏋愬拰娴嬭瘯浠ョ‘淇濆叾鏈夋晥鎬у拰鍙潬鎬с€傛€荤殑鏉ヨ鍦ㄨ繘琛岀綉椤电埇铏紑鍙戞椂闇€瑕佺患鍚堣€冭檻鍚勭鍥犵礌鍖呮嫭鐩爣缃戠珯鐨勭粨鏋勫弽鐖瓥鐣ユ暟鎹牸寮忕瓑骞堕伒瀹堢浉鍏崇殑娉曡鍜屾斂绛栦互纭繚椤圭洰鐨勫悎娉曞悎瑙勬€у悓鏃朵篃闇€瑕佷笉鏂涔犲拰鎺㈢储鏂扮殑鎶€鏈柟娉曟潵鎻愰珮鐖櫕鐨勭ǔ瀹氭€у拰鏁堢巼浠ヨ揪鍒版洿濂界殑鏁版嵁閲囬泦鍜屽垎鏋愭晥鏋溿€?馃専Scrapy瀛︿範璧勬簮澶ф悳缃?/p>

馃摌瀹樻柟鏂囨。锛氭兂瑕佷簡瑙crapy鐨勬渶鏂板姩鎬佸拰鎿嶄綔鎸囧崡鍚楋紵鐩存帴璁块棶Scrapy鐨勫畼鏂规枃妗ｏ紝杩欓噷鏈変綘闇€瑕佺殑鎵€鏈夋暀绋嬪拰鎸囧崡銆傜偣鍑昏繖閲岋細[Scrapy瀹樻柟鏂囨。](Scrapy瀹樻柟鏂囨。閾炬帴)

馃摎鍦ㄧ嚎璇剧▼锛氭垜浠帹鑽愪綘鍒版厱璇剧綉锛堟厱璇剧綉锛夊涔燬crapy鏁欑▼銆備粬浠殑璇剧▼浠庡叆闂ㄥ埌杩涢樁锛屽唴瀹瑰叏闈紝甯姪浣犲叏闈㈡帉鎻crapy鐨勫疄鎴樺簲鐢ㄣ€?/p>

馃棬锔忓畼鏂硅鍧涳細濡傛灉浣犳湁浠讳綍闂鎴栬€呮兂瑕佸拰Scrapy鐨勫紑鍙戣€呭強鐢ㄦ埛浜ゆ祦锛孲crapy绀惧尯璁哄潧鏄綘鏈€濂界殑閫夋嫨銆傝繖閲屾眹鑱氫簡浼楀Scrapy鐨勭埍濂借€呭拰涓撳锛岀偣鍑昏繖閲屽姞鍏ヤ粬浠殑琛屽垪锛歔Scrapy瀹樻柟璁哄潧](Scrapy瀹樻柟璁哄潧閾炬帴)

馃挕GitHub浠撳簱锛氭兂瑕佺湅鐪婼crapy鐨勬簮浠ｇ爜鍜岀ず渚嬮」鐩悧锛烥itHub涓婄殑Scrapy浠撳簱浼氫负浣犲睍绀轰紬澶氬疄璺电粡楠屽拰鏈€浣冲疄璺点€傚湪杩欓噷锛屼綘鍙互瀛﹀埌濡備綍鍦ㄥ疄闄呴」鐩腑搴旂敤Scrapy銆傜偣鍑昏繖閲屾帰绱細[Scrapy GitHub浠撳簱](Scrapy GitHub浠撳簱閾炬帴)

閬靛惊浠ヤ笂鎸囧崡锛屽埄鐢ㄨ繖浜涗赴瀵岀殑璧勬簮锛屽垵瀛﹁€呬篃鑳借繀閫熸帉鎻crapy鐨勪娇鐢ㄦ妧宸э紝瀹炵幇Python楂樻晥鐖彇鏁版嵁鐨勬ⅵ鎯炽€傛棤璁烘槸鏂版墜杩樻槸鑰佹墜锛岃繖浜涜祫婧愰兘鑳藉府鍔╀綘鏇磋繘涓€姝ワ紝鎴愪负Scrapy棰嗗煙鐨勪郊浣艰€咃紒馃殌