python - Scrapy a website that need use cookies -


I am creating a scrap to scrap a website, but this website is working with cookies, I do not I know how I can make instructions to scrape the website's data by using cookies. 

  class dmozspider (spider): name = "dmoz" allowed_domains = ["dmoz.org "] Start_urls = [" http: //Www.dmoz.org/Computers/Programming/Languages/Python/Books/ "] DRF Pars (self, response): cell = selector (Re: =) Site = sel.HtmlXPathSelector ('// ul [@ class = "directory-url"] / Li') Item = [] For sites in the sites: items = website () item ['name'] = site. xpath ('a / text ()'). Remove () item ['url'] = site Exped ('a / @ href'). Remove () items. Upload (item) return item   
 How can I add cookies to this URL correctly   
 
 
 @ omair_77, you can override your spider's  start_requests  method to add cookies. Initial request by your spider:  
  def start_reques ts (self): return [request (url = "http://www.example.com", cookies = {'currency': 'USD', 'country': 'you' })   
 In this way, your spider The first request of the person will be with those cookies, and your  parse  method will be with the first response.

Comments

Post a Comment

Popular posts from this blog

How can I add a seperate action listener to all 14 of my buttons at once that can determine player turn?(java) -

I have a big code, for a game I'm calling Tommy's Kumamak. I know that this is not beautiful, but I am very naive and for some reasons I know from using arrays, anyway, my problem is that I can not add a variable outside the listener and within the action listener I can modify the variable (pTurn ++) from How can I go about adding a count to that variable from inside the action listener? 1 error was found: file: c: \ user \ public \ togiz_kumalak.java [line: 130] error: local variables pTurn is accessed from within internal class; Need to be declared final int pTurn = 0; Button1.addActionListener {new zero-action action (action event e) {pTurn ++; if (pTurn% 2 == 0) {display.append ("\ n Player 2 cup 1");} If ( PTurn% 2! = 0) {display.append ("\ nPlayer 1Cup 1")}}); In that case, move the contents of the functionalities into orbit and then call the action For example, display the method from: Private Zero Button Press {pTurn ++; If (pTurn% 2...
Read more

php - Sorting an multidimension array using usort fails -

I tried to use custom-made YouTube fun cats when I ran 5 cell arrays in 30 seconds and then a Throwing a serious error. I tried to know that I have been made an infinite loop, but he did not install it. iPhone 5S 32GB [Price] => 0 [Description] => A new apple product [Image_URL] = & Gt; [Business_ID] = & gt; 6 [keyword] = & gt; ["iPhone", "iPhone 5s", "Apple", "32 GB"]) [1] = & gt; array ([name] = & Gt; iPhone 4S 32g [Price] => 300 [Description] => Apple iPhone 4S = [Image_URL] => [Business_ID] => 6 [Keyword] = & gt; [ "IPhone 4S", "" iPhone "," Apple "," 32 GB "])) search: $ searchTerm =" iphone 5s "; Code: if (! SearchTerm below!) {$ SearchTermWords = Explosion (" ", $ searchTerm); $$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ ['Keyword']; $ bKeywords = $...
Read more

Simple file compression in C -

I am new to programming with C, and I need elsewhere that I can read on file compression in C I am I only need a simple accredited method, for example it will help but mostly I can read it to understand. Probably a good starting point RLE is very simple but it is not so insignificant to apply see it :
Read more