python - Scrapy a website that need use cookies -


I am creating a scrap to scrap a website, but this website is working with cookies, I do not I know how I can make instructions to scrape the website's data by using cookies. 

  class dmozspider (spider): name = "dmoz" allowed_domains = ["dmoz.org "] Start_urls = [" http: //Www.dmoz.org/Computers/Programming/Languages/Python/Books/ "] DRF Pars (self, response): cell = selector (Re: =) Site = sel.HtmlXPathSelector ('// ul [@ class = "directory-url"] / Li') Item = [] For sites in the sites: items = website () item ['name'] = site. xpath ('a / text ()'). Remove () item ['url'] = site Exped ('a / @ href'). Remove () items. Upload (item) return item   
 How can I add cookies to this URL correctly   
 
 
 @ omair_77, you can override your spider's  start_requests  method to add cookies. Initial request by your spider:  
  def start_reques ts (self): return [request (url = "http://www.example.com", cookies = {'currency': 'USD', 'country': 'you' })   
 In this way, your spider The first request of the person will be with those cookies, and your  parse  method will be with the first response.

Comments

Post a Comment

Popular posts from this blog

Java - Error: no suitable method found for add(int, java.lang.String) -

I'm in the middle of homework work and I'm stuck. At this point in my code, I think that I should have a GUI window that opens and allows me to type "inserted text number". Notice is not going anywhere but at this point, If I pass through the problem then it will be in a linked list. I am getting two of the same error for the lines. Add (index, element); And I can not seem to get past it, there is no suitable method for the error "add (int, java. string string)". Code is below, please advise. To clarify - this will not be a method error because it is a linked list. There should not be any way involved. import java.awt. *; Import java.awt.event. *; Import javax.swing. *; Import java.util. *; Import java.util.Scanner; Import java.util.LinkedList; Public class TopTenList JFrame {Private TopTenList tt; See the Private JTextArea list; Private JTextField cmdTextField; Private JTextField resultsTestfield; // This GUI window is the code for the public toptist...
Read more

c# - passing input text from view to contoller with FacebookContext using Facebook app -

I'm trying to pass an input text from the viewer to the controller in the Facebook Birthday app: Link to app: In the Home Controller [FacebookAuthorize ("email", "user_photos")] Public async Tasks & lt; ActionResult & gt; Index (FacebookContext reference) {if (ModelState.IsValid) {var user = waiting for reference. Client.Gate CurrentUserAsync & lt; MyAppUser & gt; (); View return (user); } See Return ("Error"); } [Http post] [facebook authorized ("email", "user_photo")) public async task & lieutenant; Actionsclass & gt; Index (string textile, facebookcontonte context) {if (ModelState.IsValid) {var user = wait context.Client.GetCurrentUserAsync & lt; MyAppUser & gt; (); // See my code, see the use of txt here (user); } See Return ("Error"); } View: @using facebookpostc.models @ Microsoft.AspNet.Mvc.Facebook.models using @ Microsoft.AspNet.Mvc. Facebook @model MyAppUser @ {ViewB...
Read more

Calling a C++ function from C# by passing a string with variable size to it -

I am calling a string with a C + code and calling from the C # code. The C ++ function is responsible for filling the string material. The following code is: C #party: var abc = new string builder (4096); // will have to change this var result = Basic methods. SignCrcFile (ABC); C ++: bool __cdecl SignCrcFile (char * abc) {... char * tempStr = "hello all"; // for example // copy it to ABC string strcpy (ABC, tempStr); Back true; } The problem is that tempStr's size in the C ++ function is dynamic (the above assignment for TempStr is only an example). In such a case, it is not a hard code that the size of ABC string is in the size of 4096 bytes on the C #. The first solution can be a solution to create 2 functions on the C ++ side to get the size of the string. Then allocate that size using StringBuilder in the C # side then use the next function to bring the entire string from C ++ to the C #. But no suggestion this may not be the most optimal way? ...
Read more