Waiting for answer This question has not been answered yet. You can hire a professional tutor to get the answer.
I have this code for one movie. Is there a way to web crawl more than one move at a time?
I have this code for one movie. Is there a way to web crawl more than one move at a time? I have to collect this data for the 25 top and bottom movies. Also the code main_content = urljoin(url,soup.select(".load-more-data")[0]['data-ajaxurl']) does not seem to be pulling for than the first page of data.
url = 'https://www.imdb.com/title/tt0111161/reviews?ref_=tt_ql_3'
res = requests.get(url)
soup = BeautifulSoup(res.text,"lxml")
main_content = urljoin(url,soup.select(".load-more-data")[0]['data-ajaxurl'])
response = requests.get(main_content)
broth = BeautifulSoup(response.text,"lxml")
for item in broth.select(".review-container"):
title = item.select(".title")[0].text
review = item.select(".text")[0].text
date = item.select(".review-date")[0].text
user = item.select(".display-name-link")[0].text
#score = item.select(".rating-other-user-rating")[0].text
print("Title: {}nnDate: {}nnUser: {}nnReview: {}nn".format(title,date,user,review,))