I was downloading an entire course material(PDFs) of one math course on Firefox. Then I thought that I should be able to do it in Python too. I tried, and of course you can!!!
import requests import os def download(url): chunk_size = 10 file_name = url.split('/')[-1] response = requests.get(url, stream=True) with open(file_name, 'wb') as out_file: for chunk in response.iter_content(chunk_size): out_file.write(chunk) #out_file.write(response.content) del response def open_url_text_file(urls): with open(urls) as f: for url_line in f: url_line = url_line.strip() # Remove new line(\n) characters # Otherwise downloaded files won't open!!! download(url_line) print url_line if __name__ == '__main__': open_url_text_file(text file that contains urls) #download(single url)
The script has two steps. First step, open_url_text_file() is where it opens a text file with urls. You can skip this step by calling download() with a single url. The idea is that I would have bunch of urls and then I need to run the script only once. Because of the if __name__ == ‘__main__’, you can use only functions by importing them.
You can download by out_file.write(response.content) which I commented out. However, according to Requests Documentation the other method is recommended. If your file size is not small, it’s easy to understand why.
If you are wondering why I’m not closing a file which I opened with ‘with’, you don’t need to worry about closing since ‘with’ handles it. It’s called predefined clean-up actions. According to Python docs:
“Some objects define standard clean-up actions to be undertaken when the object is no longer needed, regardless of whether or not the operation using the object succeeded or failed. “
What ‘with’ statement does is that it wraps its block with methods defined by a context manager.
So file object defines its standard clean-up action that it will be closed(__exit__) eventually and within ‘with’ statement, it knows that it’s to be closed at the exit of ‘with’ block.
Hei hei 😀