[Python]How to download a file with Requests

I was downloading an entire course material(PDFs) of one math course on Firefox. Then I thought that I should be able to do it in Python too. I tried, and of course you can!!!


import requests
import os


def download(url):
    chunk_size = 10
    file_name = url.split('/')[-1]
    response = requests.get(url, stream=True)

    with open(file_name, 'wb') as out_file:
        for chunk in response.iter_content(chunk_size):
               out_file.write(chunk)        
           #out_file.write(response.content)
       del response
       

def open_url_text_file(urls):
    with open(urls) as f:
        for url_line in f:
            url_line = url_line.strip()            # Remove new line(\n) characters
                                                # Otherwise downloaded files won't open!!!
            download(url_line)
            print url_line
            
if __name__ == '__main__':
    open_url_text_file(text file that contains urls)
    #download(single url)

The script has two steps. First step, open_url_text_file() is where it opens a text file with urls. You can skip this step by calling download() with a single url. The idea is that I would have bunch of urls and then I need to run the script only once. Because of the if __name__ == ‘__main__’, you can use only functions by importing them.

You can download by out_file.write(response.content) which I commented out. However, according to Requests Documentation the other method is recommended. If your file size is not small, it’s easy to understand why.

If you are wondering why I’m not closing a file which I opened with ‘with’, you don’t need to worry about closing since ‘with’ handles it. It’s called predefined clean-up actions. According to Python docs:

“Some objects define standard clean-up actions to be undertaken when the object is no longer needed, regardless of whether or not the operation using the object succeeded or failed. “

What ‘with’ statement does is that it wraps its block with methods defined by a context manager.
So file object defines its standard clean-up action that it will be closed(__exit__) eventually and within ‘with’ statement, it knows that it’s to be closed at the exit of ‘with’ block.

For more info on “with” statement, check Hitchhiker’s guide to Python and this blog.

Hei hei 😀

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: