Improving open-uri

When working on the Shrine library for handling file uploads, in multiple places I needed to be able to download a file from URL. If you know the Ruby standard library well, the solution might be obvious to you: open-uri.

require "open-uri"
result = open("http://example.com/image.jpg")
result #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20160524-10403-xpdakz>

Open-uri is something that I indeed very much wanted to use for my use case. It ships with Ruby, so there are no external dependencies (just Net::HTTP), and it has many benefits:

  • downloads to a unique filesystem location (using Tempfile)
  • supports HTTP/HTTPS/FTP links
  • follows redirects
  • memory efficient
  • easy basic authentication
  • easy proxy

However, also considering that in my case the URL could come from user input, open-uri turned out to have many limiations and quirks:

  • Using Kernel#open makes you vulnerable to remote code execution
  • If the remote file is smaller than 10KB, open-uri actually returns a StringIO instead of a Tempfile
  • URL’s file extension isn’t preserved in downloaded Tempfile
  • You cannot limit maximum number of redirects
  • You cannot limit maximum filesize

I’ve thought about alternatives: rest-client, curl or wget. However, rest-client was a too heavy dependency just for downloading, and I didn’t want to depend on external CLI tools. Also, none of them were able to properly limit the maximum filesize, which I found important in context of Shrine.

So, realizing that I still wanted to use open-uri, I decided to make a wrapper around it that addresses these limitations. I want to guide you through my journey, fixing one issue at a time.

Improvements

Kernel#open

Ruby has a Kernel#open method, which given a file path acts as File.open. but given a string that starts with “|”, it interprets it as a shell command and returns an IO connected to the spawned subprocess:

open("| ls") # returns an IO connected to the `ls` shell command

Open-uri extends Kernel#open with the ability to accept URLs. However, if the URL is coming from user input, we should never pass it to Kernel#open, because different users have different ideas on what is a “URL”; someone might think that | rm -rf ~ is a nice looking URL.

A little known fact is that Kernel#open just delegates to URI::(HTTP|HTTPS|FTP)#open, and we can simply use that instead:

uri = URI.parse("http://example.com/image.jpg") #=> #<URI::HTTP>
uri.open #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20160524-10403-xpdakz>

StringIO

Stangely, if the remote file has less than 10KB, open-uri will actually return a StringIO instead of a Tempfile.

uri.open #=> #<StringIO>

In context of Shrine I wanted the returned IO to always be a file, for consistency and because it could later be given for processing. We can easily fix that:

io = uri.open

if io.is_a?(StringIO)
  downloaded = Tempfile.new
  File.write(downloaded.path, io.string)
else
  downloaded = io
end

downloaded # now always a Tempfile

File extension

Surprisingly, open-uri always creates a Tempfile without a file extension, even if the url has one. In Shrine I wanted that downloaded files (which will later be uploaded) always have an extension if it’s known.

So let’s copy the downloaded IO to a new Tempfile which has a file extension, but use mv if we can so that we don’t pay any performance penalty (and that the old file also gets deleted):

io = uri.open
downloaded = Tempfile.new([File.basename(uri.path), File.extname(uri.path)])

if io.is_a?(Tempfile)
  FileUtils.mv io.path, downloaded.path
else # StringIO
  File.write(downloaded.path, io.string)
end

File.extname(downloaded.path) #=> ".jpg"

Redirects

What’s good is that open-uri can automatically follow redirects. What’s bad is that we cannot limit the maximum number of redirects. This allows the attacker to give a URL which causes a redirect loop, and open-uri would continue making requests forever. To be fair, open-uri has a detection for redirect loops, but only if URLs repeat.

So we disable open-uri’s following of redirects, which now raises OpenURI::HTTPRedirect on redirects, allowing us to reimplement it:

tries = 3

begin
  uri.open(redirect: false)
rescue OpenURI::HTTPRedirect => redirect
  uri = redirect.uri # assigned from the "Location" response header
  retry if (tries -= 1) > 0
  raise
end

Maximum filesize

Since the URL can sometimes come from the user input, I wanted to give Shrine users the ability to limit maximum filesize of the remote file. Specifically, I wanted that download aborts as soon as the “Content-Length” header reveals that the file will be too large. Luckily, open-uri has the :content_length_proc option, which calls the given proc as soon as open-uri reads “Content-Length”:

uri.open(
  content_length_proc: ->(size) { raise FileTooLarge if size > max_size },
)

However, an attacker could theoretically create an app which returns large files, but where the “Content-Length” response header is ommited on purpose. Luckily, open-uri has got our back on this one too with :progress_proc, which calls the given proc whenever a chunk is downloaded, with the current size. That means we can add it as a fallback in case “Content-Length” is missing:

uri.open(
  content_length_proc: ->(size) { raise FileTooLarge if size && size > max_size },
  progress_proc:       ->(size) { raise FileTooLarge if size > max_size },
)

User agent

It turns out that when we’re making requests to an application, but we don’t include a “User-Agent” header, most applications will start rejecting our requests after some time.

Open-uri doesn’t include a “User-Agent” by default, but allows us to easily add one, since open-uri treats any unknown option as a request header:

uri.open("User-Agent" => "MyApp/1.0")

Result

The result of this investigation is the Down gem, which incorporates all of these improvements, and more. You can use it like this:

require "down"
result = Down.download("http://example.com/image.jpg")
result #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20160524-10403-xpdakz.jpg>

More advanced downloading could look something like this:

Down.download "http://example.com/image.jpg",
  max_size: 20*1024*1024,   # 20 MB
  max_redirects: 5,         # default is 2
  proxy: "http://proxy.com" # delegates to open-uri

Conclusion

I like that I was able to make a lightweight wrapper around open-uri, which already had most of the features that I wanted, but allowed me to complete the ones that I was missing. If you want to use open-uri, but without any of the mentioned quirks, consider using Down.

Janko Marohnić

Janko Marohnić

A passionate Ruby backend developer who fell in love with Roda & Sequel, and told Rails “it’s not me, it’s you”. He enjoys working with JSON APIs and SQL databases, while prioritizing testing, and always tries to find the best library for the job. Creator of Shrine and test.vim.

comments powered by Disqus