PowerPoint Presentation

peruvianwageslaveInternet and Web Development

Feb 5, 2013 (4 years and 6 months ago)

161 views

Large Files on Rails



http
://extremia.fi/


http://extremia.net/

Processing Large Files with Paperclip


When
dealing with files
in
Rails,
it would be
better to use a
gem made by someone
else, for
example Paperclip. It’s
designed with only fairly
small files in
mind, so it
processes them
synchronously.


When using synchronous processing, it might
take several minutes to process large files. During
this time, the
Rails server instance
will
be blocked
and unable to
serve
any other clients.


A

gem called Delayed
Paperclip offers
a solution for
this using either
delayed_job

or
Resque

for
asynchronous
processing on top of
Paperclip.


http
://extremia.fi/


http://extremia.net/

Processing Large Files with Paperclip


In some cases
the gems
just won’t
cut the file.


For example, when
trying to scale your service to meet
an increased demand for file processing power by
deploying more servers.


You
have to think about files waiting for processing on
different hard drives and whether Delayed Paperclip
supports using multiple custom queues with Delayed Job
or
Resque
.


You
may want to let the user ‘upload’ a file from
somewhere other than their computer.


Y
ou may want
to keep these services completely separate
from the rest of your web
application, when
doing long
running processing jobs like with video
clips.


http
://extremia.fi/


http://extremia.net/

Direct
U
ploads


There may be also
problems with direct
uploads (via POST) in Rails applications that
become an issue especially with large files.


Since
MRI is incapable of utilizing more than
one core due to the global interpreter lock,
Rails applications are most commonly hosted
with multiple processes behind a reverse
proxy using Apache, for example.


http
://extremia.fi/


http://extremia.net/

Direct
Uploads


The application itself is hosted by a Ruby application
server (Thin, Mongrel, etc.), which is the outermost
layer of a Ruby process running in the reverse proxy
setup.


Behind that sits Rack, which is the connecting layer
between the Ruby application server and your code
inside the Rails web application framework.


If a request body (file content) is too large, it cannot all
be kept in memory and must be buffered to disk.


Depending on your choice of server software, this could be
done by both your web server and your ruby application
server.


http
://extremia.fi/


http://extremia.net/

Decoding

Large

Files


Decoding a very large request body in a Ruby
script
is
very
slow. Workarounds in the form of a fast native
module that decodes and buffers the files to disk are
available for some web servers like Apache.


Apart
from a possible out
-
of
-
memory and multiple
-
slow
-
disk
-
write problem, the request body is encoded
in multipart/form
-
data format and needs to be
decoded (by Rack
).


This is only for the request part and purely client
-
side.
The backend progress monitoring and communication
with client during file processing is entirely another
matter.



http
://extremia.fi/


http://extremia.net/