Processing Large Jobs | Drifting Ruby

Processing Large Jobs

Episode #468 by

David Kimura

Aug 4, 2024

7

Previous (#467) Episode Next (#469)

Summary

In this episode, we will upload a CSV file but need to pass it into a background job. This can prove to be difficult based on the hosting infrastructure so we'll explore some mechanisms to work around them. We'll also look to optimize Solid Queue to handle the large number of jobs.
rails solid queue background processing active storage 20:55

Mark as Watched Watch Later

Chapters

Introduction (0:00)
Generating Models (1:30)
Setting up the routes (2:53)
Setting up the imports controller (4:24)
Setting up the create action for importing (6:39)
Setting up the background jobs (9:14)
Demo (15:58)
Adding SolidQueue (16:48)
Demo again with SolidQueue (18:13)
Displaying the attachments (19:33)
Final thoughts (20:31)

Resources

Source - https://github.com/driftingruby/468-processing-large-jobs

This episode is sponsored by Honeybadger

Download Source Code

Summary

# Terminal
rails active_storage:install
rails g scaffold countries name flag:attachment
rails g controller countries/imports
rails g job countries/import
rails g job Countries::CreateRecord
bundle add solid_queue
rails g solid_queue:install
bundle exec rake solid_queue:start # should be added to the Procfile.dev

# Gemfile
gem "image_processing", "~> 1.2"
gem "solid_queue", "~> 0.3.4"

# config/routes.rb
resources :countries
namespace :countries do
  resource :imports, only: [:new, :create]
end

# app/controllers/countries/imports_controller.rb
class Countries::ImportsController < ApplicationController
  def new
  end

  def create
    file = params[:file]

    blob = ActiveStorage::Blob.create_and_upload!(
      io: file,
      filename: file.original_filename,
      content_type: file.content_type
    )
    Countries::ImportJob.perform_later(blob.signed_id)
    redirect_to countries_path, notice: "Countries are being imported..."
  end
end

# app/views/countries/imports/new.html.erb
<%= form_with url: countries_imports_path do |f| %>
  <div class="mb-3">
    <%= f.label :file, class: "form-label" %>
    <%= f.file_field :file, class: "form-control" %>
  </div>

  <%= f.submit "Import", class: "btn btn-primary" %>
<% end %>

# app/views/welcome/index.html.erb
<%= link_to "Import Countries", new_countries_imports_path %>

# app/jobs/countries/import_job.rb
require "csv"

class Countries::ImportJob < ApplicationJob
  queue_as :default

  def perform(blob_signed_id)
    blob = ActiveStorage::Blob.find_signed(blob_signed_id)

    blob.download do |data|
      encoded_data = data.force_encoding("ASCII-8BIT").encode("UTF-8", invalid: :replace, undef: :replace, replace: "?")
      CSV.parse(encoded_data, headers: true) do |row|
        Countries::CreateRecordJob.perform_later(
          name: row["Country"],
          flag_url: row["Flag URL"]
        )
      end
    end
  ensure
    blob.purge_later
  end
end

# app/jobs/countries/create_record_job.rb
require "open-uri"

class Countries::CreateRecordJob < ApplicationJob
  queue_as :default

  def perform(name:, flag_url:)
    country = Country.find_or_initialize_by(name: name)
    country.flag.attach(
      io: URI.open(flag_url),
      filename: File.basename(URI.parse(flag_url).path)
    )
    country.save!
  end
end

# config/application.rb
config.active_job.queue_adapter = :solid_queue

# config/solid_queue.yml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "*"
      threads: 1
      processes: 1
      polling_interval: 0.1

development:
 <<: *default

test:
 <<: *default

production:
 <<: *default

# app/views/countries/index.html.erb
<%= image_tag country.flag.variant(resize_to_limit: [30, 30]) if country.flag.attached? && country.flag.image? %>