Update 'README.md'
parent
c4b78d2743
commit
84125b6972
@ -1,523 +1,7 @@
|
||||
# Docker for local development
|
||||
# README
|
||||
|
||||
With a move to multiple services or even, with a monolith, multiple services as dependencies (rabbitmq, redis and postgreSQL for example) the multiplication of services to have running in a development environment tends to make things complicated.
|
||||
This repository is showing an example of how to handle service dependencies with Docker and `docker-compose`.
|
||||
|
||||
Multiple issues tend to appear as both the number of those services and of team members grow :
|
||||
- different people have different versions of those services
|
||||
- different people have different way to start them
|
||||
- onboarding a new team member starts to become a long list of patches and conditions
|
||||
The stacks of the services used as example is Ruby, RubyOnRails and Sinatra but the main idea is about the `docker-compose.yml` and `Dockerfile` files so you should be able to figure an equivalent for any other stack.
|
||||
|
||||
One solution is to rely on docker and docker-compose for each bounded context (service). This article will cover this approach in the case of two Ruby applications : one being the main application, the other being a library for the first one that can provide information critical for the first application work.
|
||||
We will be able to see how we can work with docker and docker-compose for one service, but also in the case of a multiple services fleet.
|
||||
|
||||
> Docker is a virtualisation solution that allows you to build images and run them as "containers" on your machine. It has a low overhead compared to virtual machines solutions. Public images for many services are available from the [Docker Hub](https://hub.docker.com) and other public repositories.
|
||||
|
||||
> Docker-compose is a command building upon Docker that allows you to manage a fleet of containers based on a configuration file. That file contains a list of services and options to start them including either the image name or the path towards a Dockerfile that allows the building of the container image.
|
||||
|
||||
## A simple Rails application
|
||||
|
||||
We will first get ourselves ready with a first RubyOnRails application. It will rely on postgreSQL for the main database and Redis for jobs.
|
||||
|
||||
> We consider you already have the rails gem installed on your system, preferably with a version manager such as [asdf](https://asdf-vm.com/).
|
||||
|
||||
```
|
||||
$> rails new backend -T -d postgresql
|
||||
# -T : skip the setup of a testing framework
|
||||
# -d : set the database to ...
|
||||
```
|
||||
|
||||
Once that is done we add rspec by adding the following line in the `Gemfile` and its `group :development, :test` block :
|
||||
|
||||
```
|
||||
gem 'rspec-rails'
|
||||
```
|
||||
|
||||
This can be followed by
|
||||
|
||||
```
|
||||
$> rails generate rspec:install
|
||||
```
|
||||
|
||||
That command will create a spec folder and add the spec helpers we need.
|
||||
|
||||
### docker-compose for postgresql
|
||||
|
||||
We now want to configure the database instance for our local environment, it's time to get our first `docker-compose.yml` file.
|
||||
|
||||
```
|
||||
version: "3.7"
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:13.2
|
||||
ports:
|
||||
- "5432:5432"
|
||||
environment:
|
||||
- POSTGRES_PASSWORD=postgres
|
||||
- POSTGRES_USER=postgres
|
||||
volumes:
|
||||
- data-postgres:/var/lib/postgresql/data
|
||||
volumes:
|
||||
data-postgres:
|
||||
driver: local
|
||||
```
|
||||
|
||||
We have a few important points here, first within the `postgres` section :
|
||||
- this section define the `postgres` service, it could be named `database` or `flux-capacitor`, it doesn't matter
|
||||
- `image` : the name and tag of the image we want to use
|
||||
- `ports` : the mapping of ports between the host and the container's environment. It doesn't have to be the same on each side of the ':'. Here the right side is the port used by postgresql by default and within the container. The left side is the one exposed on the host.
|
||||
- `environment` : the list of environment variables that will be defined in the container when it's started
|
||||
- `volumes` : a list of data volumes that will be mounted within the container when it starts. This volume won't be destroyed between runs of the container.
|
||||
|
||||
And then we have a `volumes` section at the end of the file defining how the volume we use in the `postgres` service is handled. Here we don't define a specific path, docker will figure out a path on its own.
|
||||
|
||||
|
||||
We can then start the container and check if it's running :
|
||||
```
|
||||
$> docker compose up -d
|
||||
Creating network "backend_default" with the default driver
|
||||
Creating volume "backend_data-postgres" with local driver
|
||||
Pulling postgres (postgres:13.2)...
|
||||
13.2: Pulling from library/postgres
|
||||
fcad0c936ea5: Pull complete
|
||||
...
|
||||
c7c8064b7a1a: Pull complete
|
||||
Digest: sha256:0eee5caa50478ef50b89062903a5b901eb818dfd577d2be6800a4735af75e53f
|
||||
Status: Downloaded newer image for postgres:13.2
|
||||
$> docker compose ps
|
||||
NAME SERVICE STATUS PORTS
|
||||
backend_postgres_1 postgres running 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp
|
||||
```
|
||||
|
||||
We can see the container is started, its name is prefixed with "backend" which is the name of my current folder. The "PORTS" column tells us which ports are exposed on the container. We can see that the port 5432 of the host machine is linked to the port 5432 of the container.
|
||||
|
||||
We need to adjust our database configuration in the rails application :
|
||||
|
||||
```
|
||||
# config/database.yml
|
||||
default: &default
|
||||
adapter: postgresql
|
||||
encoding: unicode
|
||||
# For details on connection pooling, see Rails configuration guide
|
||||
# https://guides.rubyonrails.org/configuring.html#database-pooling
|
||||
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
|
||||
host: 0.0.0.0
|
||||
port: 5432
|
||||
username: postgres
|
||||
password: postgres
|
||||
```
|
||||
|
||||
No need to touch the development or test sections; we can update the default one. The production settings should be given through an environment variable so we don't have to worry about that part here.
|
||||
All those values (host, port, username, password) are from the `docker-compose.yml` file and what we know of our workstation.
|
||||
|
||||
### database use
|
||||
|
||||
We can now use that database and proceed with ensuring the application is properly linked up by creating the development and test databases.
|
||||
|
||||
```
|
||||
$> bundle exec rails db:create
|
||||
Created database 'backend_development'
|
||||
Created database 'backend_test'
|
||||
$>
|
||||
```
|
||||
|
||||
That looks good.
|
||||
|
||||
### a first conclusion
|
||||
|
||||
This is basically how things can be done to handle those service dependencies. We can expand on this with more services such as redis or elastic search or whatever has a docker image or a way to build one.
|
||||
|
||||
Now, when a team member sets up a new computer there is no need to say "oh you need to install PostgreSQL 13.2" or have such a section in your onboarding documentation or script.
|
||||
The team member can clone the repository, and within that clone run `docker-compose up -d`. This will download and start the right version with the right configuration of PostgreSQL.
|
||||
|
||||
And it will be the same with any other service we want to have our application depend on.
|
||||
|
||||
## Beyond the simple Rails app
|
||||
|
||||
### Scaffolding a model and controller
|
||||
|
||||
Let's now say that we have a model and a controller in our application.
|
||||
|
||||
We can add a simple Book model and the controller to handle it with the scaffold generator :
|
||||
|
||||
```
|
||||
$> bundle exec rails g scaffold Book title:string author:string description:string
|
||||
```
|
||||
|
||||
This is rather lazy but will allow us to actually focus on what we want to do : working with services.
|
||||
|
||||
> At this step, as we want to start the rails app through a web server we will need to run `rails webpacker:install`.
|
||||
|
||||
We can start the application with `rails s`, and we can use our web browser to open up `http://localhost:3000/books`.
|
||||
|
||||
We can play around and create and delete books, see their list etc ...
|
||||
|
||||
### Background jobs
|
||||
|
||||
We want to make things a bit more complex now by using background jobs. One of the popular ways to do this within Rails applications is to rely on [Sidekiq](https://sidekiq.org/).
|
||||
|
||||
We can add the `sidekiq` gem to the Gemfile :
|
||||
|
||||
```
|
||||
gem 'sidekiq'
|
||||
```
|
||||
|
||||
And run `bundle install` to install the gem and its dependencies.
|
||||
|
||||
We now require an instance of Redis to run locally. Let's update our `docker-compose.yml` file for this.
|
||||
|
||||
```
|
||||
redis:
|
||||
image: redis:6.2.5
|
||||
ports:
|
||||
- "6379:6379"
|
||||
```
|
||||
|
||||
This entry is pretty small, but that's enough.
|
||||
|
||||
As we are matching the default configuration (port and so on), sidekiq should connect to it without issue, no need to worry about configuration for this example.
|
||||
|
||||
We can ensure our app has a sidekiq web interface by following https://github.com/mperham/sidekiq/wiki/Monitoring#web-ui .
|
||||
|
||||
Open up two other terminals and go to the application folder to run `rails s` in one and `sidekiq` in the other. The first one will start the application (skip that one if you already have it running in another terminal, but you will need to restart it). The other one will start a sidekiq worker.
|
||||
|
||||
Now if you go to `http://localhost:3000/sidekiq/` you will see the dashboard of the sidekiq jobs. The `http://localhost:3000/sidekiq/busy` will list the jobs and runner currently running. If you stop the runner by using `Ctrl c` in the terminal running the runner and reload that sidekiq page you will see it disappear.
|
||||
|
||||
Let's now add a simple worker and trigger it when a book is added.
|
||||
|
||||
We add a file in `app/workers/notify_worker.rb` :
|
||||
|
||||
```
|
||||
class NotifyWorker
|
||||
include Sidekiq::Worker
|
||||
|
||||
def perform(book_name, author_name)
|
||||
Rails.logger.info "#{book_name} by #{author_name} was added."
|
||||
end
|
||||
end
|
||||
|
||||
```
|
||||
|
||||
This is pretty useless but we still want it done.
|
||||
|
||||
Let's add how to trigger that job in the controller create action upon success.
|
||||
|
||||
```
|
||||
def create
|
||||
@book = Book.new(book_params)
|
||||
|
||||
respond_to do |format|
|
||||
if @book.save
|
||||
NotifyWorker.perform_async(book_params[:title], book_params[:author])
|
||||
format.html { redirect_to @book, notice: "Book was successfully created." }
|
||||
format.json { render :show, status: :created, location: @book }
|
||||
else
|
||||
format.html { render :new, status: :unprocessable_entity }
|
||||
format.json { render json: @book.errors, status: :unprocessable_entity }
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
Note the line that has been added :
|
||||
```
|
||||
NotifyWorker.perform_async(book_params[:title], book_params[:author])
|
||||
```
|
||||
|
||||
This will queue the job and the worker will pick it up.
|
||||
|
||||
### Conclusion
|
||||
|
||||
This was a bit of extra fluff but it allows us to see a few bits that are quite interesting here.
|
||||
|
||||
First we see how to add yet another service dependency to our fleet in the `docker-compose.yml`. Then we see that just like the rails app we start the worker by hand.
|
||||
|
||||
The reason why we don't use the docker-compose file to start those two services is that those are directly depending on the code we are bound to modify and work on. Thus, it's more practical to start and run them by hand. We will now see how that's not practical for developers working on another service relying on this one.
|
||||
|
||||
## A second ruby app
|
||||
|
||||
We now want to see how to work with another Ruby application, and from the point of view of a separate team.
|
||||
|
||||
This service will provide a book information database allowing users to find a book title, author name and description from a ISBN identifier.
|
||||
|
||||
We create a simple Sinatra app for this case.
|
||||
|
||||
|
||||
### A crude application
|
||||
|
||||
We mainly want to mock such a service as isbn databases might be a bit ... big.
|
||||
|
||||
So we use a very simple sinatra application :
|
||||
|
||||
```
|
||||
# app.rb
|
||||
require 'sinatra'
|
||||
require 'json'
|
||||
|
||||
ISBNS = {
|
||||
'2-7654-1005-4': {
|
||||
title: 'Lucy',
|
||||
author: 'Renard',
|
||||
description: 'A book about stars.'
|
||||
},
|
||||
'2-7754-1105-4': {
|
||||
title: 'Mountains',
|
||||
author: 'John Smith',
|
||||
description: 'A book about plains.'
|
||||
},
|
||||
}
|
||||
|
||||
get '/' do
|
||||
'Hello world!'
|
||||
end
|
||||
|
||||
get '/isbn/:isbn' do
|
||||
isbn = params[:isbn].to_sym
|
||||
|
||||
if ISBNS.keys.include?(isbn)
|
||||
response = ISBNS[isbn].to_json
|
||||
status 200
|
||||
body response
|
||||
else
|
||||
response = { error: 'Not found' }.to_json
|
||||
status 404
|
||||
body response
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The rest of the code is in the example repository.
|
||||
|
||||
This will allow us to make a simple request to the service to get the details of a book based on its isbn identifier.
|
||||
|
||||
We can test it by visiting `http://0.0.0.0:3001/isbn/2-7654-1005-4` in a web browser.
|
||||
|
||||
### What about docker here ?
|
||||
|
||||
This application doesn't require a database as we are using a constant and very limited, mocked, data. But we do want to be able to run this service easily and send requests to it when we run the main backend.
|
||||
|
||||
We could simply start the application using the `bin/http` script provided. But that would require the team working on the `backend` service to know that. It's best if that dependency is started through docker-compose too.
|
||||
|
||||
So we need a Dockerfile to define how to build and start this service.
|
||||
|
||||
```
|
||||
from ruby:3.0.2
|
||||
|
||||
RUN mkdir /var/app
|
||||
|
||||
WORKDIR /var/app
|
||||
|
||||
COPY . .
|
||||
|
||||
RUN bundle install
|
||||
|
||||
CMD ["sh", "bin/http"]
|
||||
```
|
||||
|
||||
We can then build the image to try it out :
|
||||
|
||||
```
|
||||
$> cd isbn-search
|
||||
$> docker build -t isbn-search-test .
|
||||
...
|
||||
$> docker run -p "3001:3001" -ti isbn-search-test
|
||||
```
|
||||
|
||||
We can then visit `http://0.0.0.0:3001/isbn/2-7654-1005-4` again and see it works.
|
||||
|
||||
### And how does that help us ?
|
||||
|
||||
Let's go back to the backend now.
|
||||
|
||||
We will modify the form and controller so that we only enter an ISBN and we get all the book details from our isbn-search service.
|
||||
|
||||
To make it clean we will write a small api client library in lib/.
|
||||
|
||||
We add the `excon` gem to our Gemfile and install it. Then we add a line to our application.rb file in the config section to ensure the lib folder content is loaded.
|
||||
|
||||
```
|
||||
config.autoload_paths += %W(#{config.root}/lib)
|
||||
```
|
||||
|
||||
And then we can create the client file in lib/internal/isbn_search/client.rb :
|
||||
|
||||
```
|
||||
module Internal
|
||||
module IsbnSearch
|
||||
class Client
|
||||
def initialize
|
||||
@port = ENV.fetch('ISBN_SEARCH_PORT', 3001)
|
||||
@host = ENV.fetch('ISBN_SEARCH_HOST', '0.0.0.0')
|
||||
@path = 'isbn/'
|
||||
end
|
||||
|
||||
def get(isbn)
|
||||
response = Excon.get(url_for(isbn))
|
||||
if response.status == 200
|
||||
JSON.parse(response.body)
|
||||
else
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def url_for(isbn)
|
||||
"http://#{@host}:#{@port}/#{@path}/#{isbn}"
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
This is a very simple HTTP Api client. If there is a 200 response we will extract the body and parse it before returning it. Otherwise we will return `nil`.
|
||||
|
||||
We can now use this within the controller to get the details of the book, and we can update the form.
|
||||
|
||||
```
|
||||
<%= form_with(model: book) do |form| %>
|
||||
<% if book.errors.any? %>
|
||||
<div id="error_explanation">
|
||||
<h2><%= pluralize(book.errors.count, "error") %> prohibited this book from being saved:</h2>
|
||||
|
||||
<ul>
|
||||
<% book.errors.each do |error| %>
|
||||
<li><%= error.full_message %></li>
|
||||
<% end %>
|
||||
</ul>
|
||||
</div>
|
||||
<% end %>
|
||||
|
||||
<div class="field">
|
||||
<%= form.label :isbn %>
|
||||
<%= form.text_field :isbn %>
|
||||
</div>
|
||||
|
||||
<div class="actions">
|
||||
<%= form.submit %>
|
||||
</div>
|
||||
<% end %>
|
||||
```
|
||||
|
||||
And the controller needs to be tailored as well.
|
||||
|
||||
```
|
||||
class BooksController < ApplicationController
|
||||
before_action :set_book, only: %i[ show edit update destroy ]
|
||||
|
||||
# GET /books or /books.json
|
||||
def index
|
||||
@books = Book.all
|
||||
end
|
||||
|
||||
# GET /books/1 or /books/1.json
|
||||
def show
|
||||
end
|
||||
|
||||
# GET /books/new
|
||||
def new
|
||||
@book = Book.new
|
||||
end
|
||||
|
||||
# GET /books/1/edit
|
||||
def edit
|
||||
end
|
||||
|
||||
# POST /books or /books.json
|
||||
def create
|
||||
@book = Book.new(book_info)
|
||||
|
||||
respond_to do |format|
|
||||
if @book.save
|
||||
NotifyWorker.perform_async(book_info[:title], book_info[:author])
|
||||
format.html { redirect_to @book, notice: "Book was successfully created." }
|
||||
format.json { render :show, status: :created, location: @book }
|
||||
else
|
||||
format.html { render :new, status: :unprocessable_entity }
|
||||
format.json { render json: @book.errors, status: :unprocessable_entity }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# PATCH/PUT /books/1 or /books/1.json
|
||||
def update
|
||||
respond_to do |format|
|
||||
if @book.update(book_info)
|
||||
format.html { redirect_to @book, notice: "Book was successfully updated." }
|
||||
format.json { render :show, status: :ok, location: @book }
|
||||
else
|
||||
format.html { render :edit, status: :unprocessable_entity }
|
||||
format.json { render json: @book.errors, status: :unprocessable_entity }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# DELETE /books/1 or /books/1.json
|
||||
def destroy
|
||||
@book.destroy
|
||||
respond_to do |format|
|
||||
format.html { redirect_to books_url, notice: "Book was successfully destroyed." }
|
||||
format.json { head :no_content }
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
# Use callbacks to share common setup or constraints between actions.
|
||||
def set_book
|
||||
@book = Book.find(params[:id])
|
||||
end
|
||||
|
||||
def isbn_params
|
||||
params.require(:book).permit(:isbn)
|
||||
end
|
||||
|
||||
def book_info
|
||||
Internal::IsbnSearch::Client.new.get(isbn_params[:isbn])
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
We have mostly changed the `create` and `update` actions, replaced the `book_params` method by a `isbn_params` one and added the `book_info` method to do the call to the IsbnSearch service.
|
||||
|
||||
|
||||
### Working out with docker-compose
|
||||
|
||||
Now that our backend service relies on the IsbnSearch one we need a way to start that service easily.
|
||||
As we have a Dockerfile for the IsbnSearch service we can add the following section to the `docker-compose.yml` file.
|
||||
|
||||
```
|
||||
isbn_search:
|
||||
build: ../isbn-search/
|
||||
environment:
|
||||
- PORT=3001
|
||||
ports:
|
||||
- "3001:3001"
|
||||
```
|
||||
|
||||
And then build and start it :
|
||||
|
||||
```
|
||||
$> docker-compose up -d
|
||||
Docker Compose is now in the Docker CLI, try `docker compose up`
|
||||
|
||||
Building isbn_search
|
||||
[+] Building 2.1s (10/10) FINISHED
|
||||
=> [internal] load build definition from Dockerfile
|
||||
...
|
||||
backend_redis_1 is up-to-date
|
||||
backend_postgres_1 is up-to-date
|
||||
Creating backend_isbn_search_1 ... done
|
||||
$>
|
||||
```
|
||||
|
||||
Now we can try it out and head to `http://0.0.0.0:3000/books`, click on "New Book", type in "2-7654-1005-4", click on "Create Book" and it will create the book.
|
||||
|
||||
### Conclusion
|
||||
|
||||
We now have a setup that is a lot more similar to how things are used in a real day to day setting. We have one backend service relying on multiple services both for storage but also for additional needs. All those dependencies can be built and started repeatedly through one simple command.
|
||||
|
||||
## Epilogue
|
||||
|
||||
A similar approach could be done to allow the developers working on the `isbn_search` service to start the `backend` one as a docker container, including its storage dependencies so that they can work and test their changes against a running instance of it.
|
||||
|
||||
If you are interested to know how to do that we can organise a pairing session or a workshop.
|
||||
Contact us directly at contact@imfiny.com for details and prices.
|
||||
The article has been moved to [Imfiny's blog](https://blog.imfiny.com/docker-to-handle-dependencies/).
|
||||
|
Loading…
Reference in New Issue