Build a simple REST API with Elixir

In the previous, we created all of our required endpoints, understood how to work with MongoDB and did some manual testing. If you reached this post directly, you might want to read the last post, as this is a continuation of that post.

Now that we have something working, it's the right time to add some tests and introduce some security to our endpoints. Let's head over to test/rest_api_test.exs and get started.

GitHub - saswatds/elixir-rest-api at part-4

A simple REST API server is Elixir. Contribute to saswatds/elixir-rest-api development by creating an account on GitHub.

GitHubsaswatds

Let's get started

Testing your code is simply asserting what you expect from the program and what it does. This simple idea is what software testing is all about. As software complexity increases, the number and type of assertions increase considerably and making sense of all the tests becomes pretty hard. Therefore, we build some software testing frameworks that help us organize and manage our test cases. I have been using the words assertions and test-cases inter-changeably, but before start writing tests, let's get out lingo right.

Assertion - It is the most atomic check which verifies if the expected value is the same as the actual value.
Test Case - A test case is an assumption of some behaviour we expect the program to exhibit. The test cases would contain one or more assertions to prove that the assumption holds. For example, a test case could be GET / should respond with 200 OK and the assertion could be conn.status == 200
Test Suite - A suite is an organizational strategy used to co-locate together. Let's say you were testing all GET request, you could create a suite called GET requests which would contain all test cases that test get endpoints.

If you are coming from a Javascript or Typescript world then you might be familiar with frameworks like Jest and Mocha. That provides a bunch of helper functions and scaffolding to manage your tests. In the elixir world, the equivalent framework is called ExUnit.

ExUnit — ExUnit v1.14.1

You have already been introduced to this in Part 1 of this course, but here is a refresher.

defmodule RestApiTest.Router do
  # Bringing ExUnit's case module to scope and configure it to run
  # tests in this module concurrently with tests in other modules
  # https://hexdocs.pm/ex_unit/ExUnit.Case.html
  use ExUnit.Case, async: true
  
  # This makes the conn object avaiable in the scope of the tests,
  # which can be used to make the HTTP request
  # https://hexdocs.pm/plug/Plug.Test.html
  use Plug.Test
  
  # We call the Plug init/1 function with the options then store
  # returned options in a Module attribute opts.
  # Note: @ is module attribute unary operator
  # https://hexdocs.pm/elixir/main/Kernel.html#@/1
  # https://hexdocs.pm/plug/Plug.html#c:init/1
  @opts RestApi.Router.init([])
 
 
  # Create a test with the name "return ok"
  test "return ok" do
  	# Build a connection which is GET request on / url
    conn = conn(:get, "/")

    # Then call Plug.call/2 with the connection and options
    # https://hexdocs.pm/plug/Plug.html#c:call/2
    conn = RestApi.Router.call(conn, @opts)

    # Finally we are using the assert/2 function to check for the
    # correctness of the response
    # https://hexdocs.pm/ex_unit/ExUnit.Assertions.html#assert/2
    assert conn.state == :sent
    assert conn.status == 200
    assert conn.resp_body == "OK"
  end
end

test/rest_api_test.exs

Now you should be able to make the right connections in your head:

Assertions are performed by the assert functions provided by ExUnit.Assertions module.
Test Cases and Test Suites are defined by the test and describe macro respectively, provided by ExUnit.Case module.

🧪

Writing test cases is as much art as it is a science and beyond our scope here. Learning how to write good tests will be an entire course in itself, so if you want me to build that course, leave me a comment below. Meanwhile, browse around the internet for any good resources.

Working with database

The database is one of the most crucial dependencies and it is kind of a folklore among developers and testers who believe we should mock the dependencies and write unit tests. I feel that writing unit tests for application servers waste a bunch of time, but if we write tests which call the API and observe the database, we can cover more realistic situations with high confidence.

If we are not mocking out the database, then every test case could inadvertently affect other test cases. To prevent this from happening, we will do two simple things

Prevent running test cases in parallel
Clean up our database after every test

The known downside of this approach is that the tests would run slow, but hey, our tests are super reliable.

defmodule RestApiTest.Router do
  use ExUnit.Case  # removed async to prevent running cases in parallel
  use Plug.Test

  @opts RestApi.Router.init([])
    
  test "GET / should return ok" do ... end

  describe "Posts" do
      # The setup callback is called before each test executes and the on_exit after each test is complete
      # We will use this hook to list all the mongo db collections and for each of
      # the collection to clear out the entire collection. This way for every test
      # case we will start from a clean slate
      setup do
        on_exit fn ->
          Mongo.show_collections(:mongo)
          |> Enum.each(
    		fn col -> Mongo.delete_many!(:mongo, col, %{}) 
    	end)
      end
  end
end

test/rest_api_test.exs - insert lines in file

👨🏻‍🔬

Assignment 1: What happens if the tests start with data already present in the DB? The very first test could potentially fail, making our tests flaky!
Find out which other ExUnit callback you could use to clean the DB once before any test case is executed and implement the same.

Automate Testing

Creating a Post

Let's call our endpoint through the test and create a Post and then check if the response we obtain is correct and the database is updated accordingly

defmodule RestApiTest.Router do
  ...

  describe "Posts" do
	...
    
    setup do ... end
    
    test "POST /post should return ok" do
      # Assert that there are no elements in the db
      assert Mongo.find(:mongo, "Posts", %{}) |> Enum.count == 0

      # Make an API to call create a post
      conn = conn(:post, "/post", %{name: "Post 1", content: "Content of post"})
      conn = RestApi.Router.call(conn, @opts)

      # Checking that request was actually sent
      assert conn.state == :sent

      # Checking that response code was 200
      assert conn.status == 200

      # Asserting that response body was what we expected
      # Note: We are using pattern matching here to perform the assertion
      # The id is autogenerated by mongodb so we have not way of predicting it
      # therefore we just use _ to match to anything but expect it to exist
      assert %{
        "id" => _,
        "content" => "Content of post",
        "name" => "Post 1"
      } = Jason.decode!(conn.resp_body)

      # Assert that there is something in the db
      assert Mongo.find(:mongo, "Posts", %{}) |> Enum.count == 1
    end
  end
end

test/rest_api_test.exs - add lines to file

👨🏻‍🔬

Assignment 2: The last assertion we made here is very naive as it just checks if the post count has increased to one. Ideally, it should also check that the data stored in DB is correct. Implement your own logic to prevent MongoDB store the correct values.

Fetching all Posts

Next, we test our GET all-post endpoint to see if it returns all the posts. Here's the catch: our database will be empty but we need some posts to exist for us to be able to make this test. In the previous test, we used the API to create the post and you might be tempted to use the same code to create two posts, but I strongly recommend we use the DB to directly create the post instead of calling the API. This way if there was ever a bug in the post-creating endpoint it would not cause the get endpoint to fail.

To help us create two posts we define a function createPosts which contains our logic to create two posts and it also returns the ids of the created posts as a list of strings. The rest of the code is pretty self-explanatory.

defmodule RestApiTest.Router do
  ...

  describe "Posts" do
	...
    
    test "POST /post should return ok" do ... end
    
    # A simple helper function that will help up quickly create
    # two posts in the databse
    def createPosts() do
      result = Mongo.insert_many!(:mongo, "Posts",[
        %{name: "Post 1", content: "Content 1"},
        %{name: "Post 2", content: "Content 2"},
      ])

     # The ids are BSON ObjectId which we are encoding to string for easier consumption
      result.inserted_ids |> Enum.map(fn id -> BSON.ObjectId.encode!(id) end)
    end

    test "GET /posts should fetch all the posts" do
      createPosts()

      conn = conn(:get, "/posts")
      conn = RestApi.Router.call(conn, @opts)

      assert conn.state == :sent
      assert conn.status == 200

      resp = Jason.decode!(conn.resp_body);

      assert Enum.count(resp) == 2
      assert %{
        "id" => _,
        "content" => "Content 1",
        "name" => "Post 1"
      } = Enum.at(resp, 0)
      assert %{
        "id" => _,
        "content" => "Content 2",
        "name" => "Post 2"
      } = Enum.at(resp, 1)

    end
  end
end

test/rest_api_test.exs - add lines to file

Fetching a single post

defmodule RestApiTest.Router do
  ...

  describe "Posts" do
	...
    
    test "GET /posts should fetch all the posts" do ... end
    
    test "GET /post/:id should fetch a single post" do
      [id | _] = createPosts() # using pattern matching to get the first id

      conn = conn(:get, "/post/#{id}") # string interpolation
      conn = RestApi.Router.call(conn, @opts)

      assert conn.state == :sent
      assert conn.status == 200

      assert %{
        "id" => _,
        "content" => "Content 1",
        "name" => "Post 1"
      } = Jason.decode!(conn.resp_body)
    end
  end
end

test/rest_api_test.exs - add lines to file

Updating a post

defmodule RestApiTest.Router do
  ...

  describe "Posts" do
	...
    
    test "GET /post/:id should fetch a single post" do ... end
    
    test "PUT /post/:id should update a post" do
      [id | _] = createPosts()

      conn = conn(:put, "/post/#{id}", %{content: "Content 3"})
      conn = RestApi.Router.call(conn, @opts)

      assert conn.state == :sent
      assert conn.status == 200

      assert %{
        "id" => _,
        "content" => "Content 3",
        "name" => "Post 1"
      } = Jason.decode!(conn.resp_body)
    end
  end
end

test/rest_api_test.exs - add lines to file

Deleting a post

defmodule RestApiTest.Router do
  ...

  describe "Posts" do
	...
    
    test "PUT /post/:id should update a post" do ... end
    
    test "DELETE /post/:id should delete a post" do
      [id | _] = createPosts()

      assert Mongo.find(:mongo, "Posts", %{}) |> Enum.count == 2

      conn = conn(:delete, "/post/#{id}", %{content: "Content 3"})
      conn = RestApi.Router.call(conn, @opts)

      assert conn.state == :sent
      assert conn.status == 200

      assert Mongo.find(:mongo, "Posts", %{}) |> Enum.count == 1
    end
  end
end

test/rest_api_test.exs - add lines to file

Basic Authentication

Our server is running, it's interacting well with the database and has written some automation tests to prove the correctness of the application. In the next part, we want to explore how we can deploy our application and serve some real traffic. Before we do that we need some way to prevent nefarious actors from accessing our endpoints and causing havoc. For this purpose, in the short term, we can utilize the Basic Authentication Scheme and restrict access to all our endpoints with a username and password.

Basic auth is not the best way to secure servers. It is easily susceptible to timing attacks and other vulnerabilities. In future posts, we will explore and implement other authentication and authorization strategies like OAuth2, etc, but for now, this should suffice.

The basic auth scheme is very simple and all we need to do is set the Authorization header. The values start with the word Basic then whitespace followed by Base64 encoding of the username and password joined by :.

🧠

If you are thinking what happens if the username contains the character :? Excellent Question! You have now started to thinking like a good developer. Reading RFC-7617 will give you the answer and explain in details the whole scheme.

Oh, now that's easy. I should implement this on my own. Now hold one and remember this golden rule. Never start by implementing your own version of a security library. Always find one which is open-source and maintained by the community. I don't want to discourage you to build it yourself, but take some time to read the implementation first to understand all the security aspects that need to be taken off.

Plug.BasicAuth

In our case, we will use the Plug.BasicAuth module that comes as part of Plug. First order of business, let's add these little notorious usernames and passwords somewhere. We could have them in the environment variables, but also the config should also be fine.

📣

Please replace <username> and <password> with the strings that you decide as your username and password.

import Config

config :rest_api, port: 8080
config :rest_api, database: "rest_api_db"
config :rest_api, pool_size: 3
config :rest_api, :basic_auth, username: "<username>", password: "<passwors>"

config/dev.env.exs - add lines to file

import Config

config :rest_api, port: 80
config :rest_api, database: "rest_api_db"
config :rest_api, pool_size: 3
config :rest_api, :basic_auth, username: "<username>", password: "<passwors>"

config/prod.env.exs - add lines to file

import Config

config :rest_api, port: 8081
config :rest_api, database: "rest_api_db"
config :rest_api, pool_size: 3
config :rest_api, :basic_auth, username: "<username>", password: "<passwors>"

config/test.env.exs - add lines to file

The next step is to configure the application to use the basic auth module for our router.

defmodule RestApi.Router do
  alias RestApi.JSONUtils, as: JSON

  use Plug.Router
  import Plug.BasicAuth

  plug(Plug.Logger)
  plug(:match)
  plug(:basic_auth, Application.get_env(:rest_api, :basic_auth))

  plug(Plug.Parsers,
    parsers: [:json],
    pass: ["application/json"],
    json_decoder: Jason
  )
 ...
 end

lib/rest_api/router.ex - add lines to file

There is a curious observation to be made here. We are used the use keyword to get the Plug.Router but when using Plug.BasicAuth do it using import?

There is very good official documentation about this here: alias, require, and import. But the TLDR; is

use <module> - This brings in all the functions from the module to the current module and after that, it calls the __using__ macro on the module. If you are coming from an object-oriented programming world you can imagine it to be an inheritance, or using the extends keyword.
import <module> - This just brings in functions to be used in the current module. There is no additional code executed in the parents' module lifecycle.

Then add the :basic_auth plug between the :match and :dispatch pipeline.

Tests

Now if you run all your tests, they should error out something like this.

  1) test GET / should return ok (RestApiTest.Router)
     test/rest_api_test.exs:8
     Assertion with == failed
     code:  assert conn.status == 200
     left:  401
     right: 200
     stacktrace:
       test/rest_api_test.exs:15: (test)

As you can see our tests rightly detect that the server rejected our request with a 401 Unauthorized status code.

To fix this we simply have to add the following lines to all our tests. We are going to use the put_req_header API to set the Authorization header. Note that header keys are almost always case-insensitive so we don't need to

defmodule RestApiTest.Router do
  #    ...
  test "GET / should return ok" do
    conn = conn(:get, "/")
    conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))

    conn = RestApi.Router.call(conn, @opts)
    #    ...
  end

  describe "Posts" do
	#    ...
    test "POST /post should create a post" do
   	#  ...
      # Make an API to call create a post
      conn = conn(:post, "/post", %{name: "Post 1", content: "Content of post"})
      conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))
    
      conn = RestApi.Router.call(conn, @opts)
	 # ...
    end

   # ...
    
    test "GET /posts should fetch all the posts" do
      createPosts()

      conn = conn(:get, "/posts")
      conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))
    
      conn = RestApi.Router.call(conn, @opts)
     # ...
    end

    test "GET /post/:id should fetch a single post" do
      [id | _] = createPosts()

      conn = conn(:get, "/post/#{id}")
      conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))
    
      conn = RestApi.Router.call(conn, @opts)

     # ...
    end

    test "PUT /post/:id should update a post" do
      [id | _] = createPosts()

      conn = conn(:put, "/post/#{id}", %{content: "Content 3"})
      conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))
    
      conn = RestApi.Router.call(conn, @opts)
	 # ...
    end

    test "DELETE /post/:id should delete a post" do
	  # ...

      conn = conn(:delete, "/post/#{id}", %{content: "Content 3"})
      conn = put_req_header(conn, "authorization", encode_basic_auth("<username>", "<password>"))
    
      conn = RestApi.Router.call(conn, @opts)
      # ...
    end
  end
end

test/rest_api_test.exs - add lines to file

That concludes part 4. To summarize,

You now have automated tests for your application.
You have added basic auth to protect your endpoint.

In the next post, we shall see how we can set up CI/CD for our application using GitHub Actions and Deploy it to some cloud hosting.

If you liked this tutorial then please support me by sharing this post with your friends. If you have questions consider becoming a member and adding your feedback as comments

Disclaimer: The ideas and opinions in this post are my own and not the views of my employer.