User Primary

In this post, we will add an HTTP interface to sherl using webmachine. The goal is a simple API that will shorten long URLs and redirect short URLs to their original page. Sending a GET to /http://long.it/is/your/url/here will create a short URL and display it. This will make the service easy to use by hand, as a user can simply prepend “http://sherl.com/” to an address in their browser’s address bar. All GET requests that don’t match “/http:” will be interpreted as short URLs and will redirect (HTTP 301) to the original page or give a 404 error if the specified short code was not found in the sherl database.

The first step is to obtain the latest webmachine code. You can find a tarball under the get source link on webmachine’s bitbucket page, or you can install Mercurial and pull down the latest bits. On OS X, here’s what I did:

sudo port install mercurial
hg clone https://userprimary@bitbucket.org/justin/webmachine/
cd webmachine
make

Next, create a skeleton webmachine project using the new_webmachine.erl script.

$PATH_TO_WEBMACHINE/scripts/new_webmachine.erl sherlweb .
cd sherlweb
make
./start-dev.sh

At this point, you should have a working webmachine application running at
http://localhost:8000/ that will return a Hello, new world message in a browser. Webmachine maps resources to erlang modules in which you define callback functions to implement different parts of HTTP. Webmachine brings the protocol back to web application frameworks and makes it easy to create a REST API that behaves according to the HTTP specification. A nice overview of how webmachine works is provided on the wiki. On that page is a link to an HTTP flowchart that gives a visual explanation to webmachine’s approach. The flowchart presents HTTP as a sequence of decision points covering all of the nooks and crannies of the protocol that you’d probably prefer to ignore. A webmachine resource can define a function for each decision point to control how the request should flow through the diagram. Fortunately, sane default functions are available for nearly all of the decision points so you only need to implement the parts of the protocol that are relevant to your application.

There are three significant benefits to the approach taken by the webmachine framework. First, it is much easier to end up with a web application that actually behaves according to the HTTP spec. This makes your application easier for clients to make use of and allows common pieces of web infrastructure like caches and proxies to play nice with your service. Secondly, by breaking down a resource into side-effect free functions, you end up with code that is easy maintain and easy to test. Finally, webmachine comes with a very clever visual debugger that displays an interactive version of the HTTP flowchart and allows you to inspect the request and decision point function output at each step. In this post we’ll just be scratching the surface of webmachine’s features, but you have to start somewhere.

We’ll begin customizing the skeleton by defining the resources in the sherlweb/priv/dispatch.conf file. This file maps URLs to resource modules and is similar to the routes configuration in a Ruby on Rails application. The dispatch configuration consists of a list of resource definitions. Each definition is a three element tuple: a pattern describing the URLs to match, the name of a module, and initialization parameters to send to the module.

%% dispatch.conf
{["http:", '*'], shorten_resource, []}.
{['*'], lookup_resource, []}.

The first directive will match resources that start with “/http:” and dispatch them to the shorten_resource module. The second directive matches any resource and will dispatch to the lookup_resource module. Matching is done in order with the first match winning so the second definition acts as a catch-all.

Now we’ll implement minimal versions of shorten_resource and lookup_resource. You can use the default sherlweb_resource module as a template and replace the to_html function as follows:

%% shorten_resource.erl
%% module boilerplate excluded
 
to_html(ReqData, State) ->
    "/" ++ LongUrl = wrq:raw_path(ReqData),
    ShortUrl = make_short_url(LongUrl, ReqData),
    {["<html><body><a href=\"", ShortUrl, "\">", ShortUrl,
      "</a></body></html>"],
     ReqData, State}.
 
make_short_url(LongUrl, ReqData) ->
    {ok, Code} = sherl:encode(LongUrl),
    Host = wrq:get_req_header("host", ReqData),
    "http://" ++ Host ++ "/" ++ Code.

%% lookup_resource.erl
%% module boilerplate excluded
 
to_html(ReqData, State) ->
    {ok, LongUrl} = sherl:decode(wrq:disp_path(ReqData)),
    {["<html><body><a href=\"", LongUrl, "\">", LongUrl,
      "</a></body></html>"],
     ReqData, State}.

In addition, we need to update sherlweb.app by removing sherlweb_resource and adding the above two modules (also delete sherlweb_resource.erl from the src directory). Then modify sherlweb.erl so that the sherl application implemented in Part 3 will be started as part of the web app initialization. The modified sherlweb:start/0 looks like this:

%% from sherlweb.erl
start() ->
    sherlweb_deps:ensure(),
    ensure_started(crypto),
    ensure_started(webmachine),
    ensure_started(mnesia),
    ensure_started(sherl),
    application:start(sherlweb).

At this point, rebuilding and restarting the project should allow you to test the basic API and see a very rudimentary HTML-based shortening service. To refine the service, we will iterate on the lookup_resource module so that it will return a 301 redirect for a valid short URL and a 404 otherwise. To do that we will implement the resource_exists, previously_existed, and moved_permanently callbacks. The new version of lookup_resource.erl looks like this:

-module(lookup_resource).
-export([init/1, moved_permanently/2, previously_existed/2,
         resource_exists/2]).
 
-include_lib("webmachine/include/webmachine.hrl").
 
-record(sherl, {long_url = undefined}).
 
init([]) -> {ok, #sherl{}}.
 
resource_exists(ReqData, State) ->
    {Status, LongUrl} = sherl:decode(wrq:disp_path(ReqData)),
    case Status of
        ok -> {false, ReqData, State#sherl{long_url = LongUrl}};
        _ -> {false, ReqData, State}
    end.
 
previously_existed(ReqData, State) ->
    case State#sherl.long_url of 
        undefined ->
            {false, ReqData, State};
        _ ->
            {true, ReqData, State}
    end.
 
moved_permanently(ReqData, State) ->
    case State#sherl.long_url of 
        undefined ->
            {false, ReqData, State};
        LongUrl ->
            {{true, LongUrl}, ReqData, State}
    end.

Based on the flowchart, the resource_exists function will be called first. It looks up the short code using sherl:decode/1 and updates the resource state record with the long URL if it is found. The resource_exists function always returns false since we never want to return content from this resource. We could optimize this a bit and return 404 without further decision function processing in the case that we don’t have a long URL for the requested short code. Implementing this short circuit is left as an exercise for the reader. Next the flow will call previously_existed which returns true if we found a long URL matching the requested short code and false otherwise. When previously_existed returns false, the server responds with a 404. When it returns true, the flow calls moved_permanently and webmachine generates a 301 response.

At this point we have a simple, but functioning URL shortening service backed by sherl, an OTP application that usees mnesia for storage, and fronted (is that a word?) by webmachine. This seems like as good a stopping point as any. I’m cleaning up the code and plan on posting it on github when it’s ready.

UPDATE the code for sherl is now available on github:
http://github.com/seth/sherl/tree/master

In this installment, we’ll turn the core sherl_db module into a proper OTP application by creating a sherl gen_server, a supervisor, and a an application resource file. While the sherl_db module can already be used as a server (the tests in part 2 show that it can be called concurrently), setting up a gen_server will provide a standardized framework for packaging the code as well as a nice abstraction layer should we decide to replace the mnesia-based sherl_db module with something else.

First we will tackle the gen_server part of the show. The erlang-mode for Emacs provides a very useful template for creating new gen_servers; even if you don’t use Emacs, you might want to steal the template idea as it proves quite useful. The gen_server will live in a module called simply sherl and in addition to the required gen_server behavior callbacks, exports the following API:

-export([start_link/0, stop/0, encode/1, decode/1]).

A client will call sherl:encode to turn a URL into a code that we will use as a short URL. The code we will use is the base 62 encoding of an integer that the sherl_db module assigns to the given URL. We choose base 62 as it allows us to represent short URL codes using A-Z, a-z, and 0-9 which will keep our short URLs very clean and easy to parse — and in particular, will reserve “.” for use as a format extension so that web users can append “.json” or “.xml” to retrieve the short URL meta data in a desired format. Below is the definition for the main sherl public API:

start_link() ->
    gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
 
stop() ->
  gen_server:call(?SERVER, stop).
 
encode(Url) ->
    gen_server:call(?SERVER, {encode, Url}).
 
decode(Code) ->
    gen_server:call(?SERVER, {decode, Code}).

While you don’t have to define a wrapper API for a gen_server, it makes using it that much more convenient. With the above API, client code can simply call sherl:encode(Url) instead of gen_server:call(sherl, {encode, Url}).

As you can see, each API call maps to a message sent as a tuple with first element describing the desired action. The server will read the message and dispatch to the appropriate function to respond to the client’s request.

Before going further with the API, a short diversion to implement the init callback. Here it is:

init([]) ->
    sherl_db:start([]),
    {ok, #state{}}.

Next up is implementing the handle_call callback. The is where the real work of the server gets done. In implementing the callback code, one realizes that what is really going on is message passing. The client sends the sherl gen_server a message and the sherl server examines the message to determine how to respond. Here’s what we need to support the API defined above:

handle_call({decode, Code}, _From, State) ->
    case sherl_db:get_url(base62:decode(Code)) of
        undefined ->
            {reply, {not_found, Code}, State};
        UrlRec ->
            {reply, {ok, UrlRec#url.url}, State}
    end;
handle_call({encode, Url}, _From, State) ->
    Rec = sherl_db:get_code(Url),
    Code = base62:encode(Rec#url.code),
    {reply, {ok, Code}, State};
handle_call(stop, _From, State) ->
    sherl_db:stop(),
    {stop, normal, stopped, State};
handle_call(_Request, _From, State) ->
    {reply, ignored, State}.

Our server doesn’t maintain any state between calls, but if it did, we would thread the state data through all the callbacks using the State variable that you see passed in as an argument and returned by all of the callbacks. Ok, takes care of the sherl gen_server. Now we need a supervisor and an application module. As these follow fairly mechanically from their required callbacks, I will simply list them below.

-module(sherl_sup).
-behaviour(supervisor).
-export([start_link/0]).
-export([init/1]).
-define(SERVER, ?MODULE).
 
start_link() ->
    supervisor:start_link({local, ?SERVER}, ?MODULE, []).
 
init([]) ->
    Sherl = {sherl_main, {sherl, start_link, []},
              permanent, 2000, worker,[sherl]},
    {ok, {{one_for_one, 20, 60}, [Sherl]}}.

-module(sherl_app).
-behaviour(application).
-export([start/2, stop/1]).
 
start(_Type, _StartArgs) ->
    case sherl_sup:start_link() of
        {ok, Pid} -> 
            {ok, Pid};
        Error ->
            Error
    end.
 
stop(_State) ->
    ok.

Finally, we need an application resource file (sherl.app) and that goes like this:

%% -*- mode: erlang -*-
{application, sherl, 
 [{description, "sherl URL shortening core service"}, 
  {vsn, "0.1"}, 
  {modules, [base62, sherl, sherl_app, sherl_db, sherl_sup]}, 
  {registered, [sherl, sherl_sup]}, 
  {applications, [kernel, stdlib, mnesia]}, 
  {mod, {sherl_app, []}}, 
  {start_phases, []} 
 ]}.

I put sherl.app in the src directory along with the *.erl files and have the build process copy it to ebin/sherl.app since application resource files need to be on your code path for the application module to find them.

We now have an OTP application. Here’s some example usage from the Erlang shell.

Eshell V5.7.1  (abort with ^G)
1> application:start(mnesia).
ok
2> application:start(sherl).
ok
3> sherl:encode("http://userprimary.net/user/2009/06/20/writing-a-url-shortening-service-in-erlang-part-3/").
{ok,"6"}
4> sherl:decode("6").
{ok,"http://userprimary.net/user/2009/06/20/writing-a-url-shortening-service-in-erlang-part-3/"}
5> sherl:decode("abc").
{not_found,"abc"}
6> gen_server:call(sherl, blah).
ignored
7> application:stop(sherl).

=INFO REPORT==== 20-Jun-2009::22:19:02 ===
    application: sherl
    exited: stopped
    type: temporary
ok
8>

In the sherl.app resource file, we listed mnesia as an application that must be running for sherl to start. The application framework will verify that mnesia is running and error out before starting sherl if mnesia is not running. It will not, however, start mnesia for you which is why, in the demo above, I had to explicitly start mnesia before starting sherl — even though the sherl_db start function will start mnesia if needed.

So far we’ve written the core database module, sherl_db, tested it using Common Test, and packaged it into an OTP application. In the next installment, we’ll add the HTTP interface for the sherl URL shortening service using the webmachine framework.

UPDATE I realized that the initialization of sherl isn’t quite right when starting for the first time. After posting a question to the Erlang questions mailing list, I came up with the following adjustment to sherl_db.erl based on the helpful replies I received:

start(Nodes) ->
    case is_fresh_startup() of
        true ->
            case mnesia:system_info(is_running) of
                yes ->
                    error_logger:info_report("stopping mnesia"),
                    mnesia:stop();
                _ -> pass
            end,
            mnesia:create_schema(Nodes),
            error_logger:info_report("mnesia schema created"),
            error_logger:info_report("starting mnesia"),
            mnesia:start(),
            mnesia:create_table(url, [{disc_copies, Nodes},
                                      {attributes, record_info(fields, url)},
                                      {index, [url]}]),
            mnesia:create_table(counter, [{disc_copies, Nodes},
                                          {attributes,
                                           record_info(fields, counter)}]),
            error_logger:info_report("mnesia tables created");
        {exists, Tables} ->
            ok = mnesia:wait_for_tables(Tables, 20000)
    end.
 
%% @spec is_fresh_startup() -> true | false
%% @doc Returns true if mnesia has not been initialized with
%% the sherl schema.
%% Thanks to Dale Harvey for this function posted to
%% the erlang questions mailing list.
is_fresh_startup() ->
    Node = node(),
    case mnesia:system_info(tables) of
        [schema] -> true;
        Tbls ->
            case mnesia:table_info(schema, cookie) of
                {_, Node} -> {exists, Tbls};
                _                 -> true
            end
    end.

The issue was that since mnesia is listed as a required application in sherl.app, it needs to be started before the sherl application can start. But once mnesia is started, you cannot (easily) create a schema. The approach taken above is to detect this fresh start situation and stop mnesia in order to initialize the desired schema. Other approaches, which perhaps are more appropriate for a production environment, that were suggested on the list include using the builder module which will create a load script that could be customized or to make use of mnesia:change_config (although I believe this requires a separate running mnesia).